After a lot of testing I have worked out this problem. The following has been changed.

• The policy is set to NO for server evaluation. This means the certificate is checked for client authentication. Obviously the server certificate will not have this! Setting this to YES will actually check if extendedKeyUsage is set to serverAuth for the server certificate.

• SecTrustSetAnchorCertificates and SecTrustSetAnchorCertificatesOnly should always be called before evaluation and not only if you are providing your own anchor certificates. You need to call this with an empty array, otherwise the system known anchor certificates are not used for evaluation. Even installed trusted root certificates from MDM are working then.

Here is a working sample based on the first code:

if (challenge.protectionSpace.authenticationMethod == NSURLAuthenticationMethodServerTrust) {
    // create trust from protection space
    SecTrustRef trustRef;
    int trustCertificateCount = SecTrustGetCertificateCount(challenge.protectionSpace.serverTrust);

    NSMutableArray* trustCertificates = [[NSMutableArray alloc] initWithCapacity:trustCertificateCount];
    for (int i = 0; i < trustCertificateCount; i++) {
        SecCertificateRef trustCertificate =  SecTrustGetCertificateAtIndex(challenge.protectionSpace.serverTrust, i);
        [trustCertificates addObject:(id) trustCertificate];
    }            

    // set evaluation policy
    SecPolicyRef policyRef;
    // set to YES to verify certificate extendedKeyUsage is set to serverAuth
    policyRef = SecPolicyCreateSSL(YES, (CFStringRef) challenge.protectionSpace.host);
    SecTrustCreateWithCertificates((CFArrayRef) trustCertificates, policyRef, &trustRef);

    [trustCertificates release];

    // load known certificates from keychain and set as anchor certificates
    NSMutableDictionary* secItemCopyCertificatesParams = [[NSMutableDictionary alloc] init];    
    [secItemCopyCertificatesParams setObject:(id)kSecClassCertificate forKey:(id)kSecClass];
    [secItemCopyCertificatesParams setObject:@"Server_Cert_Label" forKey:(id)kSecAttrLabel];
    [secItemCopyCertificatesParams setObject:(id)kCFBooleanTrue forKey:(id)kSecReturnRef];
    [secItemCopyCertificatesParams setObject:(id)kSecMatchLimitAll forKey:(id)kSecMatchLimit];

    CFArrayRef certificates;
    certificates = nil;
    SecItemCopyMatching((CFDictionaryRef) secItemCopyCertificatesParams, (CFTypeRef*) &certificates);

    if (certificates != nil && CFGetTypeID(certificates) == CFArrayGetTypeID()) {
        SecTrustSetAnchorCertificates(trustRef, certificates);
        SecTrustSetAnchorCertificatesOnly(trustRef, NO);
    } else {
        // set empty array as own anchor certificate so system anchos certificates are used too!
        SecTrustSetAnchorCertificates(trustRef, (CFArrayRef) [NSArray array]);
        SecTrustSetAnchorCertificatesOnly(trustRef, NO);
    }

    SecTrustResultType result;
    OSStatus trustEvalStatus = SecTrustEvaluate(trustRef, &result);
    if (trustEvalStatus == errSecSuccess) {
        if (result == kSecTrustResultConfirm || result == kSecTrustResultProceed || result == kSecTrustResultUnspecified) {
            // evaluation OK
            [challenge.sender useCredential:[NSURLCredential credentialForTrust: challenge.protectionSpace.serverTrust] forAuthenticationChallenge:challenge];
        } 
        else {
            // evaluation failed 
            // ask user to add certificate to keychain
        }
    } 
    else {
        // evaluation failed - cancel authentication
        [[challenge sender] cancelAuthenticationChallenge:challenge];
    }
}

Hope this will help someone.

I've tested your code and there is nothing wrong with it. The problem seems to be related with the format of the certificates that you are trying to get the public key.

The function SecCertificateCreateWithData() assumes that the certificate that you are providing is in DER format. Most certificates that you find are encoded in base64 like the famous .pem format. I've tested your code with a correctly formatted DER certificate (the certificate form developer.apple.com converted to DER with openssl) and the public key is correctly extracted.

To convert a .pem certificate to DER simply use openssl in terminal:

openssl x509 -in developer.apple.com.pem  -outform der -out cert.der

After that the output certificate file should work with no problems with your code.

But you can convert the certificate on the application itself, you only need to grab de x509 base64 encoded certificate (assuming that you are using .pem encoded certificates) and convert it to binary.

There is an example how you can do it:

This code will assume that the certificate is encoded in the following standard:

-----BEGIN CERTIFICATE-----
< your base64 encoded certificate goes here >
-----END CERTIFICATE-----

The code to convert this certificate to binary DER is:

-(NSData *)getBinaryCertificateFromPemEncodedFile:(NSString *)filename andExtension:(NSString *)extension
{
    NSString* filePath = [[NSBundle mainBundle] pathForResource:filename ofType:extension];
    NSString *pemCert = [NSString stringWithContentsOfFile:filePath encoding:NSUTF8StringEncoding error:nil];
    //The header and footer conforms to .pem specificatio
    NSString *header = @"-----BEGIN CERTIFICATE-----";
    NSString *footer = @"-----END CERTIFICATE-----";

    NSString *base64Cert;
    NSScanner *scanner = [NSScanner scannerWithString:pemCert];
    //First we ignore the header part
    [scanner scanString:header intoString:nil];
    //Then we copy the base64 string excluding the footer
    [scanner scanUpToString:footer intoString:&base64Cert];

    //The reason I'm using NSDataBase64DecodingIgnoreUnknownCharacters is to exclude possible line breaks in the encoding
    NSData *binaryCertificate = [[NSData alloc] initWithBase64EncodedString:base64Cert options:NSDataBase64DecodingIgnoreUnknownCharacters];

    return binaryCertificate;
}

Then a small adaptation in your perfectly functional code does the trick:

-(SecKeyRef)readPublicKeyFromCertificate:(NSData *)binaryCertificate {

    NSData *encodedKey = binaryCertificate;

    CFDataRef myCertData = (CFDataRef)CFBridgingRetain(encodedKey);

    SecCertificateRef cert = SecCertificateCreateWithData(kCFAllocatorSystemDefault, myCertData);
    SecPolicyRef policy = SecPolicyCreateBasicX509();

    SecTrustRef trust;
    //If you only have one certificate you don't need to put it inside an array
    OSStatus check =  SecTrustCreateWithCertificates(cert, policy, &trust);

    if (check != noErr)
    {
        NSLog(@"Problem extracting public key from certificate");
        return nil;
    }

    SecTrustResultType trustResult;
    SecTrustEvaluate(trust, &trustResult);
    SecKeyRef pub_key_leaf = SecTrustCopyPublicKey(trust);

    return pub_key_leaf;
}

Then just call it:

NSData *data = [self getBinaryCertificateFromPemEncodedFile:@"developer" andExtension:@"pem"];
SecKeyRef key = [self readPublicKeyFromCertificate:data];
NSLog(@"%@", key);

And if your certificate is "valid" you should see:

2014-09-15 21:52:13.275 cert[15813:60b] <SecKeyRef algorithm id: 1,
key type: RSAPublicKey, version: 2, block size: 2048 bits, exponent: {hex: 10001, decimal: 65537},
modulus: BE19E30F47F2D31F27D576CF007B3E615F986D14AFD0D52B825E01E90BA3E1CBB6F3A472E6AECDC28BC13D0B6E58FC497ACF61D80F274E4799602DA4F819E54ADDE2FBFA89FC4EB2172501DDED8DE0FBDDBC5550CC018C73E1FD8152C905DE850862B8D57596025DE1908D8337E95637AF0F52C4A11DA178FF737DCE09471BC0A49DAD7DB39F1BA1B693D3A12F9CA50EF388B50292C73076BF1EEE412A5CFA940E99D4CF07F17FAC87F0D0E2FC8FA3ACDDEEFCCE8AFEC407B94536FCB1E4ACF34773728D189F85EAE4347E0BF868D25C7CE89F8A29B4E6865C68F4F915DFA540549EE9333007145D367FE2852622AAD776F3E5D505A02E5155CC8646A01C1031,
addr: 0x9a48200>

For testing I've used the certificate from developer.apple.com, you can check the public key in the log and compare it.

So, after digging around i found out that SecPolicyCreateSSL is working as expect, still not 100% why SecPolicyCreateBasicX509 is not.

So for future ref if someone have this issue, this is what we did instead,

SecPolicyRef policy = SecPolicyCreateSSL(true, NULL);
SecTrustRef testTrust;

OSStatus status = SecTrustCreateWithCertificates((__bridge CFArrayRef)fullChain, policy, &testTrust);
status = SecTrustEvaluate(testTrust, &trustResult);

CFRelease(testTrust);
CFRelease(policy);

return (status == errSecSuccess) && (kSecTrustResultUnspecified == trustResult || kSecTrustResultProceed == trustResult);;

(basically using SecPolicyCreateSSL instead)