TLS: How to verify a certificate self-signed with an RSA key
I am trying to use OpenSSL and TLS to provide SSH-like trust model. That means that the two peers have an RSA key pair stored each one, no certificates. They have also exchanged their public keys prior to the TLS session establishment. To achieve this I am using some OpenSSL's undocumented functions so I still have doubts, here is how:
During initialisation peers generate an in-memory, temporary x509 certificate. It contains the minimal dummy data I could get away with, i.e. a dummy CN, issuer, and many years before expiring. These are never checked anyway. SSL context is setup to exchange both certificates both ways.
The important step is that the certificate contains the host's public key and it is signed using the private key:
This certificate is assigned to SSL_CTX and used for the whole application lifetime.
What I'm mostly worried about is the verification
process when the TLS session is established. I'm disabling all verification performed by OpenSSL using SSL_CTX_set_cert_verify_callback(always_true_func) and roll my own like this:
The question: Is this proper usage of the OpenSSL API? Do you see any security holes, especially on the last part, verification of the received key? Any better way to achieve the same result?
EDIT: The answer: To verify that a self-signed certificate received during TLS handshake matches the stored RSA key, there is no need to check the certificate's signature. i.e. there is no need for X509_verify. The comparison of the received public key to the expected one is enough.
The reason is that (quoting Dr Stephen Henson, OpenSSL project core developer) "Depending on the ciphersuite either an RSA decryption operation or an RSA signature operation is performed by the server. So if the handshake completes successfully you can be sure that the same key is used as the one present in the certificate ".