使用OpenSSL内存BIO实现握手时出现版本号错误的问题求助

Hey there, let's break down your OpenSSL memory BIO handshake issues step by step. Those error messages are super common when you're first working with memory BIOs—unlike network BIOs, they don't handle data transfer automatically, so you have to manually pass data between client and server buffers during the handshake.

First, let's decode your errors #

• The server's wrong version number error usually means either:
  • Your client and server are configured with incompatible SSL/TLS versions (e.g., one uses TLS 1.2, the other is forced to TLS 1.3 only)
  • You're not properly transferring full handshake data between the memory BIOs, leading to malformed packets.
• The client's ossl_statem_client_read_transition: unexpected message error ties into this: it gets data that doesn't match the expected state in the handshake flow, because the data transfer between BIOs is broken.

Key Fixes & Example Code #

Let's build a minimal working example that handles memory BIO handshake correctly. We'll cover initialization, data transfer, and the handshake loop.

1. Initialize SSL Contexts (Client + Server)

Make sure both ends use compatible protocol versions. Avoid hardcoding old protocols like SSLv3—use the modern, flexible methods instead.

#include <openssl/ssl.h>
#include <openssl/bio.h>
#include <stdio.h>
#include <stdlib.h>

SSL_CTX* create_ssl_ctx(bool is_server) {
    const SSL_METHOD* method = is_server ? TLS_server_method() : TLS_client_method();
    SSL_CTX* ctx = SSL_CTX_new(method);
    if (!ctx) {
        perror("Failed to create SSL_CTX");
        exit(EXIT_FAILURE);
    }

    // Server needs cert/key; client can skip verification for testing (don't do this in production!)
    if (is_server) {
        if (SSL_CTX_use_certificate_file(ctx, "server.crt", SSL_FILETYPE_PEM) <= 0 ||
            SSL_CTX_use_PrivateKey_file(ctx, "server.key", SSL_FILETYPE_PEM) <= 0 ||
            !SSL_CTX_check_private_key(ctx)) {
            perror("Failed to load server cert/key");
            SSL_CTX_free(ctx);
            exit(EXIT_FAILURE);
        }
    } else {
        SSL_CTX_set_verify(ctx, SSL_VERIFY_NONE, NULL);
    }

    return ctx;
}

2. Transfer Data Between Memory BIOs

Since memory BIOs don't handle network I/O, we need a helper to copy data from one BIO's output to another's input.

int transfer_bio_data(BIO* src, BIO* dst) {
    char buf[4096];
    int bytes_read, bytes_written;
    int total = 0;

    while ((bytes_read = BIO_read(src, buf, sizeof(buf))) > 0) {
        bytes_written = BIO_write(dst, buf, bytes_read);
        if (bytes_written != bytes_read) {
            fprintf(stderr, "Failed to write all data to BIO\n");
            return -1;
        }
        total += bytes_written;
    }

    // Temporary lack of data isn't an error (BIO_should_retry means try again later)
    if (bytes_read < 0 && !BIO_should_retry(src)) {
        fprintf(stderr, "BIO_read failed\n");
        return -1;
    }

    return total;
}

3. Handshake Loop (Client + Server)

The critical part: you have to loop through SSL_connect/SSL_accept calls, handling SSL_ERROR_WANT_READ/SSL_ERROR_WANT_WRITE by transferring data between BIOs each time.

int main() {
    // Initialize OpenSSL
    OPENSSL_init_ssl(OPENSSL_INIT_LOAD_SSL_STRINGS, NULL);
    OPENSSL_init_crypto(OPENSSL_INIT_LOAD_CRYPTO_STRINGS, NULL);

    // Create contexts and BIOs
    SSL_CTX* server_ctx = create_ssl_ctx(true);
    SSL_CTX* client_ctx = create_ssl_ctx(false);

    BIO* server_read = BIO_new(BIO_s_mem());
    BIO* server_write = BIO_new(BIO_s_mem());
    BIO* client_read = BIO_new(BIO_s_mem());
    BIO* client_write = BIO_new(BIO_s_mem());

    // Set up SSL objects
    SSL* server_ssl = SSL_new(server_ctx);
    SSL_set_bio(server_ssl, server_read, server_write);
    SSL_set_accept_state(server_ssl);

    SSL* client_ssl = SSL_new(client_ctx);
    SSL_set_bio(client_ssl, client_read, client_write);
    SSL_set_connect_state(client_ssl);

    // Run handshake loop
    int client_ret = SSL_connect(client_ssl);
    int server_ret;
    bool handshake_done = false;

    while (!handshake_done) {
        int client_err = SSL_get_error(client_ssl, client_ret);
        int server_err;

        if (client_err == SSL_ERROR_WANT_WRITE) {
            // Client has data to send—pass to server's read BIO
            transfer_bio_data(client_write, server_read);
            // Let server process the data
            server_ret = SSL_accept(server_ssl);
            server_err = SSL_get_error(server_ssl, server_ret);
            
            if (server_err == SSL_ERROR_WANT_WRITE) {
                // Server has response data—pass to client's read BIO
                transfer_bio_data(server_write, client_read);
            } else if (server_ret == 1) {
                // Server handshake done; check client again
                client_ret = SSL_connect(client_ssl);
                if (client_ret == 1) handshake_done = true;
            }
        } else if (client_err == SSL_ERROR_WANT_READ) {
            // Client needs data—let server generate it first
            server_ret = SSL_accept(server_ssl);
            server_err = SSL_get_error(server_ssl, server_ret);
            
            if (server_err == SSL_ERROR_WANT_WRITE) {
                transfer_bio_data(server_write, client_read);
            }
            client_ret = SSL_connect(client_ssl);
            if (client_ret == 1) handshake_done = true;
        } else if (client_ret == 1) {
            handshake_done = true;
        } else {
            fprintf(stderr, "Client error: %s\n", ERR_error_string(ERR_get_error(), NULL));
            goto cleanup;
        }
    }

    printf("Handshake completed successfully on both ends!\n");

cleanup:
    SSL_free(server_ssl);
    SSL_free(client_ssl);
    SSL_CTX_free(server_ctx);
    SSL_CTX_free(client_ctx);
    // BIOs are automatically freed by SSL_free, so no need to call BIO_free here
    return 0;
}

Critical Notes to Avoid Your Errors #

• Always handle SSL_ERROR_WANT_READ/SSL_ERROR_WANT_WRITE: These aren't failures—they mean OpenSSL needs more data to read or has data to write. You must transfer the BIO data and retry the handshake call.
• Use compatible protocol versions: TLS_server_method() and TLS_client_method() automatically negotiate the highest compatible version, avoiding version mismatches.
• Don't skip data transfer: Every time OpenSSL signals it needs to read/write, you have to copy the data between the client and server BIOs—this is the "network" layer you're replacing with memory buffers.

内容的提问来源于stack exchange，提问作者xvnm