如何通过OpenSSL限制SSL重协商次数而非完全禁用以防范DoS攻击？

Great question—this is a crucial security concern for SSL/TLS servers, especially when facing potential DoS attacks via renegotiation floods. Let’s walk through your options step by step.

OpenSSL doesn’t include a native flag to directly cap renegotiation counts, but you can build this logic yourself using custom callbacks and connection-specific tracking. Here’s how:

Custom Callback + Per-Connection Counter #

You can track renegotiation attempts per SSL connection using OpenSSL’s extended data storage and an info callback:

• First, register an index to store a counter in each SSL object (this lets you attach custom data to connections).
• Use an info callback to detect when a renegotiation handshake starts, increment the counter, and block further attempts once your threshold is hit.

Example code snippet:

// Initialize an index to store our renegotiation counter
static int reneg_count_idx = -1;

void init_reneg_limit() {
    reneg_count_idx = SSL_get_ex_new_index(0, "renegotiation_count", NULL, NULL, NULL);
}

// Info callback to monitor handshake events
void ssl_info_callback(const SSL *ssl, int where, int ret) {
    // Check if this is a renegotiation handshake start
    if ((where & SSL_CB_HANDSHAKE_START) && SSL_in_renegotiation(ssl)) {
        int *count = SSL_get_ex_data(ssl, reneg_count_idx);
        if (count) {
            (*count)++;
            // Block after 3 attempts (adjust this threshold as needed)
            if (*count > 3) {
                SSL *writable_ssl = (SSL *)ssl;
                SSL_shutdown(writable_ssl);
                SSL_set_error(writable_ssl, SSL_ERROR_SSL);
            }
        }
    }
}

// Set up the SSL context with the callback
SSL_CTX *ctx = SSL_CTX_new(TLS_server_method());
SSL_CTX_set_info_callback(ctx, ssl_info_callback);

// Initialize the counter for new connections
SSL *ssl = SSL_new(ctx);
int *count = malloc(sizeof(int));
*count = 0;
SSL_set_ex_data(ssl, reneg_count_idx, count);

Renegotiation Callback (OpenSSL 1.1.0+) #

If you’re using OpenSSL 1.1.0 or newer, you can use the dedicated SSL_CTX_set_renegotiate_callback() to control renegotiation permissions directly. This callback lets you approve or reject each renegotiation attempt based on your counter:

int renegotiate_callback(SSL *ssl, int initiated_by_client) {
    int *count = SSL_get_ex_data(ssl, reneg_count_idx);
    // Reject if we've hit our limit
    if (count && *count >= 3) return 0;
    // Increment counter if we allow the attempt
    if (count) (*count)++;
    return 1;
}

// Attach the callback to your context
SSL_CTX_set_renegotiate_callback(ctx, renegotiate_callback);

Beyond counting attempts, here are additional ways to harden your server against renegotiation-based DoS:

• Disable session resumption on renegotiation: Set SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION to force full handshakes for every renegotiation. This increases the attacker’s resource cost (though it also uses more server resources, so pair it with other limits):

  SSL_CTX_set_options(ctx, SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION);
  

• Shorten session timeouts: Use SSL_CTX_set_timeout() to limit how long sessions can be reused for renegotiation. This reduces the window for attackers to exploit stale sessions:

  SSL_CTX_set_timeout(ctx, 3600); // Time out sessions after 1 hour
  

• Upgrade to TLS 1.3: TLS 1.3 eliminates traditional renegotiation entirely, replacing it with more secure, lightweight session resumption mechanisms. This is the most robust long-term fix if your clients support it.

内容的提问来源于stack exchange，提问作者Kethiri Sundar