如何判断OpenSSL解密时密钥无效或与加密密钥不一致

Hey there,

Great question—this is a common pitfall when working with OpenSSL's EVP encryption/decryption API, since a return value of 0 can stem from multiple issues, not just an invalid or mismatched key. Let’s walk through how to distinguish key-related failures from other errors, and how to communicate this clearly to your users.

First, Understand Why OpenSSL Returns 0 #

Each of the EVP functions you’re calling can fail for different reasons:

• EVP_CIPHER_CTX_new(): Rarely fails, but usually due to memory allocation issues (not key-related).
• EVP_DecryptInit_ex(): Fails if the cipher algorithm is invalid, the key/IV length doesn’t match the cipher requirements (e.g., passing a 16-byte key for AES-256), or context setup goes wrong.
• EVP_DecryptUpdate(): Fails mostly due to invalid input data (e.g., null pointers, corrupted data) rather than key issues.
• EVP_DecryptFinal_ex(): This is where you’ll most commonly catch key/IV mismatches. When the key is wrong, the decrypted plaintext’s PKCS#7 padding (the default for most ciphers) won’t be valid, so this call will fail with a "bad decrypt" error.

Step 1: Capture Detailed Error Information #

OpenSSL maintains an error stack that logs specific failure reasons. You can retrieve these errors using ERR_get_error() and convert them to human-readable strings with ERR_error_string(). Make sure to handle thread safety (if your code is multi-threaded) and clear the error stack before making OpenSSL calls to avoid leftover errors from previous operations.

Step 2: Distinguish Key-Related Failures #

Here’s how to modify your decryptData method to detect and report key-specific issues:

// Note: Added length parameters for encryptedData and decryptedData for safe operation
int decryptData(unsigned char* key, unsigned char* iv, 
                unsigned char* encryptedData, int encryptedDataLen, 
                unsigned char* decryptedData, int* decryptedDataLen) {
    
    EVP_CIPHER_CTX* ctx = EVP_CIPHER_CTX_new();
    if (!ctx) {
        // Memory allocation failure, not key-related
        return -1;
    }

    // Replace EVP_aes_256_cbc() with your actual cipher algorithm
    if (EVP_DecryptInit_ex(ctx, EVP_aes_256_cbc(), NULL, key, iv) != 1) {
        unsigned long err = ERR_get_error();
        // Check if failure is due to invalid key length for the cipher
        if (ERR_GET_REASON(err) == EVP_R_INVALID_KEY_LENGTH) {
            EVP_CIPHER_CTX_free(ctx);
            return -2; // Custom error code: Invalid key length for chosen cipher
        }
        // Other init failures (e.g., invalid cipher)
        EVP_CIPHER_CTX_free(ctx);
        return -1;
    }

    int updateLen;
    if (EVP_DecryptUpdate(ctx, decryptedData, &updateLen, encryptedData, encryptedDataLen) != 1) {
        // Usually data corruption or input issues, not key problems
        EVP_CIPHER_CTX_free(ctx);
        return -1;
    }
    *decryptedDataLen = updateLen;

    int finalLen;
    if (EVP_DecryptFinal_ex(ctx, decryptedData + updateLen, &finalLen) != 1) {
        unsigned long err = ERR_get_error();
        // "Bad decrypt" indicates padding validation failed—most likely wrong key/IV
        if (ERR_GET_REASON(err) == EVP_R_BAD_DECRYPT) {
            EVP_CIPHER_CTX_free(ctx);
            return -3; // Custom error code: Key/IV mismatch or tampered data
        }
        // Other finalization failures (e.g., buffer too small)
        EVP_CIPHER_CTX_free(ctx);
        return -1;
    }
    *decryptedDataLen += finalLen;

    EVP_CIPHER_CTX_free(ctx);
    return 0; // Success
}

Key Notes to Consider: #

• Custom Error Codes: Use distinct return codes (like -2 and -3) to signal specific key-related issues to your caller, who can then display appropriate messages to the user (e.g., "Invalid key length—AES-256 requires a 32-byte key" or "Decryption failed: Check your key or IV").
• Padding vs. No Padding: If you’re using a cipher mode without padding (e.g., AES-ECB with EVP_CIPHER_CTX_set_padding(ctx, 0)), EVP_DecryptFinal_ex() won’t fail for key mismatches—instead, you’ll get garbled plaintext. For these cases, add an extra validation step: include a checksum (like CRC32) or HMAC in the encrypted data, then verify it after decryption. If the check fails, you can infer the key was invalid.
• Error String Reliability: While ERR_error_string() gives human-readable messages, using ERR_GET_REASON(err) to check the numeric error reason is more reliable across different OpenSSL versions.
• Clear the Error Stack: Before making OpenSSL calls, call ERR_clear_error() to ensure you’re only capturing errors from the current operation.

Communicating to Users #

Once you’ve identified the failure type:

• For EVP_R_INVALID_KEY_LENGTH: Tell the user their key is the wrong length for the encryption algorithm you’re using (specify the required length).
• For EVP_R_BAD_DECRYPT: Explain that the key/IV might be incorrect, or the encrypted data was tampered with (since both can cause padding validation to fail).

内容的提问来源于stack exchange，提问作者sam2215