使用OpenSSL开发C语言SSL服务器时遇ASN1编码及密钥设置错误

Hey there, let's work through this ASN.1 encoding error you're hitting with x509_sign—I've dealt with similar OpenSSL quirks before, so let's break down the most likely causes and fixes:

1. First: Verify Your Private Key & Certificate Files #

ASN.1 errors often stem from invalid or mismatched key/certificate pairs. Let's rule this out first with OpenSSL command-line checks:

• Check if your private key is valid:

  openssl rsa -in your_private_key.pem -check -noout
  

  If this throws errors, your PEM file is corrupted, missing headers/footers, or encrypted without the correct password.
• Verify key-certificate pairing:
  Compare the public key hash of your private key and certificate—they must match:

  # Get private key's public key hash
  openssl rsa -in your_private_key.pem -pubout | openssl md5
  # Get certificate's public key hash
  openssl x509 -in your_cert.pem -pubkey -noout | openssl md5
  

  If the hashes don't match, you're using a private key that doesn't correspond to the certificate—this will definitely break x509_sign.

2. Fix Your RSA Key Loading & EVP_PKEY Setup #

Looking at your code snippet, there are a couple of potential pitfalls here:

• You might be loading the RSA key incorrectly:
  Ensure you're using the right function to load your RSA private key. For unencrypted PEM keys, use PEM_read_RSAPrivateKey:

  FILE *key_file = fopen("your_private_key.pem", "r");
  if (!key_file) {
      perror("Failed to open private key file");
      return -1;
  }
  RSA *pkey_RSA = PEM_read_RSAPrivateKey(key_file, NULL, NULL, NULL);
  fclose(key_file);
  
  if (!pkey_RSA) {
      unsigned long err = ERR_get_error();
      char err_buf[256];
      ERR_error_string_n(err, err_buf, sizeof(err_buf));
      fprintf(stderr, "Failed to load RSA private key: %s\n", err_buf);
      return -1;
  }
  

  If your key is encrypted, you'll need to pass a password callback or plaintext password to PEM_read_RSAPrivateKey.
• Fix your error handling for EVP_PKEY_set1_RSA:
  Your current code calls ERR_get_error() twice, which overwrites the original error code (each call pulls the next error from the queue). Store the error code once and reuse it:

  EVP_PKEY *pkey = EVP_PKEY_new();
  if (!pkey) {
      fprintf(stderr, "Failed to create EVP_PKEY object\n");
      RSA_free(pkey_RSA);
      return -1;
  }
  
  if (EVP_PKEY_set1_RSA(pkey, pkey_RSA) == 0) {
      unsigned long err = ERR_get_error();
      char err_buf[256];
      ERR_error_string_n(err, err_buf, sizeof(err_buf));
      fprintf(stderr, "EVP_PKEY_set1_RSA failed: 0x%lx - %s\n", err, err_buf);
      EVP_PKEY_free(pkey);
      RSA_free(pkey_RSA);
      return -1;
  }
  

  Also, always clean up allocated resources (like pkey and pkey_RSA) on failure to avoid leaks.

3. Ensure Correct Digest Algorithm Usage for x509_sign #

If your key and certificate are valid, the issue might be with the digest algorithm you're using to sign the certificate. Make sure you're using a digest that's compatible with your RSA key (e.g., SHA-256, SHA-384) and that you're initializing the EVP_MD correctly:

// Example of signing an X.509 certificate with SHA-256
if (!X509_sign(x509_cert, pkey, EVP_sha256())) {
    unsigned long err = ERR_get_error();
    char err_buf[256];
    ERR_error_string_n(err, err_buf, sizeof(err_buf));
    fprintf(stderr, "x509_sign failed: 0x%lx - %s\n", err, err_buf);
    // Cleanup resources here
    return -1;
}

Avoid outdated digests like MD5 or SHA-1, as they're no longer supported in modern OpenSSL versions and can cause encoding errors.

4. Check for Corrupted Certificate ASN.1 Structure #

Use OpenSSL to validate your certificate's ASN.1 structure:

openssl x509 -in your_cert.pem -text -noout

If this command fails to print the certificate details, your certificate file has invalid ASN.1 encoding—you'll need to regenerate the certificate properly (e.g., using openssl req and openssl x509 commands).

内容的提问来源于stack exchange，提问作者tweet