Ubuntu服务器AS2通信HANDSHAKE_FAILURE：如何正确配置SSL证书？

It’s super common to hit handshake errors when setting up AS2, especially with self-signed certificates—let’s walk through the most likely culprits based on your setup.

First, let’s recap what you did: you generated a self-signed cert with openssl req -x509 -newkey rsa:2048 -nodes -keyout server.key -out server.crt -days 365 and sent it to your partner, but their connection fails with HANDSHAKE_FAILURE. Here’s what to check step by step:

1. Does your certificate’s Common Name (CN) match your server’s address? #

When TLS handshake happens, the client (your partner’s system) checks if the certificate’s CN matches the hostname/IP they’re connecting to. If you skipped filling in the correct CN during the openssl prompt, this will cause an immediate failure.

To verify, run this command on your server:

openssl x509 -in server.crt -text -noout | grep "Subject:"

Look for CN= in the output—it needs to exactly match the domain or IP your partner uses to reach your AS2 server. If it’s wrong, you’ll need to re-generate the certificate with the correct CN.

2. Has your partner trusted your self-signed certificate? #

Self-signed certs aren’t trusted by default by any system. Your partner needs to import your server.crt into their trusted root certificate store. If they skipped this step, their client will reject your certificate during handshake.

Double-check with them that they’ve added your cert to their trust store (the process varies by their OS/AS2 software—e.g., Windows uses Certificates MMC, Linux uses /etc/ssl/certs or the app-specific store).

3. Are your private key and certificate properly paired? #

It’s easy to mix up keys and certs, but if server.key doesn’t match server.crt, your AS2 server can’t complete the handshake. Verify they’re a pair with these commands:

Check the private key is valid:

openssl rsa -in server.key -check

Compare the public key from the cert and the key:

# Extract public key from cert
openssl x509 -in server.crt -pubkey -noout > cert_pub.key
# Extract public key from private key
openssl rsa -in server.key -pubout > key_pub.key
# Compare the two files
diff cert_pub.key key_pub.key

If the diff shows no output, they match. If there’s a difference, you need to re-generate the cert/key pair together (don’t reuse an old key with a new cert, or vice versa).

4. Does your AS2 server have proper access to the cert/key files? #

On Ubuntu, most services run with restricted permissions. If your AS2 server can’t read server.key or server.crt, it can’t present the certificate during handshake.

Fix the permissions:

chmod 600 server.key server.crt
chown [as2-service-user]:[as2-service-group] server.key server.crt

Replace [as2-service-user] and [as2-service-group] with the user/group your AS2 server runs as (e.g., www-data if it’s a web-based AS2 server like OpenAS2).

5. Does your certificate have the right extended key usages? #

AS2 requires certificates to be valid for server authentication, digital signature, and key encryption. The default openssl req -x509 command might not include these extensions, which can cause some strict AS2 clients to reject the cert.

Check your cert’s extensions:

openssl x509 -in server.crt -text -noout | grep -A 5 "X509v3 Extended Key Usage:"

You should see TLS Web Server Authentication, Digital Signature, and Key Encipherment. If not, re-generate the cert with proper extensions. Here’s how:

Create a small config file (name it as2_cert.cnf):

[req]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
x509_extensions = v3_ext

[dn]
C = CN
ST = YourProvince
L = YourCity
O = YourOrganization
OU = YourDepartment
CN = your-as2-server-domain.com  # Replace with your actual server domain/IP

[v3_ext]
subjectAltName = @alt_names
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth

[alt_names]
DNS.1 = your-as2-server-domain.com
IP.1 = 192.168.1.100  # Add this if your partner connects via IP

Then generate the cert with this config:

openssl req -x509 -newkey rsa:2048 -nodes -keyout server.key -out server.crt -days 365 -config as2_cert.cnf

This creates a cert with all the required extensions, plus Subject Alternative Names (SANs)—many modern clients require SANs even if the CN is correct.

Bonus: Check TLS version and cipher suites #

If all certificate checks pass, it’s worth confirming your AS2 server uses TLS 1.2 or higher (AS2 doesn’t support older versions like TLS 1.0) and that it supports cipher suites your partner’s client uses. Most AS2 servers let you configure these in their settings.

Try these steps first—certificate issues are almost always the root cause of HANDSHAKE_FAILURE in AS2 setups. Let me know if any of these checks turn up a problem!

内容的提问来源于stack exchange，提问作者user984621