Let’s walk through why your Windows application is failing to establish an HTTPS connection with your Nginx server, and how to fix it based on your provided details.

Key Observations from Your Setup #

First, let’s recap the critical context we have:

• Nginx version: 1.12.2 (released in 2017, end-of-life with no updates)
• OpenSSL version: 1.0.1f (released in 2014, no longer supported with security or bug fixes)
• SSL config enables TLSv1/TLSv1.1/TLSv1.2 and uses ECDHE/AES-based cipher suites
• Wireshark captures show Windows app requests, with responses from two servers (Server1/Server2) but no successful handshake completion

Top Root Causes & Fixes #

1. Outdated OpenSSL/Nginx Versions (Most Likely Culprit) #

OpenSSL 1.0.1f is extremely old—it lacks proper support for many modern TLS features and has known compatibility gaps with Windows’ TLS stack (even older Windows 7/Server 2008 R2 can run into handshake mismatches, and newer Windows 10/11 builds will reject connections to such outdated servers). Nginx 1.12.2 also no longer receives security patches or bug fixes.

Fix:

• Upgrade Nginx to a supported LTS version (e.g., 1.24.x or newer)
• Upgrade OpenSSL to at least 1.1.1 (LTS, supported until 2025) or 3.x. This will resolve most core compatibility issues with Windows’ TLS implementation.

2. Incomplete Certificate Chain #

Windows’ certificate store is strict about valid certificate chains. If your cert.pem only contains the leaf server certificate (and not the intermediate CA certificates), the Windows app will reject the server’s identity, triggering a handshake failure.

Fix:

• Verify and complete your certificate chain: Combine your leaf certificate with all intermediate CA certificates into cert.pem (order matters: leaf first, then intermediates from closest to root CA).
• You can check the chain by running:

  openssl x509 -in /etc/nginx/cert/cert.pem -text -noout
  

  Ensure the "Issuer" field links up to a trusted root CA.

3. Cipher Suite Mismatch #

Your current cipher list includes ECDHE-based suites, but old OpenSSL versions have limited support for ECDHE key exchange. Windows apps might not support the specific ciphers you’ve prioritized, leading to a "no common cipher suite" error.

Fix:
Update your ssl_ciphers to a more Windows-friendly set, using Mozilla’s recommended compatible configuration:

ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA";

Keep ssl_prefer_server_ciphers on—this list balances security and Windows compatibility.

4. Protocol Version Compatibility #

While you’ve enabled TLSv1 to TLSv1.2, some older Windows apps might default to deprecated protocols (like SSLv3, which you’ve correctly disabled), or newer Windows apps might expect TLSv1.2 with specific extensions that old OpenSSL doesn’t support.

Fix:

• If you don’t need to support pre-Windows 7 systems, disable TLSv1 and TLSv1.1 for better security:

  ssl_protocols TLSv1.2 TLSv1.3; # Add TLSv1.3 only after upgrading OpenSSL to 1.1.1+
  

• If legacy Windows support is required, keep TLSv1/TLSv1.1 enabled but ensure your cipher list includes suites compatible with those versions.

Deep Dive with Wireshark #

To narrow down the issue further, check your capture for:

• Handshake failure stage: Does the server fail to send a Server Hello after the Client Hello? Or does it send an Alert message?
• Alert code: Common codes like 40 (Handshake Failure) point to cipher mismatches, while 46 (Certificate Unknown) indicates an invalid certificate chain.
• Server1 vs Server2 differences: If one server works and the other doesn’t, compare their SSL configs, OpenSSL versions, or certificate chains to spot discrepancies.

内容的提问来源于stack exchange，提问作者zhangjpn