Nginx中ssl_preread on导致SSL负载无法转发至上游TCP服务器的原因问询

背景与最终配置 #

近期实现Nginx SSL负载转发至上游TCP服务器，经过多次调试后得到可用配置如下：

stream {
    map $ssl_server_name $stream_map {
        aby3_task_1 upstream_task_1;
        aby3_task_2 upstream_task_2;
    }

    upstream upstream_task_1 {
        server 127.0.0.1:1313;
    }

    upstream upstream_task_2 {
        server 127.0.0.1:1314;
    }

    server {
        listen 8185 ssl;
        ssl_certificate /home/ttt/nginx-cfg/cert/server1.crt;
        ssl_certificate_key /home/ttt/nginx-cfg/cert/server1.key;
        proxy_pass $stream_map;
        ssl_preread off;
    }

    error_log /etc/nginx/logs/error.log debug;
}

问题排查过程 #

最初配置中加入了ssl_preread on，但一直无法按服务器名（SNI）转发SSL流量到对应上游TCP服务器。误删该配置项后，转发功能恢复正常，由于ssl_preread默认值为off，最终配置保留了ssl_preread off。

日志对比分析 #

为明确ssl_preread on的影响，收集了转发正常与异常时的调试日志：

正常转发日志（ssl_preread off） #

# This is the log when nginx can forward ssl payload to upstream tcp server.
2023/01/28 14:25:17 [info] 2952111#2952111: *1204 client 192.168.16.184:33868 connected to 0.0.0.0:8185
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 posix_memalign: 000055B8CEC6B700:256 @16
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 generic phase: 0
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 generic phase: 1
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 generic phase: 2
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 tcp_nodelay
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 SSL_do_handshake: -1
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 SSL_get_error: 2
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 epoll add event: fd:4 op:1 ev:80002001
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 event timer add: 4: 60000:8285386327
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 SSL handshake handler: 0
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 SSL_do_handshake: 1
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 SSL: TLSv1.2, cipher: "ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD"
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 event timer del: 4: 8285386327
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 generic phase: 2
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 ssl preread handler
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 proxy connection handler
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 malloc: 000055B8CEC92630:448
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 posix_memalign: 000055B8CEC8D1C0:256 @16
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 malloc: 000055B8CEC87C60:16384
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 post event 000055B8CECC87A0
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 stream map started
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 stream script var: "aby3_task_3"
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 stream map: "aby3_task_3" "upstream_task_3"
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 stream script var: "upstream_task_3"
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 get rr peer, try: 1
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 stream socket 18
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 epoll add connection: fd:18 ev:80002005
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 connect to 127.0.0.1:1315, fd:18 #1205
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 proxy connect: -2
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 event timer add: 18: 60000:8285386327
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 delete posted event 000055B8CECC87A0
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 SSL_read: -1
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 SSL_get_error: 2
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 event timer del: 18: 8285386327
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 stream proxy connect upstream
2023/01/28 14:25:17 [debug] 2952111#2952111: *1204 tcp_nodelay

异常转发日志（ssl_preread on） #

# This is the log when nginx can't forward ssl payload to upstream tcp server.
2023/01/28 14:23:21 [info] 2950336#2950336: *1192 client 192.168.16.184:60998 connected to 0.0.0.0:8185
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 posix_memalign: 000055B8CEC6A530:256 @16
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 generic phase: 0
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 generic phase: 1
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 generic phase: 2
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 tcp_nodelay
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL_do_handshake: -1
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL_get_error: 2
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 epoll add event: fd:11 op:1 ev:80002001
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 event timer add: 11: 60000:8285270365
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL handshake handler: 0
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL_do_handshake: 1
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL: TLSv1.2, cipher: "ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD"
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL reused session
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 event timer del: 11: 8285270365
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 generic phase: 2
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 ssl preread handler
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 malloc: 000055B8CEC877F0:16384
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL_read: -1
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 SSL_get_error: 2
2023/01/28 14:23:21 [debug] 2950336#2950336: *1192 event timer add: 11: 30000:8285240369

对比可见，正常日志中出现了proxy connection handler，说明Nginx触发了代理连接建立流程；而异常日志中没有该环节，停留在SSL读取等待状态。

原因解释 #

ssl_preread的设计场景是四层透明SSL转发：Nginx不终止SSL连接，仅读取ClientHello中的SNI信息，然后将原始加密流量转发到对应上游。这种模式下，Nginx不需要配置SSL证书，因为它不参与SSL握手。

但你的场景是七层SSL终止后转发：Nginx自身作为SSL服务器（配置了证书、listen ssl），先完成与客户端的SSL握手，解密流量后再转发到上游TCP服务器。此时开启ssl_preread on会导致逻辑冲突：

1. ssl_preread会尝试在SSL握手前读取ClientHello，但Nginx已经因为listen ssl触发了SSL握手流程，两者的处理逻辑相互干扰。
2. 开启ssl_preread后，Nginx会进入“预读取SSL数据”的模式，不会触发`