WSO2 MI 4.2.0多Pod部署时持续出现Socket连接重置错误排查

环境信息 #

• WSO2 MI 4.2.0
• JDK 17
• Rocky Linux 8.9（兼容RHEL）

问题现象 #

在K8s中通过wrapper部署WSO2 MI，单Pod实例运行无错误，但副本数增加至2时，日志频繁出现连接重置错误，每5秒重复两次：

INFO日志 #

[2024-02-07 10:23:27,878] ERROR {SourceHandler} I/O error: Connection reset java.net.SocketException: Connection reset
at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)
at org.apache.http.nio.reactor.ssl.SSLIOSession.receiveEncryptedData(SSLIOSession.java:482)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:544)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:840)

DEBUG日志 #

[2024-02-21 11:33:41,826] DEBUG {JsonReadOnlyStream} - <init>
[2024-02-21 11:33:41,832] DEBUG {JsonReaderDelegate} - #JsonReaderDelegate. Setting XMLStreamReader: org.apache.synapse.commons.staxon.core.json.JsonXMLStreamReader
[2024-02-21 11:33:41,832] DEBUG {JsonReaderDelegate} - #getLocalName. old=jsonObject, new=jsonObject
[2024-02-21 11:33:41,834] DEBUG {OMSourcedElementImpl} - forceExpand: changing prefix from  to 
[2024-02-21 11:33:41,834] DEBUG {JsonReaderDelegate} - #getName. old=loggerName, new=loggerName
[2024-02-21 11:33:41,834] DEBUG {JsonReaderDelegate} - #getLocalName. old=loggerName, new=loggerName
[2024-02-21 11:33:41,834] DEBUG {JsonReaderDelegate} - #getName. old=loggerName, new=loggerName
[2024-02-21 11:33:41,834] DEBUG {JsonReaderDelegate} - #getName. old=loggingLevel, new=loggingLevel
[2024-02-21 11:33:41,834] DEBUG {JsonReaderDelegate} - #getLocalName. old=loggingLevel, new=loggingLevel
[2024-02-21 11:33:41,834] DEBUG {JsonReaderDelegate} - #getName. old=loggingLevel, new=loggingLevel
[2024-02-21 11:33:41,834] DEBUG {JsonReaderDelegate} - #getName. old=jsonObject, new=jsonObject
[2024-02-21 11:33:41,834] DEBUG {AddressingHelper} - [MessageContext: logID=634fdab486ade390d77c61f5a397f27a11a51b39fbe16bfe] isReplyRedirected: ReplyTo is null. Returning false
[2024-02-21 11:33:41,834] DEBUG {MessageProcessorSelector} - Message format is: application/json; message formatter returned by AxisConfiguration: org.wso2.micro.integrator.core.json.JsonStreamFormatter@4c7b692a
[2024-02-21 11:33:41,835] DEBUG {BuilderUtil} - Input contentType (application/json)
[2024-02-21 11:33:41,835] DEBUG {BuilderUtil} - CharSetEncoding defaulted (UTF-8)
[2024-02-21 11:33:41,835] DEBUG {JsonStreamFormatter} - #writeTo. Wrote JSON payload to output stream. MessageID: urn:uuid:6f4c8123-2615-40bf-b526-29431a3987e5
[2024-02-21 11:33:41,835] DEBUG {LoggingNHttpServerConnection} - http-incoming-33080: Produce output
[2024-02-21 11:33:41,835] DEBUG {LoggingNHttpServerConnection} - http-incoming-33080: HTTP/1.1 200 OK
[2024-02-21 11:33:41,835] DEBUG {LoggingNHttpServerConnection} - http-incoming-33080: Consume input
[2024-02-21 11:33:41,836] DEBUG {LoggingNHttpServerConnection} - http-incoming-33080: Close connection
[2024-02-21 11:33:41,836] DEBUG {SourceHandler} - http-incoming-33080: Keep-Alive connection was closed:  Remote Address : /127.0.0.1:48336
[2024-02-21 11:33:41,836] DEBUG {SourceConnections} - Shutting down connection forcefully http-incoming-33080
[2024-02-21 11:33:41,836] DEBUG {LoggingNHttpServerConnection} - http-incoming-33080: Shutdown connection
[2024-02-21 11:33:42,942] DEBUG {SSLIOSession} - I/O session http-incoming-33081-1 10.255.18.41:8253&lt;-&gt;10.255.12.1:49604[ACTIVE][r:][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set timeout 180000
[2024-02-21 11:33:42,942] DEBUG {SSLIOSession} - I/O session http-incoming-33081-1 10.255.18.41:8253&lt;-&gt;10.255.12.1:49604[ACTIVE][r:][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set attribute CONNECTION_INFORMATION
[2024-02-21 11:33:42,943] ERROR {SourceHandler} - I/O error: Connection reset java.net.SocketException: Connection reset
    at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)
    at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.receiveEncryptedData(SSLIOSession.java:482)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:544)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
    at java.base/java.lang.Thread.run(Thread.java:840)

已尝试操作 #

• 为服务添加集群角色绑定，问题仍存在
• 在WSO2 MI 4.1.0 + Rocky Linux 8.8（RHEL 8.8）环境中未出现该问题
• 更换为Rocky Linux 8.8部署WSO2 MI 4.2.0，问题依旧

可能原因分析 #

结合环境差异和现象，核心排查方向集中在WSO2 MI 4.2.0的集群相关配置或底层依赖变化：

1. HTTP/NIO连接池与Keep-Alive配置冲突
   WSO2 MI 4.2.0可能调整了HTTP NIO的连接管理策略，多副本场景下K8s Service的负载均衡会将长连接分发到不同Pod，导致原Keep-Alive连接被非目标Pod重置。建议对比4.1.0的deployment.toml中[transport.http]下的keep_alive、max_connections等参数配置。

2. 集群通信配置缺失或冲突
   WSO2 MI 4.2.0可能新增了集群内部通信组件，多副本时未正确配置K8s环境的集群发现机制（如DNS成员发现），导致实例间误发起连接被对方拒绝重置。检查deployment.toml中[clustering]模块的配置，确保集群发现逻辑正确、通信端口无冲突。

3. 底层HTTP组件版本bug
   WSO2 MI 4.2.0可能升级了Apache HttpComponents等依赖库，新版本NIO reactor在多实例负载均衡场景下存在连接处理异常。对比4.1.0和4.2.0的依赖版本，尝试回退相关组件验证问题是否消失。

4. K8s Service或网络策略问题
   单副本正常但多副本异常，可能是Service的会话保持配置导致连接在实例间切换时被中断，或是网络策略限制了Pod间的某些端口通信。检查Service的sessionAffinity配置，以及集群内的网络策略规则。

5. JDK 17适配问题
   WSO2 MI 4.2.0对JDK 17的适配可能存在细节缺陷，多实例场景下的NIO Socket处理出现异常。尝试切换到JDK 11（4.1.0常用版本）验证是否解决问题。

内容的提问来源于stack exchange，提问作者SriniK