Jenkins JNLP remoting模式下Agent频繁Ping超时问题如何解决？

故障现象 #

• Jenkins Agent通过JNLP remoting协议连接Master节点
• 初始连接可稳定运行约20分钟，随后连续三次ping尝试均失败，Agent断开后重新连接成功，全程累计产生约30分钟的服务不可用时间

根因定位 #

从日志可以判断是ChannelPinger的ping请求超时触发的强制断连，默认Jenkins的ping超时时间为4分钟，连续三次失败后才会触发重连逻辑，常见触发原因如下：

• 中间网络设备（防火墙、负载均衡、NAT网关）的空闲连接超时配置小于Jenkins的ping间隔，长时间无业务流量的JNLP长连接被中间设备静默切断
• Agent或Master节点网络波动、丢包率过高，导致ping包无法在超时时间内完成往返
• Jenkins remoting组件版本过低，存在已知的ping线程死锁、超时类bug

修复方案 #

• 调整Agent启动参数，添加-pingInterval 30000 -pingTimeout 60000，将ping间隔改为30秒、超时时间改为60秒，避免连接被中间设备判定为空闲后切断
• 调整中间网络设备配置，将防火墙、NAT网关的TCP空闲连接超时阈值调整到5分钟以上，保留合法的JNLP长连接
• 升级Jenkins主程序、remoting组件、Agent版本到最新稳定版，修复已知的连接超时相关bug
• 配置Agent自动重连策略，启动参数添加-noReconnect false -reconnectWaitTime 10000，断开后10秒自动尝试重连，大幅减少服务不可用时长

原始故障日志 #

Aug 30, 2021 7:12:32 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 30, 2021 7:12:32 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkinsmain.cloud:9000
Aug 30, 2021 7:12:32 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Aug 30, 2021 7:12:32 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: xx:d9:de:d9:xx.xx.xx.xx:b3
Aug 30, 2021 7:12:32 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Aug 30, 2021 7:31:32 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel JNLP4-connect connection to jenkinsmain.cloud/xx.xx.101.4:9000.
java.util.concurrent.TimeoutException: Ping started at 1630351652299 hasn't completed by 1630351892300
        at hudson.remoting.PingThread.ping(PingThread.java:133)
        at hudson.remoting.PingThread.run(PingThread.java:89)

Aug 30, 2021 7:36:32 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel JNLP4-connect connection to jenkinsmain.cloud/xx.xx.101.4:9000.
java.util.concurrent.TimeoutException: Ping started at 1630351952301 hasn't completed by 1630352192302
        at hudson.remoting.PingThread.ping(PingThread.java:133)
        at hudson.remoting.PingThread.run(PingThread.java:89)

Aug 30, 2021 7:41:32 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel JNLP4-connect connection to jenkinsmain.cloud/xx.xx.101.4:9000.
java.util.concurrent.TimeoutException: Ping started at 1630352252300 hasn't completed by 1630352492301
        at hudson.remoting.PingThread.ping(PingThread.java:133)
        at hudson.remoting.PingThread.run(PingThread.java:89)

Aug 30, 2021 7:43:01 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Aug 30, 2021 7:43:11 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Performing onReconnect operation.
Aug 30, 2021 7:43:11 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: onReconnect operation failed.
Aug 30, 2021 7:43:11 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [https://jenkinsmain.cloud/]
Aug 30, 2021 7:43:11 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Aug 30, 2021 7:43:11 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: jenkinsmain.cloud
  Agent port:    9000
  Identity:      xx:d9:de:d9:xx.xx.xx.xx:b3
Aug 30, 2021 7:43:11 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 30, 2021 7:43:11 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkinsmain.cloud:9000
Aug 30, 2021 7:43:11 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Aug 30, 2021 7:43:11 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: xx:d9:de:d9:xx.xx.xx.xx:b3    
Aug 30, 2021 7:43:11 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

内容的提问来源于stack exchange，提问作者Ajit Bhaskaran