Kubernetes中Ignite出现Failed to wait for recovery descriptor reservation错误咨询

错误含义解析 #

这个错误属于Ignite TCP通信模块（TcpCommunicationSpi）的会话恢复机制范畴：

• Ignite节点间通过NIO会话通信，当连接意外断开时，会触发会话恢复流程，用GridNioRecoveryDescriptor跟踪未确认消息计数、重发状态、连接状态等关键信息。
• "Failed to wait for recovery descriptor reservation" 表示：在尝试恢复会话时，Ignite无法在超时时间内获取到恢复描述符的预留锁，或者等待对端节点确认预留状态超时。
• 从日志里的duration=5001ms可以看出，刚好超出了Ignite默认的5秒恢复等待超时时间，这直接触发了该错误。

结合场景分析 #

你提到当时存在其他进程网络通信问题（高延迟、超时），这正是引发该错误的核心原因：

• K8s环境下的Pod间网络波动、延迟过高，会导致Ignite节点间的心跳和消息确认超时，触发会话断开与恢复流程。
• 恢复流程中，由于网络延迟，对端节点无法及时响应预留请求，最终导致本地节点等待超时，抛出该错误。

相关错误日志 #

12-Jun-2023 14:58:56.811 SEVERE [Timer-15] org.apache.ignite.logger.java.JavaLogger.error Failed to wait for recovery descriptor reservation [desc=GridNioRecoveryDescriptor [acked=455168, resendCnt=0, rcvCnt=455198, sentCnt=455199, reserved=true, lastAck=455168, nodeLeft=false, node=TcpDiscoveryNode [id=************, consistentId=******, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 127.0.0.1, ******], sockAddrs=HashSet [/[0:0:0:0:0:0:0:1%lo]:47500, /127.0.0.1:47500, foo-bar-ignite-1-1-0.foo-bar-ignite-1-1.*****.svc.cluster.local/[******]:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1686364720306, loc=false, ver=2.11.1#20211220-sha1:eae1147d, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=4187847680, bytesSent=3335905975, bytesRcvd0=0, bytesSent0=0, select=false, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=TcpCommunicationSpi, finished=true, heartbeatTs=1686365362565, hashCode=2136585786, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi%]]], writeBuf=java.nio.DirectByteBuffer[pos=5560 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=455168, resendCnt=0, rcvCnt=455198, sentCnt=455199, reserved=true, lastAck=455168, nodeLeft=false, node=TcpDiscoveryNode [id=************, consistentId=******, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 127.0.0.1, ******], sockAddrs=HashSet [/[0:0:0:0:0:0:0:1%lo]:47500, /127.0.0.1:47500, foo-bar-ignite-1-1-0.foo-bar-ignite-1-1.*********.svc.cluster.local/[fd00:85:4001:5:9a2a:7c61:5749:de64]:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1686364720306, loc=false, ver=2.11.1#20211220-sha1:eae1147d, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=455168, resendCnt=0, rcvCnt=455198, sentCnt=455199, reserved=true, lastAck=455168, nodeLeft=false, node=TcpDiscoveryNode [id=************, consistentId=******, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 127.0.0.1, ******], sockAddrs=HashSet [/[0:0:0:0:0:0:0:1%lo]:47500, /127.0.0.1:47500, foo-bar-ignite-1-1-0.foo-bar-ignite-1-1.c6f362c5-5e67-5b2a-a72f-*******.svc.cluster.local/[*******]:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1686364720306, loc=false, ver=2.11.1#20211220-sha1:eae1147d, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], closeSocket=true, outboundMessagesQueueSizeMetric=org.apache.ignite.internal.processors.metric.impl.LongAdderMetric@69a257d1, super=GridNioSessionImpl [locAddr=/[******]:41956, rmtAddr=foo-bar-ignite-1-1-0.foo-bar-ignite-1-1.*****.svc.cluster.local/[******]:47100, createTime=1686364720981, closeTime=1686365447813, bytesSent=3335905975, bytesRcvd=4187847680, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1686364720981, lastSndTime=1686365359902, lastRcvTime=1686365350069, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@747d999d, directMode=true], GridConnectionBytesVerifyFilter], accepted=false, markedForClose=true]]]

12-Jun-2023 14:58:56.811 INFO [Timer-15] org.apache.ignite.logger.java.JavaLogger.info TCP client created [client=null, node addrs=[foo-bar-ignite-1-1-0.foo-bar-ignite-1-1.*****.svc.cluster.local/[*****]:47100, /[0:0:0:0:0:0:0:1%lo]:47100, /127.0.0.1:47100], duration=5001ms]

内容的提问来源于stack exchange，提问作者RichardFeynman