K8s环境Ignite客户端连接时出现Discovery worker阻塞错误

故障现象 #

在Kubernetes环境部署Ignite 2.11.1版本集群，客户端节点尝试加入集群时触发tcp-disco-msg-worker线程阻塞异常。

• 相同配置在其他部署环境可长期稳定运行
• 故障环境与正常环境的唯一配置差异：故障环境新增了Kubernetes资源限制（resource limits）规则

报错日志 #

服务端侧报错 #

[20:07:24,201][SEVERE][tcp-disco-msg-worker-[crd]-#2-#48][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [workerName=db-checkpoint-thread, threadName=db-checkpoint-thread-#78, blockedFor=4562s]
[20:07:24] Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=db-checkpoint-thread, igniteInstanceName=null, finished=false, heartbeatTs=1657047082073]]]
[20:07:27,073][SEVERE][tcp-disco-msg-worker-[crd]-#2-#48][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [workerName=sys-stripe-0, threadName=sys-stripe-0-#1, blockedFor=4072s]

Thread [name="qtp2015455415-61", id=61, state=TIMED_WAITING, blockCnt=1, waitCnt=702]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78322a4, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:973)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1023)
        at java.lang.Thread.run(Thread.java:748)
 
Thread [name="qtp2015455415-60", id=60, state=RUNNABLE, blockCnt=3, waitCnt=679]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@7a7b4390
        - locked java.util.Collections$UnmodifiableSet@220c68db
        - locked sun.nio.ch.EPollSelectorImpl@4f569077
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:183)
        at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:190)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
        at java.lang.Thread.run(Thread.java:748)
 
Thread [name="qtp2015455415-59", id=59, state=TIMED_WAITING, blockCnt=1, waitCnt=684]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78322a4, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:973)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1023)
        at java.lang.Thread.run(Thread.java:748)
 
Thread [name="qtp2015455415-58", id=58, state=TIMED_WAITING, blockCnt=1, waitCnt=688]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78322a4, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:973)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1023)
        at java.lang.Thread.run(Thread.java:748)

客户端侧报错 #

06-Jul-2022 15:16:04.772 INFO [exchange-worker-#45%igniteClientInstance%] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [java.lang.OutOfMemoryError]. The
following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.lang.OutOfMemoryError]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1427)
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1415)
                at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1254)
                at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1215)
                at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3219)
                at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
                at java.base/java.lang.Thread.run(Thread.java:833)
06-Jul-2022 15:16:43.252 INFO [page-lock-tracker-timeout] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.apache.ignite.internal.processors.cache.persi
stence.diagnostic.pagelocktracker.SharedPageLockTracker$State]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker$State]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1427)
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1415)
                at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1254)
                at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1215)
                at org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker.getThreadOperationState(SharedPageLockTracker.java:285)
                at org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker.hangThreads(SharedPageLockTracker.java:301)
                at org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker.access$200(SharedPageLockTracker.java:53)
                at org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker$TimeOutWorker.iteration(SharedPageLockTracker.java:359)
                at org.apache.ignite.internal.util.worker.CycleThread.run(CycleThread.java:49)
06-Jul-2022 15:25:42.469 INFO [tcp-comm-worker-#1%igniteClientInstance%-#28%igniteClientInstance%] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [java.lan
g.OutOfMemoryError]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.lang.OutOfMemoryError]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1427)
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1415)
                at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1254)
                at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1215)
                at org.apache.ignite.spi.communication.tcp.internal.CommunicationWorker.body(CommunicationWorker.java:194)
                at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
                at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$6.body(TcpCommuicationSpi.java:928)
                at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

根因判定 #

故障直接触发原因是新增的Kubernetes资源限制配置不合理，结合日志细节可定位到两个核心问题：

1. 资源配额不足导致核心线程长时间停顿
   • 服务端日志显示db-checkpoint-thread（持久化检查点线程）、sys-stripe-0（集群核心工作线程）阻塞时长最高达4562秒，这类线程是Ignite运行的关键路径，对调度延迟容忍度极低。
   • 客户端日志反复出现无法加载java.lang.OutOfMemoryError类的报错，说明客户端所在容器已经触发内存不足，JVM抛出OOM时甚至无法完成类加载，Web应用实例被强制停止，Ignite的通信、分区交换线程全部异常退出。
   • 绝大多数同类场景都是两个配置错误导致：一是CPU limit设置过低触发K8s CFS CPU节流，线程拿不到时间片被判定为阻塞；二是内存limit只计算了JVM堆内存，没有算上Ignite占用的大量堆外页缓存、网络缓冲区内存，导致进程内存触限触发频繁Full GC或者OOM。
2. Ignite故障处理器配置错误
   服务端日志显示SYSTEM_WORKER_BLOCKED、SYSTEM_CRITICAL_OPERATION_TIMEOUT两类严重故障被加入了忽略列表，核心线程阻塞后节点既不退出也不重启，一直处于僵死状态，最终导致整个集群发现链路卡住，无法处理客户端加入请求。

修复步骤 #

• 调整Kubernetes资源限制配置
  • CPU配置：Ignite服务端、客户端节点的CPU request和limit设置为相同值，避免CFS节流，单服务端节点CPU配额不低于2核，按每4GB堆内存配1核CPU的比例扩容。
  • 内存配置：内存limit总值 = JVM堆内存最大值（-Xmx参数值） + Ignite堆外页缓存大小 + 1~2GB系统预留内存，禁止将内存limit设置为和JVM堆内存相等的值。
• 修正Ignite故障处理器配置
  将SYSTEM_WORKER_BLOCKED、SYSTEM_CRITICAL_OPERATION_TIMEOUT从忽略故障类型列表中移除，保证核心线程阻塞时节点可以自动退出重启，避免僵死节点影响集群可用性。
• 验证修复效果
  重启集群后观察服务端日志，确认无系统线程阻塞的SEVERE级别报错，再接入客户端验证连通性；可通过容器监控确认CPU节流率为0、内存使用率不超过limit的80%。

内容的提问来源于stack exchange，提问作者RichardFeynman