Apache Ignite 2.12部署Kubernetes多日后查询失败求助

问题背景 #

我们的应用基于Kubernetes部署Apache Ignite 2.12版本，采用二进制对象存储方案。初期运行正常，但运行数日后出现缓存无法查询的问题，重启Pod可临时恢复，但生产环境无法频繁执行该操作。Pod内的错误栈信息如下：

Thread [name="query-#1741%ignite-service%", id=2004, state=TIMED_WAITING, blockCnt=0, waitCnt=29]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3f30bea2, ownerName=null, ownerId=-1]
        at java.base@11.0.17/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.17/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.17/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11.0.17/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458)
        at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.17/java.lang.Thread.run(Thread.java:829)

Thread [name="Connection evictor", id=2021, state=TIMED_WAITING, blockCnt=0, waitCnt=123]
        at java.base@11.0.17/java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.base@11.0.17/java.lang.Thread.run(Thread.java:829)

Thread [name="sys-#1773%ignite-service%", id=2048, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4cd18e1b, ownerName=null, ownerId=-1]
        at java.base@11.0.17/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.17/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.17/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11.0.17/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458)
        at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.17/java.lang.Thread.run(Thread.java:829)

2022-11-15 04:22:20.783  WARN [app-service,,] 1 --- [otlist-service%] o.a.i.i.p.cache.CacheDiagnosticManager   : Page locks dump:

2022-11-15 04:22:29.834 ERROR [app-service,,] 1 --- [otlist-service%] o.apache.ignite.internal.util.typedef.G  : Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [workerName=disco-event-worker, threadName=disco-event-worker-#44%ignite-service%, blockedFor=834s]
2022-11-15 04:22:29.837  WARN [app-service,,] 1 --- [otlist-service%]                                          : Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=disco-event-worker, igniteInstanceName=ignite-service, finished=false, heartbeatTs=1668485315026]]]

org.apache.ignite.IgniteException: GridWorker [name=disco-event-worker, igniteInstanceName=ignite-service, finished=false, heartbeatTs=1668485315026]
    at java.base@11.0.17/jdk.internal.misc.Unsafe.park(Native Method) ~[na:na]
    at java.base@11.0.17/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323) ~[na:na]
    at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178) ~[ignite-core-2.12.0.jar:2.12.0]
    at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) ~[ignite-core-2.12.0.jar:2.12.0]
    at org.apache.ignite.internal.IgniteKernal.resetLostPartitions(IgniteKernal.java:3797) ~[ignite-core-2.12.0.jar:2.12.0]
    at com.app.dao.ignite.IgniteProvider.lambda$resetLostPartition$3(IgniteProvider.java:267) ~[app-core-2022.4.2-RC1.jar:2022.4.2-RC1]
    at com.app.dao.ignite.IgniteProvider$$Lambda$2152/0x0000000800eab440.accept(Unknown Source) ~[na:na]
    at java.base@11.0.17/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
    at com.app.dao.ignite.IgniteProvider.resetLostPartition(IgniteProvider.java:264) ~[app-core-2022.4.2-RC1.jar:2022.4.2-RC1]
    at com.app.dao.ignite.IgniteProvider.lambda$getOrStartIgniteNode$37431c07$1(IgniteProvider.java:103) ~[app-core-2022.4.2-RC1.jar:2022.4.2-RC1]
    at com.app.dao.ignite.IgniteProvider$$Lambda$1640/0x0000000800bce840.apply(Unknown Source) ~[na:na]
    at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager$UserListenerWrapper.onEvent(GridEventStorageManager.java:1492) ~[ignite-core-2.12.0.jar:2.12.0]
    at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:894) ~[ignite-core-2.12.0.jar:2.12.0]
    at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:879) ~[ignite-core-2.12.0.jar:2.12.0]
    at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record0(GridEventStorageManager.java:350) ~[ignite-core-2.12.0.jar:2.12.0]
    at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:313) ~[ignite-core-2.12.0.jar:2.12.0]
    at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:3074) ~[ignite-core-2.12.0.jar:2.12.0]
    at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:3291) ~[ignite-core-2.12.0.jar:2.12.0]
    at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:3094) ~[ignite-core-2.12.0.jar:2.12.0]
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125) ~[ignite-core-2.12.0.jar:2.12.0]
    at java.base@11.0.17/java.lang.Thread.run(Thread.java:829) ~[na:na]

问题分析 #

核心故障点在于系统关键线程disco-event-worker被阻塞，该线程负责处理集群Discovery事件（节点加入/离开、拓扑变更等），是Ignite集群的核心控制线程。

从错误栈可追踪到阻塞路径：

1. Discovery线程处理事件时，触发自定义监听器IgniteProvider的逻辑
2. 监听器中同步调用resetLostPartitions，并通过GridFutureAdapter.get()等待操作完成
3. 该Future在特定场景下无法正常完成，导致Discovery线程长期阻塞（长达834秒）
4. 系统线程阻塞引发集群元数据处理异常，最终导致缓存查询失效

解决方案 #

1. 修复自定义监听器的阻塞逻辑 #

Discovery线程不能被任何同步阻塞操作占用，需将resetLostPartitions的调用改为异步执行：

// 原同步阻塞代码
ignite.resetLostPartitions(cacheNames);

// 修改为异步执行，使用独立线程池
ExecutorService executor = Executors.newFixedThreadPool(2);
executor.submit(() -> {
    try {
        ignite.resetLostPartitions(cacheNames);
    } catch (Exception e) {
        // 处理异常日志
        log.error("Failed to reset lost partitions", e);
    }
});

2. 升级Ignite版本 #

Ignite 2.12存在多个与分区恢复、线程阻塞相关的已知BUG，升级到2.13及以上稳定版本，官方已修复此类场景下的Future等待死锁问题。

3. 调整Failure Handler配置 #

当前配置tryStop=false会忽略系统线程阻塞故障，改为tryStop=true可让节点在故障时自动重启，避免影响整个集群：

<!-- Ignite配置文件中修改Failure Handler -->
<property name="failureHandler">
    <bean class="org.apache.ignite.failure.StopNodeOrHaltFailureHandler">
        <property name="tryStop" value="true"/>
        <property name="timeout" value="60000"/>
    </bean>
</property>

4. 优化Kubernetes网络配置 #

K8s环境下的网络波动可能触发分区丢失事件，需确保集群网络稳定：

• 调整Ignite的failureDetectionTimeout参数，适配K8s网络延迟
• 配置K8s节点间的网络QoS，保证Ignite节点的通信带宽

内容的提问来源于stack exchange，提问作者Aditya Mavinasara