启用Write Through的Ignite缓存因PME引发死锁问题咨询

我们在JVM服务器模式运行Ignite时，遇到以下场景触发的死锁：

• 创建启用Write Through的缓存
• 在CacheWriter.write()实现中：
  • 等待1秒触发后续操作
  • 尝试读取另一个缓存
• 在write()执行期间，独立线程触发新缓存创建
• 最终触发死锁，原因链：
  • cache.put()已获取读锁
  • 缓存创建触发Partition Map Exchange（PME）启动
  • PME尝试获取全部16个锁，但因已有读锁阻塞等待
  • cache.get()因等待当前PME完成无法结束

生产环境已出现该问题，上述为复现步骤。Write Through仅读取缓存，缓存创建完全在独立线程执行。

疑问解答 #

1. 为何PME未获取全部锁就阻塞cache.get()？ #

Ignite的PME流程启动后会立即进入全局屏障阶段：所有新的缓存操作（包括get）都会先检查是否有正在进行的PME，若有则直接等待PME完成，不会等到PME拿到所有锁才开始阻塞。这种设计是为了保证数据一致性——PME涉及集群拓扑与分区映射的变更，必须确保所有操作要么在PME前完成，要么在PME后执行，避免出现中间状态的分区视图不一致，优先级高于性能优化。

2. 为何PME停止所有操作，而非仅相关缓存/缓存组？ #

Ignite的PME本质是全局拓扑变更流程，哪怕只是创建单个缓存，也会触发集群节点重新计算全局分区映射表（比如新缓存加入后，缓存组的分区分布可能调整，所有节点需要同步更新分区视图）。早期2.x版本的Ignite中，PME是全局阻塞的，所有缓存操作都会被挂起；后续版本虽针对缓存组做了优化，但默认情况下创建新缓存仍会触发全局PME，因为集群需要统一更新所有节点的分区元数据。

3. 死锁解决方案 #

针对该场景，有以下可行解决方向：

方案1：调整Write Through实现，避免在write()内执行缓存操作

CacheWriter.write()是在缓存操作的锁上下文（持有读锁）内执行的，此时触发需要等待PME的操作必然引发死锁。可以将缓存读取逻辑移出write()：

• 在write()中记录需要读取的数据，提交到独立线程池异步执行
• 使用Ignite异步API（cache.getAsync()），但禁止在当前锁上下文内阻塞等待异步结果

方案2：提前初始化所有缓存，避免运行时动态创建

如果业务允许，在Ignite集群启动时就创建好所有需要的缓存，彻底规避运行时创建缓存触发PME的场景，这是最彻底的解决方式。

方案3：配置PME并发锁策略（Ignite 2.9+版本适用）

Ignite 2.9及以上版本支持缓存组级别的PME优化与锁公平性配置：

• 设置IGNITE_DISTRIBUTED_LOCK_FAIRNESS=true，调整锁的获取顺序，降低死锁概率
• 为新创建的缓存指定独立缓存组，缩小PME的影响范围

方案4：使用withSkipStore()绕过Write Through（特定场景适用）

若部分写操作无需触发Write Through，可使用cache.withSkipStore().put()跳过write()逻辑，避免进入持有锁的代码路径。

线程栈信息 #

执行cache.put()和Write Through的线程 #

"main" #1 prio=5 os_prio=0 tid=0x0000000003505000 nid=0x43f4 waiting on condition [0x000000000334b000]
   java.lang.Thread.State: WAITING (parking)
               at sun.misc.Unsafe.park(Native Method)
               at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
               at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
               at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
               at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4870)
               at org.apache.ignite.internal.processors.cache.GridCacheAdapter.repairableGet(GridCacheAdapter.java:4830)
               at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1463)
               at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:1128)
               at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:688)
               at ReadWriteThroughInterceptor.write(ReadWriteThroughInterceptor.java:70)
               at org.apache.ignite.internal.processors.cache.GridCacheLoaderWriterStore.write(GridCacheLoaderWriterStore.java:121)
               at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.put(GridCacheStoreManagerAdapter.java:585)
               at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.update(GridCacheMapEntry.java:6468)
               at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:6239)
               at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:5923)
               at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:4041)
               at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.access$5700(BPlusTree.java:3935)
               at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:2039)
               at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1923)
               at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1734)
               at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1717)
               at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:441)
               at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2327)
               at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2553)
               at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:2016)
               at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1833)
               at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1692)
               at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:300)
               at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:481)
               at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:441)
               at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:249)
               at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1147)
               at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:615)
               at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2571)
               at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2550)
               at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1337)
               at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:868)
               at com.eqtechnologic.eqube.cache.tests.readerwriter.WriteReadThroughTest.writeToCache(WriteReadThroughTest.java:54)
               at com.eqtechnologic.eqube.cache.tests.readerwriter.WriteReadThroughTest.lambda$runTest$0(WriteReadThroughTest.java:26)
               at com.eqtechnologic.eqube.cache.tests.readerwriter.WriteReadThroughTest$$Lambda$1095/2028767654.execute(Unknown Source)
               at org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:50)
               at org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:37)
               at org.junit.jupiter.api.Assertions.assertDoesNotThrow(Assertions.java:3060)
               at WriteReadThroughTest.runTest(WriteReadThroughTest.java:24)

PME线程等待锁 #

"exchange-worker-#39" #56 prio=5 os_prio=0 tid=0x0000000022b91800 nid=0x450 waiting on condition [0x000000002866e000]
               java.lang.Thread.State: WAITING (parking)
                              at sun.misc.Unsafe.park(Native Method)
                              - parking to wait for  <0x000000076e73b428> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
                              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
                              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
                              at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897)
                              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222)
                              at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lockInterruptibly(ReentrantReadWriteLock.java:998)
                              at org.apache.ignite.internal.util.StripedCompositeReadWriteLock$WriteLock.lock0(StripedCompositeReadWriteLock.java:192)
                              at org.apache.ignite.internal.util.StripedCompositeReadWriteLock$WriteLock.lockInterruptibly(StripedCompositeReadWriteLock.java:172)
                              at org.apache.ignite.internal.util.IgniteUtils.writeLock(IgniteUtils.java:10487)
                              at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.updateTopologyVersion(GridDhtPartitionTopologyImpl.java:272)
                              at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.updateTopologies(GridDhtPartitionsExchangeFuture.java:1269)
                              at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:1028)
                              at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3370)
                              at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3197)
                              at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
                              at java.lang.Thread.run(Thread.java:748)

内容的提问来源于Stack Exchange，提问作者Atul Dhatrak