Eclipselink v2.7.10偶现ConcurrencyException问题求助：线程持有CacheKey锁未释放导致等待超时

大家好，我这边遇到一个棘手的Eclipselink并发问题，想请教下社区的大佬们。

最近几个月，我们在使用Eclipselink v2.7.10时偶尔会遇到ConcurrencyException，触发频率极低——几百个实例里每周才会在随机实例上出现几次。异常信息如下：

Caused by: org.eclipse.persistence.exceptions.ConcurrencyException: 
Exception Description: Wait was interrupted. 
Message: [Summary current cache key of thread Thread-2096 (ActiveMQ-client-global-threads) Stuck thread problem: unique tiny message number (98) 
 The Thread [Thread-2096 (ActiveMQ-client-global-threads)]  appears to be stuck (possible dead lock ongoing). 
 The thread is working in the context of (CacheKey) = (--- CacheKey  (ObjectNull. Most likely not yet in server session cache and in the process of being created.):  
 (primaryKey: 22,137,168,321) (object: null) (object hash code: 0) (cacheKeyClass: org.eclipse.persistence.internal.identitymaps.SoftCacheKey) (cacheKey hash code: 1194985224) 
 (current cache key owner/activeThread: Thread-1930 (ActiveMQ-client-global-threads)) (getNumberOfReaders: 0)  (concurrencyManagerId: 29,047,152) (concurrencyManagerCreationDate: 2024-12-03 09:00:08.616)  (totalNumberOfTimeCacheKeyAcquiredForReading:  0)  (totalNumberOfTimeCacheKeyReleasedForReading:  0)  (totalNumberOfTimeCacheKeyReleasedForReadingBlewUpExceptionDueToCacheKeyHavingReachedCounterZero:  0)  (depth: 2) ---) . 
 The thread has been stuck for: (40,004 ms) 
  Bellow we will describe the ActiveLocks, DeferredLocks and ReadLocks for this thread. Summary of active locks owned by thread Thread-2096 (ActiveMQ-client-global-threads) Listing of all ACTIVE Locks.
Thread Name: Thread-2096 (ActiveMQ-client-global-threads) 
0 Active locks.
Summary of deferred locks (could not be acquired and cause thread to wait for object building to complete) of thread Thread-2096 (ActiveMQ-client-global-threads) Listing of all DEFERRED Locks.
Thread Name: Thread-2096 (ActiveMQ-client-global-threads) 
1 Deferred locks.
Deferred lock nr: 0 , Deferred cache key: --- CacheKey  (ObjectNull. Most likely not yet in server session cache and in the process of being created.):  (primaryKey: 22,137,168,321) (object: null) (object hash code: 0) (cacheKeyClass: com.company.technical.server.persistence.internal.cache.MonitoredSoftCacheKey) (cacheKey hash code: 1194985224) (current cache key owner/activeThread: Thread-1930 (ActiveMQ-client-global-threads)) (getNumberOfReaders: 0)  (concurrencyManagerId: 29,047,152) (concurrencyManagerCreationDate: 2024-12-03 09:00:08.616)  (totalNumberOfTimeCacheKeyAcquiredForReading:  0)  (totalNumberOfTimeCacheKeyReleasedForReading:  0)  (totalNumberOfTimeCacheKeyReleasedForReadingBlewUpExceptionDueToCacheKeyHavingReachedCounterZero:  0)  (depth: 2) ---
Summary of read locks acquired by thread Thread-2096 (ActiveMQ-client-global-threads) Listing of all READ Locks. Step 001 - sparse summary loop over all read locks acquired:
Thread Name: Thread-2096 (ActiveMQ-client-global-threads) 
0 read locks. The lockManager for this thread is null. 
]
    at deployment.com.company.external//org.eclipse.persistence.exceptions.ConcurrencyException.waitWasInterrupted(ConcurrencyException.java:110)
    at deployment.com.company.external//org.eclipse.persistence.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:668)
    at deployment.com.company.external//org.eclipse.persistence.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:472)
    at deployment.com.company.external//org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5365)
    at deployment.com.company.external//org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:1002)
    at deployment.com.company.external//org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInternal(ObjectBuilder.java:774)
    at deployment.com.company.external//org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:727)
    at deployment.com.company.external//org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:705)
    at deployment.com.company.external//org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:861)
    at deployment.com.company.external//org.eclipse.persistence.queries.Cursor.buildAndRegisterObject(Cursor.java:312)
    at deployment.com.company.external//org.eclipse.persistence.queries.ScrollableCursor.retrieveNextObject(ScrollableCursor.java:590)
    at deployment.com.company.external//org.eclipse.persistence.queries.ScrollableCursor.loadNext(ScrollableCursor.java:413)
    at deployment.com.company.external//org.eclipse.persistence.queries.ScrollableCursor.hasNext(ScrollableCursor.java:279)
    at deployment.com.company.external//org.eclipse.persistence.queries.ScrollableCursor.hasMoreElements(ScrollableCursor.java:268)

从异常信息分析，共享缓存中的一个CacheKey被某个线程锁定后再也没有释放，其他线程尝试获取或释放延迟锁时，等待40秒超时后就抛出了这个异常。但奇怪的是，持有锁的线程明明已经正常结束了上下文，没有抛出任何异常。目前我们的临时解决方案是要么失效共享缓存，要么重启服务器实例，但本地环境一直没法复现这个问题。

我们还发现这个问题和实例负载，以及某个实体启用隔离缓存高度相关：之前回滚了隔离缓存的启用后，问题消失了好几个月；重新启用后，问题又出现了。因为隔离缓存能帮我们避免竞争问题，所以我们不想放弃它，于是自己做了一个临时修复方案：在处理流程结束时，检查当前线程是否持有未释放的活跃锁，如果有的话就显式释放：

Thread currentThread = Thread.currentThread();
DeferredLockManager deferredLockManager = ConcurrencyManager.getDeferredLockManager(currentThread);
if (deferredLockManager != null) {
    // Release active locks on the thread.
    if (!deferredLockManager.getActiveLocks().isEmpty()) {
        LOGGER.warn("DeferredLockManager has active locks : {}", getDetail(deferredLockManager));

        deferredLockManager.releaseActiveLocksOnThread();
    }
    // Remove the thread entry
    ConcurrencyManager.removeDeferredLockManager(currentThread);
}

这个临时修复目前效果不错，但还是会偶尔出现（每周最多几次，随机实例）。我们在社区里也找到几个类似的问题讨论，但还没找到根本解决办法。

希望有遇到过类似问题或者对Eclipselink缓存机制熟悉的朋友能给点思路，非常感谢！

备注：内容来源于stack exchange，提问作者suenda