Solr 9.2.1索引极慢：仅单线程运行其余线程阻塞，求优化方案

问题诊断 #

从线程栈分析，所有阻塞线程均卡在ReentrantReadWriteLock$FairSync的等待逻辑上，具体是执行Add操作时尝试获取VersionInfo的读锁失败。唯一处于运行状态的线程正在执行DeleteByQuery操作，该操作触发了一系列需持有全局锁的流程（如更新日志刷新、搜索器重建、索引文件删除），导致所有Add线程无法获取锁，完全阻塞。

阻塞线程栈详情 #

"qtp1155757579-16" #16 prio=5 os_prio=0 cpu=1070.17ms elapsed=337.00s tid=0x00007f915acc4000 nid=0xc716e waiting on condition  [0x00007f840985a000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.17/Native Method)
    - parking to wait for  <0x00007f8e1249aa60> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.17/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.17/AbstractQueuedSynchronizer.java:885)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11.0.17/AbstractQueuedSynchronizer.java:1009)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11.0.17/AbstractQueuedSynchronizer.java:1324)
    at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@11.0.17/ReentrantReadWriteLock.java:738)
    at org.apache.solr.update.VersionInfo.lockForUpdate(VersionInfo.java:127)
    at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:351)
    at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:236)
    at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
    at com.ptc.solr.update.processor.PTCUpdateRequestProcessor.processAdd(PTCUpdateRequestProcessor.java:194)
    at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:123)
    at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:342)
    at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:286)
    at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:338)
    at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283)
    at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:236)
    at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:303)
    at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283)
    at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:193)
    at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:126)

运行线程栈详情 #

"qtp1155757579-25" #25 prio=5 os_prio=0 cpu=3635.83ms elapsed=336.95s tid=0x00007f915acd3800 nid=0xc7178 runnable  [0x00007f8409611000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.fs.UnixNativeDispatcher.unlink0(java.base@11.0.17/Native Method)
    at sun.nio.fs.UnixNativeDispatcher.unlink(java.base@11.0.17/UnixNativeDispatcher.java:156)
    at sun.nio.fs.UnixFileSystemProvider.implDelete(java.base@11.0.17/UnixFileSystemProvider.java:236)
    at sun.nio.fs.AbstractFileSystemProvider.delete(java.base@11.0.17/AbstractFileSystemProvider.java:105)
    at java.nio.file.Files.delete(java.base@11.0.17/Files.java:1142)
    at org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:344)
    at org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:309)
    at org.apache.lucene.store.NRTCachingDirectory.deleteFile(NRTCachingDirectory.java:128)
    - locked <0x00007f8e11678d10> (a org.apache.lucene.store.NRTCachingDirectory)
    at org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:37)
    at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:763)
    at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:757)
    at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:655)
    at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:686)
    at org.apache.lucene.index.IndexWriter.decRefDeleter(IndexWriter.java:5851)
    - locked <0x00007f8e1180b968> (a org.apache.solr.update.SolrIndexWriter)
    at org.apache.lucene.index.StandardDirectoryReader.lambda$doClose$1(StandardDirectoryReader.java:466)
    at org.apache.lucene.index.StandardDirectoryReader$$Lambda$910/0x00007f83f55bdc58.close(Unknown Source)
    at org.apache.lucene.index.StandardDirectoryReader.doClose(StandardDirectoryReader.java:482)
    at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:265)
    at org.apache.lucene.index.IndexReader.close(IndexReader.java:411)
    - locked <0x00007f90153053a0> (a org.apache.lucene.index.StandardDirectoryReader)
    at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:687)
    at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381)
    at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:366)
    at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:246)
    at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2329)
    at org.apache.solr.update.UpdateLog.openRealtimeSearcher(UpdateLog.java:745)
    - locked <0x00007f8e11df7a20> (a org.apache.solr.update.UpdateLog)
    at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(DirectUpdateHandler2.java:590)
    - locked <0x00007f8e0d55a1a8> (a java.lang.Object)
    at org.apache.solr.update.processor.RunUpdateProcessorFactory$RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:86)
    at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:59)
    at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:276)
    at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDeleteByQuery(DistributedUpdateProcessor.java:1011)
    at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDeleteByQuery(DistributedUpdateProcessor.java:980)
    at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:938)
    at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:913)
    at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:830)
    at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:59)
    at org.apache.solr.handler.loader.JavabinLoader.delete(JavabinLoader.java:230)
    at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:140)
    at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:74)

优化建议 #

1. 分离DeleteByQuery与批量索引任务 #

• 批量索引期间暂停所有DeleteByQuery操作，待全量索引完成后再执行数据清理。
• 若必须执行DeleteByQuery，拆分小范围查询条件（如按时间、ID分片），减少单次操作的锁持有时间，避免长时间阻塞Add线程。

2. 调整并发更新与锁配置 #

• 检查solrconfig.xml中updateHandler的concurrentUpdates参数，设置为与CPU核心数匹配的并发值。
• 修改VersionInfo的锁实现，将公平锁改为非公平锁（需调整代码或通过反射修改），公平锁会严格按线程排队顺序分配锁，非公平锁可提升并发吞吐量，需注意线程饥饿风险。

3. 减少实时搜索器刷新频率 #

• 批量索引期间关闭autoSoftCommit，或大幅延长其间隔（如设置为30分钟以上），避免频繁触发搜索器重建。
• 禁用实时GET功能（设置updateHandler的enableRealtimeGet="false"），减少UpdateLog.openRealtimeSearcher的调用次数。

4. 优化批量索引策略 #

• 使用SolrJ进行批量提交，每次提交1000-10000条文档，降低HTTP请求次数和锁竞争频率。
• 增大updateHandler的bufferSize参数，提升内存缓存文档数量，减少磁盘写入次数。
• 采用ConcurrentUpdateSolrClient替代普通HttpSolrClient，在客户端侧实现并发提交，同时控制并发数避免服务端过载。

5. 分布式集群优化 #

• 若使用SolrCloud，将5000万数据均匀分散到多个分片，每个分片由独立节点处理，降低单个节点的锁竞争压力。
• 开启分片并行写入，确保文档路由正确的前提下，允许多个线程同时写入同一分片。

6. 硬件与IO优化 #

• 将索引存储迁移至SSD磁盘，提升文件删除、写入的IO性能，缩短锁持有时间。
• 调整Linux内核参数（如vm.dirty_ratio、vm.dirty_background_ratio）优化磁盘写入效率。

内容的提问来源于stack exchange，提问作者Cliff Tian