Lucene索引1400万文档耗时约5分钟，如何实现性能优化？

I have a file with 14 million records, each containing 2 fields. I'm using the following configuration to index the documents:

private fun getIndexWriter(directory: String): IndexWriter = try {
    val indexWriterConfig = IndexWriterConfig(KeywordAnalyzer())
    indexWriterConfig.ramBufferSizeMB = 2048.0
    indexWriterConfig.openMode = IndexWriterConfig.OpenMode.CREATE_OR_APPEND
    IndexWriter(FSDirectory.open(Paths.get(directory)), indexWriterConfig)
} catch (exception: Exception) {
    logger.error("File missing: {} ", exception.message)
    throw Exception("File missing: ${exception.message}")
}

fun indexDocs(documents: List<Document>)= mutex.withLock {
    coroutineScope {
        val indexedDocs = getIndexWriter("directory_path").use { indexWriter ->
            try {
                documents.forEach { document ->
                    try {
                        indexWriter.addDocument(document)
                    } catch (exception: IOException) {
                        logger.error("Failed to add document: {} to passive index", document)
                    }
                }
                indexWriter.flush()
                indexWriter.commit()
            } catch (exception: IOException) {
                logger.error("Failed to commit")
            }
            indexWriter.docStats.numDocs
        }
    }
}

Currently, it takes about 5 minutes to complete indexing and start the application. I'd like to know if there are feasible optimization solutions to reduce indexing time. Thanks for your help!

Great question—indexing 14M records efficiently in Lucene definitely has room for tuning. Let's break down actionable optimizations tailored to your code and use case:

1. Batch Documents Instead of Adding One-by-One #

Right now you're calling addDocument for each individual record. Lucene is optimized for bulk operations, so switch to addDocuments (plural) which takes a collection. This cuts down on overhead from repeated method calls and internal processing:

// Replace the forEach loop with bulk addition
indexWriter.addDocuments(documents)

If memory is a concern, split your large list into smaller batches (e.g., 10k-100k documents per batch), but bulk adding is still far more efficient than single-document calls.

2. Skip Unnecessary Flush/Commit Calls #

Your code explicitly calls flush() followed by commit() after adding all documents. By default, Lucene automatically flushes when the ram buffer is full (your 2048MB setting is reasonable). Explicitly flushing before commit is redundant—commit already triggers a flush if needed. Remove the indexWriter.flush() line to save extra I/O operations.

Also, if this is a one-time full index build (not incremental appends), set OpenMode.CREATE instead of CREATE_OR_APPEND. CREATE skips checking for existing index files, which saves some startup overhead.

3. Tune IndexWriterConfig for Speed #

• Disable Compound File Format: For large indexes, disable the compound file format during indexing (you can enable it later if needed for read performance):

  indexWriterConfig.setUseCompoundFile(false)
  

• Adjust Merge Policy: Use a higher merge factor to reduce the number of segment merge operations. TieredMergePolicy is a good default to tweak:

  val mergePolicy = TieredMergePolicy()
  mergePolicy.mergeFactor = 30 // Default is 10; higher means fewer merges
  indexWriterConfig.mergePolicy = mergePolicy
  

• Simplify Deletion Policy: If you don't need point-in-time recovery or old index versions, use a policy that only keeps the latest commit:

  indexWriterConfig.indexDeletionPolicy = KeepOnlyLastCommitDeletionPolicy()
  

4. Optimize Document Creation #

Ensure your Document objects are built efficiently:

• Reuse Field instances where possible (Fields are reusable after being added to a Document) since you're using the same field names repeatedly.
• Stick to StringField instead of TextField (since you're using KeywordAnalyzer, StringField is indexed without tokenization and faster to process).

5. Parallelize Document Parsing (Not Indexing) #

Lucene's IndexWriter isn't thread-safe for concurrent addDocuments calls, but you can parallelize the document parsing/creation step. Split your input file into chunks, parse each chunk into Documents in parallel coroutines, then feed the batches sequentially to the IndexWriter. This lets you use CPU resources for parsing while waiting for I/O during indexing.

6. Boost I/O Performance #

• Use an SSD instead of an HDD—Lucene indexing is I/O heavy, and SSDs drastically reduce write latency.
• Exclude the index directory from real-time antivirus scanning or filesystem overhead if possible.
• Ensure the index directory is on a filesystem with high throughput.

7. Trim Unnecessary Operations #

• If you don't need an immediate count of indexed docs, fetch indexWriter.docStats.numDocs only after commit (a minor but easy win).
• If most documents index successfully, lower the log level for individual document failures from error to warn to reduce logging overhead.

Start with the bulk add and removing the flush call first—those are quick wins with immediate impact. Combining these optimizations should cut down your indexing time significantly.

内容的提问来源于stack exchange，提问作者Mohamed Niyaz