基于Lucene构建大规模用户文档nxn相似度矩阵的技术问询

Alright, let's tackle this problem step by step—building an efficient n×n similarity matrix with Lucene, especially scaling up to 100k+ users. Here's how I'd approach it:

First, ditching the single concatenated string for each user is key to efficiency and accuracy:

• Create a dedicated Lucene Document for each user, with a multi-valued field (e.g., named content) where each entry corresponds to one of the user's documents. This keeps document boundaries intact while letting Lucene aggregate all content for a user.
• Add a non-analyzed user_id field (use StringField) to every document—this is critical for linking search results back to specific users later.

Lucene offers built-in tools, but choose based on your use case:

• BM25: Lucene's default similarity metric, great for ranking relevance based on term frequency and inverse document frequency. Works well for most content types.
• Cosine Similarity: Ideal if you want to measure overlap in content themes. Use Lucene's VectorSimilarityQuery or precompute term vectors to calculate this efficiently.
• Jaccard Similarity: Useful for focusing on unique term overlap, though less efficient for long documents.

Calculating a 100k×100k matrix directly means 10^10 operations—completely impractical. Instead:

• Leverage MoreLikeThis for Targeted Similarity Queries:
  • After indexing all users, use Lucene's MoreLikeThis component to generate a similarity query for each user's content. Run this query against the index to fetch only the most similar users (e.g., top 100), instead of computing every pair.
  • Tune MoreLikeThis parameters (min/max term frequency, max similar docs returned) to reduce unnecessary computations.
• Precompute Term Vectors:
  • Enable term vector storage during indexing (FieldType.setStoreTermVectors(true)). This lets you pull sparse word vectors for each user directly from the index. Use a distributed framework like Spark to batch-calculate similarities across all vectors—far faster than single-node processing.
• Distributed Indexing with SolrCloud/Elasticsearch:
  • For 100k+ users, a single Lucene node may hit limits. Use a distributed Lucene-based system to shard the index, then parallelize similarity calculations across nodes.

• Diagonal Elements: The diagonal (user vs. self) should always be 1.0—no need to compute this, just pre-fill the matrix with these values.
• Special Character Handling: Use Lucene's PatternReplaceFilter to normalize special characters (like &, /, +) before indexing. This prevents them from breaking tokenization and skewing similarity scores.
• Performance Tuning:
  • Use MMapDirectory for large indexes—it’s faster for random access than other directory types.
  • Disable unnecessary field storage (only keep user_id and content with term vectors) to shrink index size and speed up queries.
  • Cache frequent similarity queries if multiple users have overlapping content patterns.

// Initialize index reader and searcher
IndexReader reader = DirectoryReader.open(FSDirectory.open(Paths.get("/path/to/index")));
IndexSearcher searcher = new IndexSearcher(reader);

// Configure MoreLikeThis
MoreLikeThis mlt = new MoreLikeThis(reader);
mlt.setFieldNames(new String[]{"content"});
mlt.setMinTermFreq(1); // Include rare terms if needed
mlt.setMaxQueryTerms(50); // Limit query complexity

// Initialize your similarity matrix (use a sparse structure to save space)
Map<String, Map<String, Float>> similarityMatrix = new HashMap<>();

// Iterate through all user documents
for (int docId = 0; docId < reader.maxDoc(); docId++) {
    if (reader.isDeleted(docId)) continue;
    Document doc = reader.document(docId);
    String userId = doc.get("user_id");
    
    // Generate query for similar users
    Query similarityQuery = mlt.like(docId);
    TopDocs topSimilar = searcher.search(similarityQuery, 100); // Get top 100 matches
    
    // Populate matrix
    similarityMatrix.putIfAbsent(userId, new HashMap<>());
    similarityMatrix.get(userId).put(userId, 1.0f); // Set diagonal to 1.0
    
    for (ScoreDoc scoreDoc : topSimilar.scoreDocs) {
        String similarUserId = reader.document(scoreDoc.doc).get("user_id");
        similarityMatrix.get(userId).put(similarUserId, scoreDoc.score);
    }
}

// Cleanup resources
reader.close();

内容的提问来源于stack exchange，提问作者Boendal