Gensim Doc2Vec most_similar方法异常：仅返回前10个标签文档相似度

Hey there, let’s break down why your Doc2Vec most_similar() method is only returning results for tags 0-9—this is a common gotcha tied to how you’re tagging your documents, and it’s fixable!

First, the Likely Root Cause: Duplicate/Non-Unique Tags #

Doc2Vec’s most_similar() returns results based on the unique document vectors stored in the model. If you’re only seeing 10 results, it almost always means your training data only has 10 unique tags across all documents—even if you have way more documents total.

For example, if you accidentally set tags using a modulo 10 operation (like tags=[i%10] when looping through documents), every document gets tagged 0-9 in a cycle. The model will only learn one vector per unique tag, so you end up with just 10 document vectors total.

Step-by-Step Debugging #

Let’s verify this first:

1. Check your TaggedDocument tags
   Print the tags of the first 20 documents to see if they repeat:

   print([doc.tags for doc in documents[:20]])
   

   If you see [0], [1], ..., [9], [0], [1], ... instead of unique values like [0], [1], ..., [19], that’s your problem.

2. Verify model docvecs count
   After training, check how many unique document vectors the model has:

   print(len(model.docvecs))
   

   If this number is 10 instead of your total document count, confirm your tag generation logic is broken.

Fix: Assign Unique Tags to Every Document #

You need to ensure each document gets a unique, non-repeating tag. The easiest way is to use the document’s index (as an integer or string):

Correct Tagging Example #

from gensim.models.doc2vec import TaggedDocument

# Assuming `corpus` is your list of text documents
documents = []
for idx, text in enumerate(corpus):
    # Split text into tokens (adjust tokenization as needed)
    tokens = text.split()
    # Assign a unique tag (use integer idx or string like f"doc_{idx}")
    documents.append(TaggedDocument(words=tokens, tags=[idx]))

Why This Works #

Doc2Vec creates a separate vector for each unique tag. By using unique tags, your model will learn a vector for every single document, and most_similar() will return results across all documents when you set topn to any valid number.

Post-Fix Verification #

After retraining with unique tags:

• len(model.docvecs) should match your total document count.
• Calling model.docvecs.most_similar(0, topn=20) (or any tag) will return the top 20 most similar documents from your full corpus.

Other Edge Cases to Rule Out #

If tags are unique but you still see limited results:

• Ensure you’re not accidentally filtering results elsewhere in your code.
• Check if you’re using dm=0 (PV-DBOW) with dbow_words=1—this shouldn’t affect document vectors, but double-check your training parameters (like epochs and vector_size) are sufficient for your dataset.

内容的提问来源于stack exchange，提问作者J. Collins