Gensim工作线程卡住求助：2000万句文档嵌入并行训练遇阻

Hey there, let's dig into that worker thread stuck issue you're hitting while training document embeddings with Gensim on 20 million sentences. Looking at your corpus iterator code, I spot a few critical issues and have actionable fixes to get your training back on track.

First: Fix the Critical Iterator Bugs #

Your current read_corpus class has two major issues that are almost certainly causing the thread block:

1. Undefined variable in loop condition: You're using num_notes < n but n isn't defined in the __iter__ scope—you meant self.n.
2. Missing yield statement: The iterator doesn't return any data! Gensim's workers will hang indefinitely waiting for corpus data that never comes.

Here's a corrected, optimized version of your iterator:

class read_corpus(object):
    def __init__(self, fname, max_notes):
        self.fname = fname
        self.max_notes = max_notes  # Renamed for clarity

    def __iter__(self):
        num_notes = 0
        with open(self.fname, 'r') as f:
            # Use `for line in f` for efficient, safe line-by-line reading
            for line in f:
                if num_notes >= self.max_notes:
                    break
                # Split only on the FIRST tab (avoids issues if sentences contain tabs)
                sentence_id, sentence = line.split('\t', 1)
                # Safely strip newline characters (rstrip avoids trimming other whitespace)
                sentence_tokens = sentence.rstrip('\n').split(' ')
                # Add your preprocessing logic here (e.g., stopword removal, lowercasing)
                yield sentence_tokens  # Yield the processed tokens to Gensim
                num_notes += 1

Next: Optimize for Parallel Training #

Even with a fixed iterator, parallel training can get stuck if you're not configuring Gensim properly:

• Set workers to a reasonable number: Don't set it higher than your CPU core count (or 1.5x at most). Too many workers cause excessive context switching and resource contention. Example:

  import os
  model = Doc2Vec(workers=min(8, os.cpu_count()))  # Adjust based on your hardware
  

• Monitor memory usage: Training on 20M sentences requires significant RAM. If your system starts swapping to disk, workers will freeze. Try reducing vector_size, window, or min_count if memory is tight.
• Preprocess data upfront: If your preprocessing (e.g., complex tokenization) is slow, preprocess all sentences once and save them to a binary format (like numpy arrays or pickle). This avoids repeated processing during training and keeps workers fed with data quickly.

Debugging Tips to Pinpoint Issues #

• Test with a small dataset: Train on 1k-10k sentences first. If the issue persists, it's likely a code bug rather than a scaling problem.
• Enable Gensim logging: This will show you exactly where training hangs. Add this at the start of your script:

  import logging
  logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
  

• Check for malformed data: A single line with missing tabs or corrupted text could crash a worker thread. Add error handling in your iterator (e.g., try/except blocks around split) to skip bad lines and log errors.

内容的提问来源于stack exchange，提问作者bbrodrigues