使用Gensim时遇TypeError：doc2bow需Unicode令牌数组而非单个字符串

Hey there! Let's fix that Gensim error and get your token frequency stats sorted out—no Python expert status required 😊

1. Fixing the doc2bow TypeError #

That error pops up because doc2bow expects a list of unicode tokens (think: individual words in a list), not a single block of text. You mentioned trying to convert to a word list and unicode already, so let's double-check your approach—maybe you missed a small step?

Here's a foolproof way to get it right:

• First, split your text into individual words (tokenize it). For simple cases, the built-in split() works, but you can also use more robust tools like nltk.word_tokenize if you need to handle punctuation/complex text better.
• In Python 3, strings are already unicode by default, so no extra conversion is needed. For Python 2, you'd wrap each word in u"" to make it unicode.

Example code for your IPython Notebook:

from gensim.corpora import Dictionary

# Your original text (replace with your actual document)
text = "Your document content goes here—this is just an example!"

# Step 1: Tokenize (split into individual words)
# Optional: Lowercase and remove punctuation to avoid duplicate tokens like "Hello" vs "hello"
import string
tokens = text.lower().translate(str.maketrans('', '', string.punctuation)).split()

# Step 2: Create a Gensim dictionary from your tokens
dictionary = Dictionary([tokens])

# Step 3: Now use doc2bow correctly with the token list
bow_vector = dictionary.doc2bow(tokens)

The key mistake people make here is passing the full string directly to doc2bow—always pass the list of tokens instead!

2. Getting Token Frequencies from Your Document #

Now that you've got the token list sorted, there are two easy ways to count how often each token appears:

Option 1: Use Gensim's Output #

The bow_vector we just created is a list of tuples like (token_id, frequency). To map these IDs back to actual words and see a readable frequency list:

# Convert bow vector to a word-to-frequency dictionary
token_frequencies = {dictionary[token_id]: count for token_id, count in bow_vector}

# Print the results
print("Token frequencies:")
for word, count in token_frequencies.items():
    print(f"{word}: {count}")

Option 2: Use Python's Built-in Counter (Simpler!) #

If you don't need Gensim for the frequency count itself, Python's collections.Counter is super straightforward:

from collections import Counter

# Use the same tokens list from earlier
freq_counter = Counter(tokens)

# Print the most common tokens (adjust the number to show more/less)
print("Most common tokens:")
for word, count in freq_counter.most_common(10):
    print(f"{word}: {count}")

This is great for quick, no-fuss frequency stats without needing to use Gensim's dictionary.

Quick Pro Tip #

Don't forget basic text preprocessing steps like lowercasing, removing punctuation, or even stopwords (common words like "the", "and" that don't add much meaning) if you want cleaner stats. For stopwords, you can use nltk.corpus.stopwords to filter them out of your tokens list.

内容的提问来源于stack exchange，提问作者Andrea Erőss