如何基于Python的lda包训练的LDA模型实现新文本预测？

Alright, I’ve been there with the lda package’s sparse documentation—let’s get you set up with saving/loading your model and predicting new text without retraining from scratch. Here’s the step-by-step solution:

The lda package doesn’t have built-in serialization, but Python’s pickle module works perfectly to save both your trained LDA model and the CountVectorizer (since you need the same vocabulary for new text). Add this after training your model:

import pickle

# Save the LDA model
with open('lda_trained_model.pkl', 'wb') as model_file:
    pickle.dump(lda_model, model_file)

# Save the CountVectorizer (critical for consistent text processing)
with open('count_vectorizer.pkl', 'wb') as vec_file:
    pickle.dump(cvectorizer, vec_file)

When you need to reuse the model later, load both files like this:

import pickle
import lda
from sklearn.feature_extraction.text import CountVectorizer

# Load the LDA model
with open('lda_trained_model.pkl', 'rb') as model_file:
    loaded_lda = pickle.load(model_file)

# Load the CountVectorizer
with open('count_vectorizer.pkl', 'rb') as vec_file:
    loaded_vectorizer = pickle.load(vec_file)

The key here is to process the new text exactly the same way you processed your training data. Let’s wrap your preprocessing logic into a reusable function first:

import nltk
from nltk.corpus import stopwords

stops = set(stopwords.words('english'))

def preprocess_new_text(text):
    # Lowercase and tokenize (match your training preprocessing)
    tokens = nltk.tokenize.word_tokenize(text.lower())
    # Remove stopwords
    cleaned_tokens = [token for token in tokens if token not in stops]
    # Join back to a single string for the vectorizer
    return ' '.join(cleaned_tokens)

Now use this function to process your new text, transform it with the loaded vectorizer, and get topic predictions:

# Your new text
new_text = '50 facts about Radio 1 & 2 as they turn 50'

# Preprocess the new text
cleaned_new_text = preprocess_new_text(new_text)

# Transform the text (use transform(), NOT fit_transform()—we don't want to update the vocabulary!)
new_text_vectorized = loaded_vectorizer.transform([cleaned_new_text])

# Get topic distribution for the new text
topic_distribution = loaded_lda.transform(new_text_vectorized)

# Print the results
print("Topic probabilities for the new text:")
for topic_num, prob in enumerate(topic_distribution[0], 1):
    print(f"Topic {topic_num}: {prob:.4f}")

Important Notes #

• Consistent Preprocessing: Make sure your new text uses the exact same cleaning steps as your training data (no adding lemmatization or bigrams now if you skipped them earlier—this will break feature consistency).
• Vocabulary Match: Never fit a new CountVectorizer for predictions; always use the one you saved with the model. This ensures the feature indices match what the LDA model was trained on.
• Version Compatibility: Keep your Python and lda package versions consistent between saving and loading to avoid pickle compatibility issues.

内容的提问来源于stack exchange，提问作者aviss