如何用pyspark.ml确定LDA推荐主题数？或解决Gensim一致性评分报错？

Hey there! Let's work through your problem of selecting the optimal number of topics (K) for your PySpark LDA model, especially since you're hitting roadblocks with Gensim on large datasets in Databricks. Here's a breakdown of actionable solutions tailored to your setup:

You're already using logLikelihood and logPerplexity—great start! These are your first line of defense for evaluating model fit:

• Log Likelihood: Higher values mean the model fits your data better. It measures how well the model explains the observed documents.
• Log Perplexity: Lower values are better. This metric quantifies how well the model can predict unseen documents; think of it as a measure of model "confusion."

To use these for K selection, iterate over a range of K values, compute both metrics, and look for the elbow point (where the rate of improvement slows down). Here's a quick code snippet to automate this:

k_candidates = range(3, 12)  # Adjust range based on your domain knowledge
ll_scores = []
lp_scores = []

for k in k_candidates:
    lda = LDA(k=k, maxIter=50, featuresCol="tf_idf", seed=1234)
    model = lda.fit(tfidfLaw)
    ll_scores.append(model.logLikelihood(tfidfLaw))
    lp_scores.append(model.logPerplexity(tfidfLaw))
    print(f"K={k} | Log Likelihood: {ll_scores[-1]:.2f} | Log Perplexity: {lp_scores[-1]:.2f}")

# Visualize in Databricks
import matplotlib.pyplot as plt
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))

ax1.plot(k_candidates, ll_scores, marker='o', color='blue')
ax1.set_title('Log Likelihood vs Number of Topics')
ax1.set_xlabel('K')
ax1.set_ylabel('Log Likelihood')

ax2.plot(k_candidates, lp_scores, marker='o', color='red')
ax2.set_title('Log Perplexity vs Number of Topics')
ax2.set_xlabel('K')
ax2.set_ylabel('Log Perplexity')

plt.tight_layout()
display(fig)

PySpark ML doesn't have a native equivalent to Gensim's CoherenceModel, but you can adapt Gensim to work with your PySpark data without crashing. The key is sampling your data—full datasets will overwhelm your Driver's memory (15.3GB in your case). Here's how to do it:

Step-by-Step Implementation: #

from gensim.models.coherencemodel import CoherenceModel
from gensim.corpora.dictionary import Dictionary

# 1. Grab the vocabulary from your CountVectorizerModel
vocab = counterModel.vocabulary

# 2. Extract top terms per topic from your PySpark LDA model
topics_df = model.describeTopics(maxTermsPerTopic=10)  # Adjust term count as needed
# Convert term indices to actual words
topics = topics_df.rdd.map(lambda row: [vocab[idx] for idx in row.termIndices]).collect()

# 3. Sample a subset of your tokenized texts (critical for memory!)
# Start with 10-20% of your data—adjust based on what your Driver can handle
sample_fraction = 0.15
sample_texts = tokenizedText.sample(fraction=sample_fraction, seed=1234)\
                          .select("Tokens")\
                          .rdd.map(lambda x: x[0])\
                          .collect()

# 4. Prepare Gensim inputs
gensim_dict = Dictionary(sample_texts)
# 5. Calculate coherence score (use 'c_v' for human-interpretable results)
coherence_model = CoherenceModel(
    topics=topics,
    texts=sample_texts,
    dictionary=gensim_dict,
    coherence='c_v'
)
coherence_score = coherence_model.get_coherence()
print(f"Coherence Score for K={model.getK()}: {coherence_score:.4f}")

Why This Works: #

• Sampling reduces the amount of data loaded into your Driver's memory, avoiding the "route unreachable" or OOM errors you saw earlier.
• Even a 10-20% sample is statistically representative enough to get meaningful coherence scores for K selection.

To make this smoother, consider these tweaks for your Databricks Runtime 6.5 ML cluster:

• Increase Driver Memory: If your budget allows, bump up the Driver memory to 30GB or more. This lets you use larger samples, improving the reliability of your coherence scores.
• Parallelize Coherence Calculation: For extremely large datasets, split your tokenized texts into chunks, compute coherence for each chunk, then average the results. You can use PySpark UDFs for this (just make sure to serialize Gensim objects properly).
• Combine Metrics: Don't rely on just one metric. Pick the K where log perplexity stops dropping sharply (elbow) and coherence score is highest—align this with your domain knowledge (e.g., does 7 topics make sense for your text corpus?).

Combine all the above into a single workflow to test multiple K values and pick the best one:

import matplotlib.pyplot as plt

k_candidates = range(3, 11)
ll_scores = []
lp_scores = []
coherence_scores = []
vocab = counterModel.vocabulary

for k in k_candidates:
    # Train LDA
    lda = LDA(k=k, maxIter=50, featuresCol="tf_idf", seed=1234)
    model = lda.fit(tfidfLaw)
    # Get built-in metrics
    ll = model.logLikelihood(tfidfLaw)
    lp = model.logPerplexity(tfidfLaw)
    # Get coherence score
    topics_df = model.describeTopics(maxTermsPerTopic=10)
    topics = topics_df.rdd.map(lambda row: [vocab[idx] for idx in row.termIndices]).collect()
    sample_texts = tokenizedText.sample(fraction=0.15, seed=1234)\
                              .select("Tokens")\
                              .rdd.map(lambda x: x[0])\
                              .collect()
    gensim_dict = Dictionary(sample_texts)
    coherence_model = CoherenceModel(topics=topics, texts=sample_texts, dictionary=gensim_dict, coherence='c_v')
    cs = coherence_model.get_coherence()
    # Store results
    ll_scores.append(ll)
    lp_scores.append(lp)
    coherence_scores.append(cs)
    print(f"K={k} | LL: {ll:.2f} | LP: {lp:.2f} | Coherence: {cs:.4f}")

# Visualize all metrics
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(18, 5))

ax1.plot(k_candidates, ll_scores, marker='o', color='blue')
ax1.set_title('Log Likelihood vs K')
ax1.set_xlabel('Number of Topics')
ax1.set_ylabel('Log Likelihood')

ax2.plot(k_candidates, lp_scores, marker='o', color='red')
ax2.set_title('Log Perplexity vs K')
ax2.set_xlabel('Number of Topics')
ax2.set_ylabel('Log Perplexity')

ax3.plot(k_candidates, coherence_scores, marker='o', color='green')
ax3.set_title('Coherence Score vs K')
ax3.set_xlabel('Number of Topics')
ax3.set_ylabel('Coherence Score')

plt.tight_layout()
display(fig)

Once you run this, look for the K that balances all three metrics—this will be your optimal topic count.

内容的提问来源于stack exchange，提问作者EJS