LDA模型生成游戏评论关键词时索引越界错误的解决咨询

问题 #

我有一组以嵌套列表形式存储的游戏文本评论JSON数据，已经完成词形还原并应用词袋模型处理。现在想通过LDA模型结合TF-IDF去重，生成各游戏的Top30关键词合并列表作为后续机器学习的输入特征，但运行代码时触发索引错误：index 1624 is out of bounds for axis 1 with size 1624。希望不删除超出索引范围的词汇，避免丢失重要数据，请问能否增大索引范围？

代码片段 #

游戏列表：

game_list = ["shenzhen_combined", "moleksyntez_combined", "gladiabots_combined", "baba_combined", "exapunks_combined", "hrmachine_combined", "infinifactory_combined", "logicbots_combined", "mechanica_combined", "mhrd_combined", "opusmag_combined", "primemover_combined", "quadcowboy_combined", "screeps_combined", "sevenbil_combined", "siliconzeroes_combined", "spacechem_combined", "stonestory_combined", "TIS100_combined", "turing_combined", "whiletrue_combined"]

核心函数：

def gen_keyw_feat(game_list, ntop=5, nwrd=6):
    rtrn_list = []

    for one_game in game_list:        
        data = load_data(one_game + ".json")["data"]
        stop_words = stopwords.words("english")
        #new_data = rem_stopwords(data)
        new_data = [one_data[1] for one_data in data]

        lemmatized_texts = lemmatization(new_data)
        data_words = gen_words(lemmatized_texts)
        id2word = corpora.Dictionary(data_words)
        texts = data_words
        corpus = [id2word.doc2bow(text) for text in texts]
        
        tfidf = TfidfModel(corpus, id2word=id2word)

        low_value = 0.03
        words = []
        words_missing_in_tfidf = []
        for i in range(0, len(corpus)):
            bow = corpus[i]
            low_value_words = []
            tfidf_ids = [id for id, value in tfidf[bow]]
            bow_ids = [id for id, value in bow]
            low_value_words = [id for id, value in tfidf[bow] if value < low_value]
            drops = low_value_words+words_missing_in_tfidf
            for item in drops:
                words.append(id2word[item])
            words_missing_in_tfidf = [id for id in bow_ids if id not in tfidf_ids]

            new_bow = [b for b in bow if b[0] not in low_value_words and b[0] not in words_missing_in_tfidf]
            corpus[i] = new_bow
        
        rtrn_list += [lda_rslts for lda_rslts in LDA_func(ntop, nwrd, corpus, one_game)]

    return list(set(rtrn_list))

程序输出 #

shenzhen_combined 133 1624
moleksyntez_combined 65 1624
gladiabots_combined 117 1624
index 1624 is out of bounds for axis 1 with size 1624

baba_combined 139 1624
index 1624 is out of bounds for axis 1 with size 1624

exapunks_combined 150 1624
index 1624 is out of bounds for axis 1 with size 1624

hrmachine_combined 264 1624
index 1624 is out of bounds for axis 1 with size 1624

infinifactory_combined 199 1624
index 1624 is out of bounds for axis 1 with size 1624

logicbots_combined 73 1624
mechanica_combined 69 1624
mhrd_combined 117 1624
index 1624 is out of bounds for axis 1 with size 1624

opusmag_combined 76 1624
index 1624 is out of bounds for axis 1 with size 1624

primemover_combined 67 1624
quadcowboy_combined 180 1624
index 1624 is out of bounds for axis 1 with size 1624

screeps_combined 255 1624
index 1624 is out of bounds for axis 1 with size 1624

sevenbil_combined 134 1624
index 1624 is out of bounds for axis 1 with size 1624

siliconzeroes_combined 74 1624
spacechem_combined 438 1624
index 1624 is out of bounds for axis 1 with size 1624

stonestory_combined 69 1624
TIS100_combined 318 1624
index 1624 is out of bounds for axis 1 with size 1624

turing_combined 5 1624
whiletrue_combined 119 1624
index 1624 is out of bounds for axis 1 with size 1624

生成的关键词列表（部分）：

['coder',
'manual',
'initial',
'hurtin',
'board',
'level',
'fun',
'alike',
'programming',
'job',
'mechanic',
'solitare',
'forever',
'byte',
'badly',
'easy',
'extremely',
'update',
'place',
'command',
'minimized',
'mini',
'once',
'miss',
'win',
'arcane',
'great',
'cocoa',
'balanced',
'then',
'recommend',
'maybe',
'quirk',
'circuit',
'line',
'hour',
'printer',
'jack',
'game',
'count',
'zacktronic',
'challenge',
'hook',
'purely',
'chip',
'documentation',
'neat',
'inadequate',
'somewhere',
'good',
'already',
'approach',
'chuckle',
'electronic',
'away',
'bed',
'last',
'asm',
'controller',
'fundamental',
'so',
'turn',
'micro',
'just',
'rewarding',
'senior',
'mediocre',
'whole',
'at',
'ever',
'number',
'primarily',
'bashingly',
'assembly',
'state',
'zachtronic',
'phone',
'balance',
'thank',
'consideration',
'solitaire',
'pain',
'swing',
'instruction',
'premium',
'thought',
'original',
'forward',
'much',
'difficulty',
'briefly',
'puzzle',
'straightforward',
'tfw',
'halp',
'physical',
'ground',
'inferior',
'screen',
'engineering',
'beautiful',
'home',
'unlikely',
'monotonous',
'unprepared',
'university',
'grasp',
'capability',
'kit',
'diddling',
'helpful',
'more',
'program',
'difficult',
'conditional',
'right',
'thinking',
'setting',
'accomplishment',
'ridiculous',
'codeslp',
'product',
'code',
'quite',
'again',
'character',
'fake',
'day',
'assignment',
'way',
'combination',
'port',
'main',
'friend',
'optimization',
'severe',
'sheet',
'absolutely',
'mind',
'many',
'pressure',
'bonus',
'annoying',
'tool',
'brain',
'smart',
'interesting',
'distro',
'fine',
'fan',
'package']

问题分析与解决方案 #

错误原因 #

这个索引越界错误并非来自你展示的代码段，而是出在LDA_func函数内部。你当前的逻辑是为每个游戏单独构建id2word词汇表，导致不同游戏的词汇ID不统一，但LDA_func可能复用了固定维度（比如1624维）的模型/特征空间，新游戏的词汇ID超出了这个范围，从而触发报错。

另外，你当前的TF-IDF过滤逻辑存在漏洞：words_missing_in_tfidf会跨文档迭代累积，导致后续文档错误过滤掉之前文档中未出现在TF-IDF中的词汇，破坏原始词汇分布，也可能间接引发ID混乱。

解决方案 #

1. 修复TF-IDF过滤逻辑

先修正当前代码中TF-IDF部分的错误，确保每个文档独立处理，避免跨文档累积过滤：

low_value = 0.03
words = []
# 移除跨迭代的words_missing_in_tfidf累积，每个文档单独处理
for i in range(0, len(corpus)):
    bow = corpus[i]
    tfidf_bow = tfidf[bow]
    # 获取当前文档中TF-IDF值低于阈值的词汇ID
    low_value_words = [id for id, value in tfidf_bow if value < low_value]
    # 获取当前文档中未被TF-IDF处理的词汇ID（通常是全局词频为1的词）
    bow_ids = set(id for id, value in bow)
    tfidf_ids = set(id for id, value in tfidf_bow)
    words_missing_in_tfidf = list(bow_ids - tfidf_ids)
    # 记录要删除的词汇
    drops = low_value_words + words_missing_in_tfidf
    words.extend([id2word[item] for item in drops])
    # 过滤当前文档的bow
    new_bow = [b for b in bow if b[0] not in drops]
    corpus[i] = new_bow

2. 构建全局统一词汇表（核心解决索引越界）

要避免索引越界且保留所有词汇，必须先构建全局统一的词汇表，再基于这个词典对所有游戏的评论进行编码：

# 第一步：先加载所有游戏数据，构建全局词汇表
all_data_words = []
for one_game in game_list:        
    data = load_data(one_game + ".json")["data"]
    new_data = [one_data[1] for one_data in data]
    lemmatized_texts = lemmatization(new_data)
    data_words = gen_words(lemmatized_texts)
    all_data_words.extend(data_words)

# 生成全局词典，保留所有词汇（不删除任何数据）
id2word = corpora.Dictionary(all_data_words)

# 第二步：基于全局词典处理每个游戏的评论
def gen_keyw_feat(game_list, id2word, ntop=5, nwrd=6):
    rtrn_list = []
    # 基于全局词典初始化TF-IDF模型
    tfidf = TfidfModel(id2word=id2word)

    for one_game in game_list:        
        data = load_data(one_game + ".json")["data"]
        new_data = [one_data[1] for one_data in data]

        lemmatized_texts = lemmatization(new_data)
        data_words = gen_words(lemmatized_texts)
        # 用全局词典生成bow
        corpus = [id2word.doc2bow(text) for text in data_words]
        
        # 应用TF-IDF过滤（已修复的逻辑）
        low_value = 0.03
        words = []
        for i in range(len(corpus)):
            bow = corpus[i]
            tfidf_bow = tfidf[bow]
            low_value_words = [id for id, value in tfidf_bow if value < low_value]
            bow_ids = set(id for id, value in bow)
            tfidf_ids = set(id for id, value in tfidf_bow)
            words_missing_in_tfidf = list(bow_ids - tfidf_ids)
            drops = low_value_words + words_missing_in_tfidf
            words.extend([id2word[item] for item in drops])
            new_bow = [b for b in bow if b[0] not in drops]
            corpus[i] = new_bow
        
        # 将全局词典传入LDA_func，确保特征维度统一
        rtrn_list += [lda_rslts for lda_rslts in LDA_func(ntop, nwrd, corpus, id2word, one_game)]

    return list(set(rtrn_list))

# 调用函数生成最终关键词
final_keywords = gen_keyw_feat(game_list, id2word)

3. 修正LDA_func实现

确保LDA_func接收全局词典id2word参数，基于全局词汇表的维度初始化LDA模型：

def LDA_func(ntop, nwrd, corpus, id2word, one_game):
    # 基于全局词典的维度初始化LDA模型
    lda_model = models.LdaModel(corpus=corpus, id2word=id2word, num_topics=ntop, random_state=42)
    # 提取每个主题的Top关键词
    top_keywords = []
    for topic in lda_model.show_topics(num_words=nwrd, formatted=False):
        top_keywords.extend([word for word, prob in topic[1]])
    # 返回去重后的关键词列表
    return list(set(top_keywords))

注意事项 #

• 全局词汇表包含所有游戏的词汇，不会丢失任何数据，从根源上解决了索引越界问题
• TF-IDF过滤需基于每个文档独立处理，避免跨文档的错误过滤
• LDA模型必须基于全局词汇表初始化，确保所有游戏的特征维度统一

内容的提问来源于stack exchange，提问作者asimichroma