使用NLTK trigram模型生成句子时遇IndexError问题求助

尝试用NLTK构建trigram模型生成以“he said”开头的句子时，出现列表索引越界错误：

Traceback (most recent call last):
  File "C:\Users\PycharmProjects\homework3 3\main.py", line 77, in <module>
    suffix = pick_word(d[prefix])
  File "C:\Users\PycharmProjects\homework3 3\main.py", line 71, in pick_word
    return random.choice(sents)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\lib\random.py", line 378, in choice
    return seq[self._randbelow(len(seq))]
IndexError: list index out of range

相关代码（仅关注trigram核心部分）：

# imports
import string
import random

import nltk

nltk.download('punkt')
nltk.download('stopwords')
nltk.download('reuters')
from nltk.corpus import reuters, stopwords
from collections import defaultdict
from nltk import FreqDist, ngrams

# input the reuters sentences
sents = reuters.sents()

# write the removal characters such as : Stopwords and punctuation
stop_words = set(stopwords.words('english'))
string.punctuation = string.punctuation + '"' + '"' + '-' + '+' + '—'
removal_list = list(stop_words) + list(string.punctuation) + ['lt', 'rt']

# generate unigrams bigrams trigrams
unigram = []
trigram = []
tokenized_text = []

for sentence in sents:
    sentence = list(map(lambda x: x.lower(), sentence))
for word in sentence:
    if word == '.':
        sentence.remove(word)
    else:
        unigram.append(word)

tokenized_text.append(sentence)
trigram.extend(list(ngrams(sentence, 3, pad_left=True, pad_right=True)))

# remove the n-grams with removable words
def remove_stopwords(x):
    y = []
    for pair in x:
        count = 0
        for word in pair:
            if word in removal_list:
                count = count or 0
            else:
                count = count or 1
        if (count == 1):
            y.append(pair)
    return (y)

trigram = remove_stopwords(trigram)

# generate frequency of n-grams
freq_tri = FreqDist(trigram)

d = defaultdict(list)

#Trigrams
for a, b, c in freq_tri:
    if (a != None and b != None and c != None):
        d[a, b].extend([c] * freq_tri[a,b,c])

#Next word prediction
s = ''

def pick_word(sents):
    "Chooses a random element."
    return random.choice(sents)

prefix = "he", "said"
print(" ".join(prefix))
s = " ".join(prefix)
for i in range(19):
    suffix = pick_word(d[prefix])

1. 句子处理循环的缩进错误 #

核心问题：你遍历路透社语料库句子的循环完全写错了，第一个for sentence in sents没有缩进后续处理逻辑，导致只处理了语料库的最后一个句子，前面所有句子都被直接跳过。同时，遍历单词的循环也没有嵌套在句子循环内部，完全没处理所有句子的内容。

修复后的循环逻辑：

for sentence in sents:
    sentence = list(map(lambda x: x.lower(), sentence))
    # 新建列表存储清理后的单词，避免遍历中修改原列表导致漏项
    cleaned_sentence = []
    for word in sentence:
        if word != '.':
            cleaned_sentence.append(word)
            unigram.append(word)
    tokenized_text.append(cleaned_sentence)
    trigram.extend(list(ngrams(cleaned_sentence, 3, pad_left=True, pad_right=True)))

2. 停用词过滤逻辑错误 #

你的remove_stopwords函数逻辑完全失效：count = count or 0的写法无法正确判断三元组是否包含停用词，最终会保留所有至少有一个非停用词的三元组，不符合过滤需求。

如果需求是保留三个词都不在移除列表中且非pad产生的None的三元组，修复函数：

def remove_stopwords(trigrams):
    cleaned = []
    for tri in trigrams:
        if all(word not in removal_list and word is not None for word in tri):
            cleaned.append(tri)
    return cleaned

3. 空列表传入random.choice #

即使修复前面的问题，也可能出现d[("he", "said")]是空列表的情况——语料库中可能没有足够的(he, said, xxx)三元组，或被过滤掉了。此时调用random.choice就会触发索引越界。

修改pick_word函数增加判断：

def pick_word(sents):
    if not sents:
        return "."  # 无可用词时返回句号终止句子
    return random.choice(sents)

同时更新句子生成逻辑，每次生成后更新前缀：

prefix = ("he", "said")
print(" ".join(prefix))
s = " ".join(prefix)
for i in range(19):
    suffix = pick_word(d[prefix])
    s += " " + suffix
    if suffix == ".":
        break  # 遇到句号停止生成
    # 更新前缀为下一次预测做准备
    prefix = (prefix[1], suffix)
print(s)

4. 标点处理冗余优化 #

原代码中重复添加双引号，可简化为：

string.punctuation += '"-+—'

内容的提问来源于stack exchange，提问作者Jesper Ezra