基于WordNet实现LESK算法后如何调用NLTK的SemCor语料库50条句子测试

1. 前置依赖导入 #

首先确认你已经导入所有需要的NLTK模块，且提前初始化了公共变量：

import nltk
from nltk.corpus import semcor, wordnet as wn, stopwords
from nltk.tokenize import WordPunctTokenizer

# 初始化公共变量
stopwords = set(stopwords.words('english'))
tokenizer = WordPunctTokenizer()

如果没下载过对应语料，先执行一次下载：

nltk.download(['semcor', 'wordnet', 'stopwords', 'punkt'])

2. 修正原有Lesk代码的已知问题 #

你的原始代码存在两处会导致结果异常的bug，先修正：

• gloss.union(example) 方法不会修改原集合，需要将返回值赋值给gloss，同时示例句子也需要分词处理
• 转义字符"替换为正常双引号
  修正后代码如下：

def overlap_context(synset, sentence):
    gloss = set(tokenizer.tokenize(synset.definition()))
    for example in synset.examples():
        # 示例句子分词后再加入词集
        gloss = gloss.union(tokenizer.tokenize(example))
    gloss = gloss.difference(stopwords)
    if isinstance(sentence, str):
        sentence = set(sentence.split(" "))
    elif isinstance(sentence, list):
        sentence = set(sentence)
    elif isinstance(sentence, set):
        pass
    sentence = sentence.difference(stopwords)
    return len(gloss.intersection(sentence))

def lesk_algorithm(word, sentence):
    best_sense = None
    max_overlap = 0.0
    word = wn.morphy(word) if wn.morphy(word) is not None else word
    for sense in wn.synsets(word):
        overlap = overlap_context(sense, sentence)
        for hyponym in sense.hyponyms():
            overlap += overlap_context(hyponym, sentence)
        if overlap > max_overlap:
            max_overlap = overlap
            best_sense = sense
    return best_sense

3. 加载SemCor前50条句子并执行测试 #

SemCor的句子已经附带了人工标注的同义词集（synset）标签，可以直接作为标准答案和你的Lesk算法预测结果对比，测试代码如下：

# 取SemCor前50条带标注的句子
test_sentences = semcor.tagged_sents(tag='sem')[:50]

correct = 0
total = 0

for sent in test_sentences:
    # 提取句子的纯文本分词列表，用于传入Lesk算法
    sent_tokens = [token.label() if isinstance(token, nltk.tree.Tree) else token for token in sent]
    sent_words = [token[0] if isinstance(token, tuple) else token for token in sent_tokens]
    for token in sent:
        # 只处理带synset标注的实词，跳过标点、无标注词
        if isinstance(token, nltk.tree.Tree) and hasattr(token.label(), 'synset'):
            target_word = token[0][0]
            true_synset = token.label().synset()
            # 调用你实现的Lesk算法预测义项
            pred_synset = lesk_algorithm(target_word, sent_words)
            if pred_synset == true_synset:
                correct +=1
            total +=1

# 输出测试结果
print(f"测试词总数：{total}")
print(f"预测正确数：{correct}")
print(f"准确率：{correct/total:.2%}")

测试说明 #

• SemCor的标注义项为人工校准的标准答案，直接和算法输出的synset对比即可判断预测是否正确
• 代码中跳过了无义项标注的虚词、标点符号，只统计实词的预测准确率
• 你可以根据需求调整测试逻辑，比如只统计特定词性的词、修改对比规则等

内容的提问来源于stack exchange，提问作者senna