无法使用NLTK加载德语单词语料库，寻求技术解决方案

问题描述 #

使用NLTK判断单词是否为有效英语/德语单词时，英语验证功能正常，但处理德语时抛出错误：

OSError: No such file or directory: 'C:\Users\Usama\AppData\Roaming\nltk_data\corpora\words\de'

用户代码示例：

import nltk
nltk.download('words')


def is_valid_word(word, language):
    if language == "english":
        words = set(nltk.corpus.words.words('en'))
    elif language == "german":
        print("HERE")
        words = set(nltk.corpus.words.words('de'))
    else:
        raise ValueError(f"Invalid language: {language}")
    return word.lower() in words


word = "hello"
language = "english"

if is_valid_word(word, language):
    print(f"{word} is a valid {language} word")
else:
    print(f"{word} is not a valid {language} word")

word = "hallo"
language = "german"

if is_valid_word(word, language):
    print(f"{word} is a valid {language} word")
else:
    print(f"{word} is not a valid {language} word")

错误原因 #

NLTK的words语料库默认仅包含英语单词，没有内置德语词库。调用nltk.corpus.words.words('de')时，系统找不到对应的德语语料文件，因此抛出路径不存在的错误。

解决方案 #

1. 使用NLTK内置德语语料库 #

NLTK提供了german_ngrams语料库，包含常用德语词汇。先下载该语料库，再调整验证逻辑：

import nltk
nltk.download('words')
nltk.download('german_ngrams')

from nltk.corpus import german_ngrams

def is_valid_word(word, language):
    if language == "english":
        words = set(nltk.corpus.words.words('en'))
    elif language == "german":
        # 从german_ngrams中提取单个德语单词
        german_words = set()
        for gram in german_ngrams.words():
            if len(gram.split()) == 1:
                german_words.add(gram.lower())
        words = german_words
    else:
        raise ValueError(f"Invalid language: {language}")
    return word.lower() in words

# 测试英语
word = "hello"
language = "english"
if is_valid_word(word, language):
    print(f"{word} is a valid {language} word")
else:
    print(f"{word} is not a valid {language} word")

# 测试德语
word = "hallo"
language = "german"
if is_valid_word(word, language):
    print(f"{word} is a valid {language} word")
else:
    print(f"{word} is not a valid {language} word")

2. 替代方案：导入本地德语词库 #

如果german_ngrams词汇量不足，可以自行下载公开的德语单词列表（如txt格式），手动加载验证：

import nltk
nltk.download('words')

def load_german_words(file_path):
    # 加载本地德语单词文件
    with open(file_path, 'r', encoding='utf-8') as f:
        return set(word.strip().lower() for word in f if word.strip())

def is_valid_word(word, language):
    if language == "english":
        words = set(nltk.corpus.words.words('en'))
    elif language == "german":
        # 替换为你的德语词库文件路径
        words = load_german_words('german_words.txt')
    else:
        raise ValueError(f"Invalid language: {language}")
    return word.lower() in words

内容的提问来源于stack exchange，提问作者Usama Hameed