运行基于fasttext的法语拼写纠错Python脚本时终端崩溃

问题背景 #

尝试在命令行运行基于fasttext实现的法语拼写纠错Python脚本，参考公开的fasttext拼写检查实现教程编写。

已执行操作 #

• 下载fasttext官方爬取语料训练的预训练词向量模型，包含bin和text两种格式文件
• 从源码编译构建fasttext命令行工具，执行命令如下：

$ git clone https://github.com/facebookresearch/fastText.git
$ cd fastText
$ make

• 完成拼写纠错脚本编写后尝试运行

相关代码 #

脚本文件名为script.py，完整代码如下：

import io
import fasttext

def load_vectors(fname):
    fin = io.open(fname, 'r', encoding='utf-8', newline='\n', errors='ignore')
    n, d = map(int, fin.readline().split())
    data = {}
    for line in fin:
        tokens = line.rstrip().split(' ')
        data[tokens[0]] = map(float, tokens[1:])
    return data

def spelltest(tests, model, vocab):
    "Run correction(wrong) on all (right, wrong) pairs; report results."
    import time
    start = time.clock()
    good, unknown = 0, 0
    n = len(tests)
    for right, wrong in tests:
        w = wrong
        if w in vocab:
            print('word: {} exists in the vocabulary. No correction required'.format(w))
        else:
            w_old = w
            w = model.get_nearest_neighbors(w, k=1)[0][1]
            print("found replacement: {} for word: {}".format(w, w_old))
        good += (w == right)
    dt = time.clock() - start
    print('{:.0%} of {} correct at {:.0f} words per second '
          .format(good / n, n, n / dt))

def Testset(lines):
    "Parse 'right: wrong1 wrong2' lines into [('right', 'wrong1'), ('right', 'wrong2')] pairs."
    return [(right, wrong)
            for (right, wrongs) in (line.split(':') for line in lines)
            for wrong in wrongs.split()]

if __name__ == "__main__":
    model = fasttext.load_model("cc.fr.300.bin")
    vocab = load_vectors("cc.fr.300.vec")
    
    spelltest(Testset(open('Memoires_secrets_09.txt')), model, vocab)
    #spelltest(Testset(open('spell-testset2.txt')), model, vocab)

报错现象 #

运行script.py时，终端输出如下警告后程序直接崩溃退出：

Warning : `load_model` does not return WordVectorModel or SupervisedModel any more, but a `FastText` object which is very similar.

运行环境 #

• 操作系统：Ubuntu 22.04 LTS
• Python版本：3.10.4

诱发原因 #

输出的警告是fasttext版本迭代后的兼容提示，本身不会导致程序崩溃，崩溃的核心原因有3个：

1. API版本不兼容：time.clock()方法在Python 3.8及以上版本已被正式移除，代码运行到计时逻辑时会直接抛出AttributeError终止进程。网上多数旧教程编写时基于Python 3.7及更早版本，未更新该接口调用。
2. 模型加载参数缺失：新版fasttext默认不初始化近邻查询所需的索引结构，直接调用get_nearest_neighbors方法会触发内部参数错误。
3. 内存占用过高：原代码的load_vectors函数会将整个cc.fr.300.vec文件的所有向量加载到内存，总占用超过12G，普通机器内存不足时会被系统直接杀死进程；且函数中存储的是map迭代器对象而非实际向量值，属于无效内存占用。

修复方案 #

按以下步骤操作即可正常运行：

1. 更换fasttext安装包为预编译wheel版本，避免源码编译带来的兼容问题：

pip uninstall fasttext -y
pip install fasttext-wheel

2. 修改代码中不兼容逻辑，优化内存占用：
   • 所有time.clock()调用替换为Python 3.10支持的time.perf_counter()
   • 加载模型时显式传入max_neighbors参数，开启近邻索引初始化
   • 重写词表加载逻辑，仅存储词的集合而非完整向量，将词表内存占用从12G以上降至200M以内
   • 打开测试集文件时显式指定utf-8编码，避免系统默认编码导致的文件读取错误
     修复后的可运行核心代码如下：

import io
import time
import fasttext

def load_vocab(fname):
    fin = io.open(fname, 'r', encoding='utf-8', newline='\n', errors='ignore')
    fin.readline()
    vocab = set()
    for line in fin:
        token = line.rstrip().split(' ', 1)[0]
        vocab.add(token)
    fin.close()
    return vocab

def spelltest(tests, model, vocab):
    start = time.perf_counter()
    good = 0
    n = len(tests)
    for right, wrong in tests:
        w = wrong
        if w in vocab:
            print('word: {} exists in the vocabulary. No correction required'.format(w))
        else:
            w_old = w
            w = model.get_nearest_neighbors(w, k=1)[0][1]
            print("found replacement: {} for word: {}".format(w, w_old))
        good += (w == right)
    dt = time.perf_counter() - start
    print('{:.0%} of {} correct at {:.0f} words per second '
          .format(good / n, n, n / dt))

def Testset(lines):
    return [(right, wrong)
            for (right, wrongs) in (line.split(':') for line in lines)
            for wrong in wrongs.split()]

if __name__ == "__main__":
    model = fasttext.load_model("cc.fr.300.bin", max_neighbors=10)
    vocab = load_vocab("cc.fr.300.vec")
    with open('Memoires_secrets_09.txt', 'r', encoding='utf-8') as f:
        spelltest(Testset(f.readlines()), model, vocab)

3. 如果运行时仍存在内存不足问题，可以使用fasttext内置的模型量化接口压缩bin模型体积，或选择官方发布的轻量级预训练词向量模型。

内容的提问来源于stack exchange，提问作者ljushu