Python脚本处理大语料遇MemoryError：如何优化提升效率？

问题背景 #

我是一名语言学家，编写了Python脚本用于计算文本语料统计数据，但处理约3GB、5亿词的大文件时，在i5处理器+16GB内存的机器上触发了MemoryError。错误信息如下：

Traceback (most recent call last):
File "/path/to/my/myscript.py", line 43, in
keywords, target_norm, reference_norm, smp_score = calculate_keywords('file1.txt', 'file2.txt')
File "/path/to/my/myscript.py", line 9, in calculate_keywords
target_text = f.read().lower().translate(str.maketrans('','','?!"():;.,“/[]'))
MemoryError

原脚本代码：

#!/usr/bin/env python3

import collections
import math
import string

def calculate_keywords(target, reference):
    with open(target, 'r') as f:
        target_text = f.read().lower().translate(str.maketrans('','','?!"():;.,“/[]'))
        target_words = target_text.split()

    with open(reference, 'r') as f:
        reference_text = f.read().lower().translate(str.maketrans('','','?!"():;.,“/[]'))
        reference_words = reference_text.split()

    target_freq = collections.Counter(target_words)
    reference_freq = collections.Counter(reference_words)

    target_total = sum(target_freq.values())
    reference_total = sum(reference_freq.values())
    
    target_norm = {}
    reference_norm = {}

    for word, freq in target_freq.items():
        target_norm[word] = freq / target_total * 1000000

    for word, freq in reference_freq.items():
        reference_norm[word] = freq / reference_total * 1000000

    smp_scores = {}
    for word, freq in target_norm.items():
        if word not in reference_norm:
            reference_norm[word] = 0
        s1 = freq + 100
        s2 = reference_norm[word] + 100
        smp_scores[word] = s1 / s2

    keywords = sorted(smp_scores, key=smp_scores.get, reverse=True)[:50]
    return keywords, target_norm, reference_norm, smp_scores
    

keywords, target_norm, reference_norm, smp_score = calculate_keywords('myfile1.txt', 'myfile2.txt')
for word in keywords:
    print(f"{word} {target_norm[word]} {reference_norm[word]} {smp_score[word]}")

错误原因 #

f.read()会把整个文件一次性加载到内存，3GB文本加上处理后的字符串、分词列表，内存占用远超16GB的承载上限，直接触发内存不足错误。

优化方案（适配非计算机专业） #

1. 逐行读取+实时统计，避免一次性加载文件 #

把一次性读取整个文件改成逐行读取，每处理一行就更新词频计数器，内存仅保留当前行和计数器数据，彻底解决内存过载问题。

2. 简化字符清理逻辑 #

用Python内置的string.punctuation标点集合，加上你需要额外移除的符号，替代手动罗列标点，更简洁且不易遗漏。

3. 按需计算，减少冗余存储 #

原脚本中target_norm和reference_norm可按需保留；若仅需计算SMP分数，甚至可以跳过字典存储，直接用频率和总词数实时计算，进一步节省内存。

优化后完整脚本 #

#!/usr/bin/env python3

import collections
import string

def calculate_keywords(target, reference):
    # 定义要移除的标点：内置标点+额外需要清理的符号
    remove_chars = string.punctuation + '“”/[]'
    translator = str.maketrans('', '', remove_chars)

    # 逐行读取目标文件，实时统计词频与总词数
    target_freq = collections.Counter()
    target_total = 0
    with open(target, 'r') as f:
        for line in f:
            cleaned_line = line.lower().translate(translator)
            words = cleaned_line.split()
            target_freq.update(words)
            target_total += len(words)

    # 逐行读取参考文件，实时统计词频与总词数
    reference_freq = collections.Counter()
    reference_total = 0
    with open(reference, 'r') as f:
        for line in f:
            cleaned_line = line.lower().translate(translator)
            words = cleaned_line.split()
            reference_freq.update(words)
            reference_total += len(words)

    # 计算归一化频率（若不需要保存全量数据，可跳过此步，直接在SMP计算时实时求值）
    target_norm = {}
    for word, freq in target_freq.items():
        target_norm[word] = freq / target_total * 1000000

    reference_norm = {}
    for word, freq in reference_freq.items():
        reference_norm[word] = freq / reference_total * 1000000

    # 计算SMP分数
    smp_scores = {}
    for word, target_n in target_norm.items():
        # 参考文件中无对应词时，归一化频率设为0
        reference_n = reference_norm.get(word, 0)
        s1 = target_n + 100
        s2 = reference_n + 100
        smp_scores[word] = s1 / s2

    # 提取Top50关键词
    keywords = sorted(smp_scores, key=smp_scores.get, reverse=True)[:50]
    return keywords, target_norm, reference_norm, smp_scores

# 执行计算并格式化输出结果
keywords, target_norm, reference_norm, smp_score = calculate_keywords('myfile1.txt', 'myfile2.txt')
for word in keywords:
    print(f"{word} {target_norm[word]:.2f} {reference_norm.get(word, 0):.2f} {smp_score[word]:.2f}")

额外提示 #

• 若内存仍紧张，可删除target_norm和reference_norm字典，直接在计算SMP分数时用freq / total * 1000000实时计算，减少一份内存占用。
• 运行脚本前关闭无关程序，释放更多可用内存。

内容的提问来源于stack exchange，提问作者Michael Gauthier