Jupyter中Enron数据集读取异常：仅读取6000封邮件而非33000+

核心问题 #

你的代码仅读取了6000封邮件，根源在于只处理了最后一个目录（enron6）的邮件。遍历enron_list的循环仅更新了ham_folder和spam_folder的变量值，但实际处理邮件的逻辑（遍历文件夹、读取文件的代码）在这个循环外面。循环结束后，变量仅保留最后一个目录的路径，前面的enron1到enron5完全未被处理。

修改后的代码 #

将处理ham和spam邮件的逻辑放到for directory in enron_list循环内部，确保每个目录都被遍历处理：

import os
import re
import datetime as dt
import pandas as pd

mails_list = []
enron_list = ['enron1', 'enron2', 'enron3', 'enron4', 'enron5', 'enron6']

def getEmail():
    print("Processing directories...")

    for directory in enron_list:
        print("...processing " + str(directory) + "...")
        ham_folder = '/xxxxxxxxx/xxx/xxxx/xxxx/SpamCode/rawdata/' + directory + '/ham'
        spam_folder = '/xxxxxxxxxx/xxxx/xxxxx/xxx/SpamCode/rawdata/' + directory + '/spam'

        # Process ham messages in current directory
        for entry in os.scandir(ham_folder):
            try:
                # 用with语句自动关闭文件，更安全
                with open(entry, encoding="latin_1") as file:
                    content = file.read().split("\n", 1)
                # 避免空文件或格式错误导致索引越界
                if len(content) < 2:
                    print(f"Skipping empty or malformed file: {entry}")
                    continue
                subject = content[0].replace("Subject: ", "")
                message = content[1]
                # 解析日期，增加匹配失败判断
                pattern = r"\d+\.(\d+-\d+-\d+)"
                match = re.search(pattern, str(entry))
                if not match:
                    print(f"Could not parse date from file: {entry}")
                    continue
                date_str = match.group(1)
                date = dt.datetime.strptime(date_str, '%Y-%m-%d')
                mails_list.append([subject, message, "ham", date])
            except UnicodeDecodeError as e:
                print("COULD NOT DECODE")
                print(f"Problem with file: {entry}")
                print(f"Error message: {e}")
                continue

        # Process spam messages in current directory
        for entry in os.scandir(spam_folder):
            try:
                with open(entry, encoding="latin_1") as file:
                    content = file.read().split("\n", 1)
                if len(content) < 2:
                    print(f"Skipping empty or malformed file: {entry}")
                    continue
                subject = content[0].replace("Subject: ", "")
                message = content[1]
                pattern = r"\d+\.(\d+-\d+-\d+)"
                match = re.search(pattern, str(entry))
                if not match:
                    print(f"Could not parse date from file: {entry}")
                    continue
                date_str = match.group(1)
                date = dt.datetime.strptime(date_str, '%Y-%m-%d')
                mails_list.append([subject, message, "spam", date])
            except UnicodeDecodeError as e:
                print("COULD NOT DECODE")
                print(f"Problem with file: {entry}")
                print(f"Error message: {e}")
                continue
        
    return mails_list

emails = getEmail()
print("All directories processed. Writing to Dataframe...")
mails = pd.DataFrame(emails, columns=["Subject", "Message", "Spam/Ham", "Date"])

mails.to_csv('mails.csv')

print("\nData processed and saved to file.\nMails contained in data:")
print("\nTotal:\t" + str(mails.shape[0]))
print(mails["Spam/Ham"].value_counts())

额外优化点 #

1. 使用with open(...)语句自动管理文件资源，避免手动关闭文件的遗漏问题。
2. 增加空文件/格式错误文件的判断，防止content[1]索引越界报错。
3. 增加日期解析失败的判断，避免re.search()返回None时调用group(1)报错。
4. 异常处理后添加continue，跳过错误文件继续处理其他邮件，不会中断整个流程。

内容的提问来源于stack exchange，提问作者vagas emprego