CSV文件写入行数与生成器长度不符问题求助

问题概述 #

现有一个包含9,174,189行的CSV文件，经筛选后符合条件的行数应为256098，但将结果写入新CSV文件后，通过wc -l new.csv统计或在vim中查看，显示行数为269100，与预期的生成器输出行数不一致。

环境与代码 #

• Python版本：3.6.9
• 行数统计命令：wc -l

CSV读写代码如下：

import csv, os

def getstuff(filename,keywords,field):
    with open(filename, "r") as csvfile:
        datareader = csv.DictReader(csvfile)
        yield datareader.fieldnames
        for row in datareader:
            for k in keywords:
                if k in row[field]:
                    yield row
                    break # row[field] might contain multiple keywords

def writestuff(gen,fileout):
    with open(fileout, "w") as csvfile:
        datawriter = csv.DictWriter(csvfile,fieldnames=next(gen))
        datawriter.writeheader()
        datawriter.writerows(gen)

if __name__ == "__main__":
    k1 = ["Staff","Aug"]
    f1 = "RFO Description"
    files = os.listdir()
    filename = files[0]
    x = getstuff(filename,k1,f1)
    writestuff(x,"new.csv")

验证过程 #

通过以下统计函数验证符合条件的行数：

def f(filename,field):
    d = {}
    with open(filename, "r") as csvfile:
        datareader = csv.DictReader(csvfile)
        for row in datareader:
            try:
                d[row[field]] += 1
            except KeyError:
                d[row[field]] = 1
    return d

将该函数生成的字典写入file.txt，传入新生成的new.csv和关键词列表后，执行cat file.txt得到结果：

Augmenting Software Products and Services,20285
Augmenting Networking Equipment,2323
Information Technology Staff Augmentation Contract (ITSAC) Services,1752
Information Technology Staff Augmentation Contract (ITSAC),212324
Deliverables-Based Information Technology Services (DBITS) - Augmenting,7947
Augmenting Hardware, Software, and Related Services for Surveillance, Security and Monitoring,9985
Software Solutions - Augmenting DIR-TSO-TMP-416,1482

上述数值总和为256098，传入原文件执行cat file.txt | grep 'Aug'得到的结果一致。另外，原文件读取后完整写入可保留900万行，说明原文件本身无问题。

已尝试的解决方法 #

• 将datawriter.writerows(gen)改为循环计数调用writerow，确认仅调用了256098次
• 修改CSV dialect参数
• 添加newline=''参数
  以上方法均未解决行数不符的问题

内容的提问来源于stack exchange，提问作者Mree