如何快速将50000个TXT文件转换为CSV文件？

我有大量TXT文件，想转成单个CSV，但处理速度极慢——夜间跑代码，只处理了4500个文件，第二天早上还在运行。有没有快速转换的方法？

原代码：

import pandas as pd
import os
import glob
from tqdm import tqdm

# create empty dataframe
csvout = pd.DataFrame(columns =["ID","Delivery_person_ID" ,"Delivery_person_Age" ,"Delivery_person_Ratings","Restaurant_latitude","Restaurant_longitude","Delivery_location_latitude","Delivery_location_longitude","Order_Date","Time_Orderd","Time_Order_picked","Weather conditions","Road_traffic_density","Vehicle_condition","Type_of_order","Type_of_vehicle", "multiple_deliveries","Festival","City","Time_taken (min)"])

# get list of files

file_list = glob.glob(os.path.join(os.getcwd(), "train/", "*.txt"))

for filename in tqdm(file_list):
    # next file/record
    mydict = {}
    with open(filename) as datafile:
        # read each line and split on "  " space
        for line in tqdm(datafile):
            # Note: partition result in 3 string parts, "key", "   ", "value"
            # array slice third parameter [::2] means steps=+2
            # so only take 1st and 3rd item
            name, var = line.partition("   ")[::2]
            mydict[name.strip()] = var.strip()
        # put dictionary in dataframe
        csvout = csvout.append(mydict, ignore_index=True)

# write to csv
csvout.to_csv("train.csv", sep=";", index=False)

示例TXT文件内容：

ID                                     0xb379
Delivery_person_ID             BANGRES18DEL02
Delivery_person_Age                 34.000000
Delivery_person_Ratings              4.500000
Restaurant_latitude                 12.913041
Restaurant_longitude                77.683237
Delivery_location_latitude          13.043041
Delivery_location_longitude         77.813237
Order_Date                         25-03-2022
Time_Orderd                             19:45
Time_Order_picked                       19:50
Weather conditions                     Stormy
Road_traffic_density                      Jam
Vehicle_condition                           2
Type_of_order                           Snack
Type_of_vehicle                       scooter
multiple_deliveries                  1.000000
Festival                                   No
City                            Metropolitian
Time_taken (min)                    33.000000

你这代码慢的核心原因是频繁调用pd.DataFrame.append()——这个操作每次都会创建新的DataFrame，复制所有现有数据，数据量越大效率越低。另外嵌套的tqdm也会额外消耗资源，还有空行没跳过做了无效处理。下面是几个有效的优化方法：

1. 先收集所有数据再一次性生成DataFrame #

不要逐文件追加到DataFrame，改用列表存储所有文件的字典数据，最后统一转成DataFrame，这能把O(n²)的时间复杂度降到O(n)：

import pandas as pd
import os
import glob
from tqdm import tqdm

# 定义列名
COLUMNS = [
    "ID", "Delivery_person_ID", "Delivery_person_Age", "Delivery_person_Ratings",
    "Restaurant_latitude", "Restaurant_longitude", "Delivery_location_latitude",
    "Delivery_location_longitude", "Order_Date", "Time_Orderd", "Time_Order_picked",
    "Weather conditions", "Road_traffic_density", "Vehicle_condition", "Type_of_order",
    "Type_of_vehicle", "multiple_deliveries", "Festival", "City", "Time_taken (min)"
]

file_list = glob.glob(os.path.join(os.getcwd(), "train/", "*.txt"))
data_list = []  # 用列表存所有文件的字典数据

for filename in tqdm(file_list):
    row_dict = {}
    with open(filename, 'r', encoding='utf-8') as f:
        for line in f:
            line = line.strip()
            if not line:  # 跳过空行
                continue
            # 按第一个空格分割，处理多空格分隔的情况
            parts = line.split(maxsplit=1)
            if len(parts) == 2:
                key, value = parts
                row_dict[key.strip()] = value.strip()
    data_list.append(row_dict)

# 一次性生成DataFrame并导出
df = pd.DataFrame(data_list, columns=COLUMNS)
df.to_csv("train.csv", sep=";", index=False)

2. 用多进程并行处理（进阶加速） #

如果文件数量特别多，还可以用多进程并行读取处理文件，利用多核CPU提升速度：

import pandas as pd
import os
import glob
from tqdm import tqdm
from multiprocessing import Pool

COLUMNS = [
    "ID", "Delivery_person_ID", "Delivery_person_Age", "Delivery_person_Ratings",
    "Restaurant_latitude", "Restaurant_longitude", "Delivery_location_latitude",
    "Delivery_location_longitude", "Order_Date", "Time_Orderd", "Time_Order_picked",
    "Weather conditions", "Road_traffic_density", "Vehicle_condition", "Type_of_order",
    "Type_of_vehicle", "multiple_deliveries", "Festival", "City", "Time_taken (min)"
]

def process_single_file(filename):
    row_dict = {}
    with open(filename, 'r', encoding='utf-8') as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            parts = line.split(maxsplit=1)
            if len(parts) == 2:
                key, value = parts
                row_dict[key.strip()] = value.strip()
    return row_dict

if __name__ == "__main__":
    file_list = glob.glob(os.path.join(os.getcwd(), "train/", "*.txt"))
    
    # 启动多进程，进程数默认是CPU核心数
    with Pool() as pool:
        data_list = list(tqdm(pool.imap(process_single_file, file_list), total=len(file_list)))
    
    df = pd.DataFrame(data_list, columns=COLUMNS)
    df.to_csv("train.csv", sep=";", index=False)

额外优化点 #

• 去掉嵌套的tqdm，只保留外层遍历文件的进度条，减少开销
• 读取文件时指定编码（比如utf-8），避免潜在的编码问题
• 用split(maxsplit=1)替代partition，更灵活处理多空格分隔的场景
• 跳过空行，避免无效的字符串处理操作

内容的提问来源于stack exchange，提问作者jacky789