如何高效从大体积半结构化文本生成Pandas DataFrame？

需求目标 #

需要从可变宽度格式的文本文件生成Pandas DataFrame，文件可达数GB规模（超百万行），最终DataFrame包含约140列，目标结构示例如下：

已尝试方案 #

编写了如下Python函数，核心思路是逐行将文本转为以列名为键的字典，累计指定行数后转为带稀疏数据类型的子DataFrame，最后合并所有子DataFrame以控制内存占用：

def generate_data_frame(data: TextIO, delimiter: list, save_point: int) -> pd.DataFrame:
    """generate_data_frame takes a text file pointer and builds a Pandas data frame out of it

    Args:
        data (TextIO): a python file pointer with lines containing data
                        separated by the delimiters
        delimiter (list): char separating the data in a line
        save_point (int): number of lines from which the data is transformed to
                        a Pandas data frame. Regulates the memory usage

        Returns:
            pandas.DataFrame: containing data in tabular form with columns
                        casted as sparse data type
    """
    # 存储每行对应的字典
    line_dictionaries = []
    # 累计指定行数后转为稀疏子DataFrame，存入列表待合并
    list_of_sub_data_frames = []

    for i, line in enumerate(data):
        line_dict = {}
        # 先用第一个分隔符拆分字段
        columns = line.split(delimiter[0])
        for column in columns:
            column_parts = column.split(delimiter[1])
            # 拆分后第一部分是列名，第二部分是值
            try:
                value = column_parts[1].rstrip("\n")
            except IndexError as e:
                # 处理空行或格式错误行
                raise IndexError(
                    f"第{i}行无法拆分为列名和对应值"
                ) from e
            line_dict[column_parts[0]] = value
        line_dictionaries.append(line_dict)
        # 达到指定行数时生成子DataFrame
        if len(line_dictionaries) >= save_point:
            logging.info("保存数据")
            sub_data_frame = pd.DataFrame.from_dict(
                line_dictionaries, dtype=pd.SparseDtype(object, np.nan)
            )
            list_of_sub_data_frames.append(sub_data_frame)
            line_dictionaries = []
    # 处理剩余行数
    sub_data_frame = pd.DataFrame.from_dict(
        line_dictionaries, dtype=pd.SparseDtype(object, np.nan)
    )
    list_of_sub_data_frames.append(sub_data_frame)
    # 合并所有子DataFrame
    data_frame = pd.concat(list_of_sub_data_frames)
    # 将带单位的列转为浮点稀疏类型
    columns_with_unit = [
        column for column in data_frame.columns if "None" not in column
    ]
    data_frame[columns_with_unit] = data_frame[columns_with_unit].astype(
        pd.SparseDtype(np.float64, np.nan)
    )
    return data_frame

现存问题 #

上述代码可正常运行，但处理速度极慢，现寻求更快速、内存更高效的DataFrame生成方案，不局限于使用Pandas工具。

内容的提问来源于stack exchange，提问作者tobi-92