如何提升Python读取并转换300个XML文件的处理效率

First, let’s break down the obvious inefficiencies and bugs in your snippet that are contributing to slow performance:

• Incorrect tqdm usage: You’re passing the directory path /data directly to tqdm, which doesn’t iterate over files. You need to generate a list of XML files first, then wrap that iterator with tqdm.
• Variable scoping risk: Your function function_from_xml_pddataframe(xmlfile) relies on a global df_xml variable. This is error-prone (e.g., if the function fails to set df_xml, you’ll reuse the previous value) and can introduce unnecessary overhead. It’s far better to have the function return the DataFrame directly.
• Redundant file filtering: Using os.listdir and checking endswith(".xml") is less efficient than using glob.glob to directly fetch all XML files in the directory, avoiding the conditional check for every file.
• Potentially slow XML parsing: The biggest bottleneck is almost certainly the function_from_xml_pddataframe itself. If it uses a slow parser (like Python’s built-in xml.etree.ElementTree without optimizations) or builds the DataFrame incrementally row-by-row, that’s where most of your time is being wasted—especially since R’s XML packages are often optimized with C/C++ under the hood.

Let’s fix these issues and speed up your workflow:

1. Fix the Loop Structure & Variable Scoping #

First, rewrite the loop to use proper file iteration and eliminate global variables:

import os
import pandas as pd
from tqdm import tqdm
import glob

def xml_to_df(xml_path):
    # Replace your existing function here, but modify it to RETURN the DataFrame
    # Example structure: parse XML, collect data, return df

# Get all XML files upfront (avoids checking every file's extension)
xml_files = glob.glob("/data/*.xml")

list_of_dataframes = []
for file in tqdm(xml_files):
    df = xml_to_df(file)
    list_of_dataframes.append(df)

all_dfs = pd.concat(list_of_dataframes, ignore_index=True)

2. Optimize XML Parsing #

The xml_to_df function is where you’ll get the biggest speed gains. Here’s how to optimize it:

• Use lxml instead of xml.etree.ElementTree: lxml is a fast, C-based XML parser. Install it with pip install lxml, then use it for parsing.
• Avoid iterative row-by-row DataFrame construction: Building a DataFrame by appending rows one at a time is extremely slow. Instead, collect data in lists/dictionaries first, then create the DataFrame in one go.

Example optimized xml_to_df structure:

from lxml import etree

def xml_to_df(xml_path):
    # Parse the XML file quickly
    tree = etree.parse(xml_path)
    root = tree.getroot()
    
    # Collect data in lists (one list per column)
    col1_data = []
    col2_data = []
    # Add columns matching your XML structure
    
    for element in root.findall(".//your_target_element"):
        col1_data.append(element.findtext("col1_tag"))
        col2_data.append(element.findtext("col2_tag"))
        # Extract other fields as needed
    
    # Create DataFrame in a single step (far faster than incremental builds)
    return pd.DataFrame({
        "col1": col1_data,
        "col2": col2_data,
        # Map columns to your collected lists
    })

3. Parallel Processing #

Since each XML file is independent, you can process them in parallel to leverage multiple CPU cores. This will drastically reduce runtime. Use concurrent.futures.ProcessPoolExecutor for this:

import concurrent.futures

# Use max_workers equal to your CPU core count (or slightly less to avoid memory issues)
with concurrent.futures.ProcessPoolExecutor(max_workers=os.cpu_count()) as executor:
    # Map each XML file to the xml_to_df function, with progress tracking
    list_of_dataframes = list(tqdm(executor.map(xml_to_df, xml_files), total=len(xml_files)))

all_dfs = pd.concat(list_of_dataframes, ignore_index=True)

Note: Ensure your xml_to_df function is picklable (no global variables or non-serializable objects) for multiprocessing to work.

4. Additional Tips #

• Skip unnecessary data: If your XML files have fields you don’t need, skip them during parsing to reduce memory usage and speed up processing.
• Specify dtypes upfront: When creating DataFrames, define column data types (e.g., dtype={"col1": int, "col2": str}) to avoid pandas inferring types, which saves time.
• Batch processing if memory is tight: If concatenating all DataFrames at once uses too much memory, write intermediate DataFrames to disk (e.g., using to_parquet) and combine them later, or process files in batches.

R’s XML packages (like xml2) are often optimized with C/C++ under the hood, and R’s vectorized operations handle data collection efficiently. By applying the above optimizations—using a fast parser, avoiding incremental DataFrame builds, and parallel processing—you should be able to get Python’s runtime close to or even faster than R’s.

内容的提问来源于stack exchange，提问作者Laura