能否用NumPy替代内置列表提升大文件数据处理代码效率？

Absolutely! NumPy's vectorized operations are perfect for accelerating this kind of data transformation—Python loops get really slow when dealing with 1GB+ files, so shifting to NumPy's C-backed array operations will make a huge difference. Let's first cover why your current code struggles, then walk through an optimized NumPy-based solution.

Why Your Current Code Is Slow #

• Full File Loading: f.readlines() loads the entire 1GB file into memory at once, which puts unnecessary strain on your system and slows down initial processing.
• Nested Python Loops: The inner loop over each coordinate point is pure Python, which is far less efficient than the vectorized operations NumPy provides (these run at C-level speed instead of interpreted Python speed).

Optimized NumPy Solution #

We'll combine batch processing (to avoid loading the whole file into memory) with NumPy vectorization (to replace the inner loop). Here's the code:

import numpy as np

def process_large_file(input_path, output_path, batch_size=1000):
    with open(input_path, "r") as infile, open(output_path, "w") as outfile:
        while True:
            # Read lines in batches to limit memory usage
            batch_lines = [infile.readline() for _ in range(batch_size)]
            # Exit loop if we've reached the end of the file
            if not any(line.strip() for line in batch_lines):
                break
            
            processed_batch = []
            for line in batch_lines:
                line = line.strip()
                if not line:
                    continue
                # Split line into ID and coordinate strings
                parts = [segment.strip() for segment in line.split(",")]
                obj_id = parts[0]
                # Collect all coordinate points as integer lists
                point_list = []
                for point_str in parts[1:]:
                    if point_str:
                        point_list.append(list(map(int, point_str.split())))
                
                if not point_list:
                    processed_batch.append(f"{obj_id}\n")
                    continue
                
                # Convert points to NumPy array for vectorized operations
                points_array = np.array(point_list, dtype=np.int32)
                # Calculate latitudes and longitudes in one go (no loop!)
                latitudes = points_array[:, 1] / 10000.0
                longitudes = points_array[:, 0] / 1000.0
                
                # Convert coordinates to strings
                coord_strings = [f"{lat} {lon}" for lat, lon in zip(latitudes, longitudes)]
                # Build the final line and add to batch
                processed_line = ",".join([obj_id] + coord_strings) + "\n"
                processed_batch.append(processed_line)
            
            # Write the entire batch at once to minimize I/O calls
            outfile.writelines(processed_batch)

# Run the processing
process_large_file("data", "output")

Key Optimizations #

1. Batch Processing: We read batch_size lines at a time instead of the whole file, keeping memory usage low even for 1GB+ files. You can adjust batch_size based on your system's memory (larger batches mean fewer I/O calls, but use more memory).
2. NumPy Vectorization: The inner loop that calculated latitude/longitude is replaced with a single array operation. This is the biggest speed gain—NumPy handles the iteration in optimized C code, which is orders of magnitude faster than pure Python loops for large datasets.
3. Batch Writing: We collect processed lines in a batch and write them all at once, reducing the number of disk I/O operations (which are inherently slow).

Optional: Further Optimize String Conversion #

If you want to push speed even more, you can use NumPy's built-in string operations to generate coordinate strings instead of list comprehension:

# Format latitudes and longitudes as strings
lat_str = np.char.mod("%.4f", latitudes)
lon_str = np.char.mod("%.3f", longitudes)
# Combine into "lat lon" strings
coord_strings = np.char.add(lat_str, " " + lon_str)
# Convert NumPy array to Python list for joining
coord_strings = coord_strings.tolist()

This keeps more of the work in NumPy's optimized layer, though the difference might be small compared to the loop replacement.

Notes #

• Test with a small sample file first to make sure the output matches your original code's results.
• If every line in your file has the same number of coordinate points, you could use np.genfromtxt to load the entire file (or batches) into a structured array for even more speed—though the above code works for variable numbers of points per line, which is more flexible.

内容的提问来源于stack exchange，提问作者Kakaa