Python多文件读取优化：300k+文件存入字典提速方案咨询

Hey there! Dealing with 300k+ files is no trivial task—1+ hours is definitely too slow. Let’s walk through practical, proven optimizations to cut down that runtime drastically:

1. Parallelize File Reads (Biggest Win!) #

File I/O is IO-bound, meaning your CPU sits idle most of the time waiting for data from disk. Parallel processing lets you utilize multiple CPU cores to read multiple files at once.

In Python, concurrent.futures.ProcessPoolExecutor is ideal here (multiprocessing avoids GIL limitations better than threads for IO-bound work). Here’s a quick example:

import os
from concurrent.futures import ProcessPoolExecutor

def process_single_file(file_path):
    # Customize this based on your file type (text/binary)
    with open(file_path, 'r', buffering=1024*1024) as f:
        content = f.read()
    # Use a meaningful key (e.g., filename, hash, or custom identifier)
    key = os.path.basename(file_path)
    return (key, content)

def build_file_dict(file_paths):
    file_dict = {}
    # Use 2-4x your CPU core count for optimal IO utilization
    with ProcessPoolExecutor(max_workers=os.cpu_count() * 2) as executor:
        # Map all file paths to the processing function
        for key, content in executor.map(process_single_file, file_paths):
            file_dict[key] = content
    return file_dict

# Replace with your actual dataset directory
dataset_dir = "/path/to/your/300k_files"
file_paths = [os.path.join(dataset_dir, fname) for fname in os.listdir(dataset_dir) if os.path.isfile(os.path.join(dataset_dir, fname))]
final_dict = build_file_dict(file_paths)

2. Reduce Disk IO Overhead #

• Use an SSD: If you’re on an HDD, switching to an SSD can cut read times by 70-90%—this is often the cheapest and most impactful fix.
• Tweak Buffering: For text files, increase the buffering parameter in open() (e.g., buffering=1024*1024 for 1MB buffers) to reduce the number of system calls.
• Avoid Small File Overhead: If your files are tiny (KB-sized), consider bundling them into a single archive (like a ZIP or Parquet file) and reading in bulk—though this adds a preprocessing step.

3. Optimize Dictionary Operations #

• Precompute Keys: Don’t generate keys inside the file-reading loop (e.g., parsing filenames every time). Precompute all keys from your file paths first to save redundant work.
• Avoid Dynamic Resizing: While Python dictionaries handle dynamic growth well, if you know the exact number of files, initialize a dict with dict.fromkeys(precomputed_keys) to avoid internal resizing overhead (minor gain, but worth it for large datasets).

4. Use Efficient Reading Methods #

• Memory Mapping for Large Files: For big binary/text files, use mmap to map the file directly into memory instead of reading it all at once. This reduces data copying between kernel and user space:

  import mmap
  
  def read_large_file(file_path):
      with open(file_path, 'r') as f:
          with mmap.mmap(f.fileno(), length=0, access=mmap.ACCESS_READ) as mm:
              content = mm.read()
      return content
  

• Skip Unnecessary Processing: If you only need a subset of each file’s content (e.g., first 10 lines, a specific field), process it during reading instead of storing the entire file in the dictionary—this saves both time and memory.

5. Miscellaneous Tips #

• Batch Processing: If memory is an issue, split your file list into batches (e.g., 10k files per batch), process each batch, and merge the dictionaries incrementally.
• Check for Bottlenecks: Use tools like cProfile to identify where your time is going—maybe it’s not the reading itself, but slow key generation or data parsing.

内容的提问来源于stack exchange，提问作者John Constantine