PowerShell脚本优化：将大量小文件（<2MB）有序整理至文件夹

Alright, let's tackle this script optimization problem step by step—dealing with 200k+ 2MB files is no small feat, so every tweak counts. Here's a breakdown of actionable optimizations, including sorting strategies and performance gains you can implement right away:

Most slowdowns here come from unnecessary I/O calls. Let's fix that first:

• Use os.scandir() instead of os.listdir(): scandir returns file objects with pre-fetched metadata (like is_file, size) instead of just filenames, which cuts down on repeated stat calls. It's significantly faster for large directories.

  import os
  file_paths = []
  with os.scandir("/your/source/dir") as entries:
      for entry in entries:
          if entry.is_file():
              file_paths.append(entry.path)
  

• Minimize folder creation checks: If you're creating folders on the fly, avoid checking "does this folder exist?" for every batch. Instead, precompute all folder names first, create them in bulk, or use os.makedirs(..., exist_ok=True) which is atomic and avoids extra checks.
• Avoid cross-partition operations: If your source and target folders are on different disk partitions, moving files becomes a copy+delete operation (way slower than just updating metadata on the same partition). Keep everything on the same drive if possible.

Regex is powerful but adds overhead—here's how to optimize sorting:

• Use natural/value-based sorting if filenames have structured numbers: If your filenames follow patterns like file_00123.txt or data-456.csv, extract the numeric part directly without regex. This is way faster than compiling/matching regex for every file.

  def sort_key(file_path):
      filename = os.path.basename(file_path)
      # Adjust this to match your filename structure
      numeric_part = filename.split("_")[1].split(".")[0]
      return int(numeric_part)
  
  file_paths.sort(key=sort_key)
  

• Precompile regex if you must use it: If your filenames are unstructured and require regex, compile the pattern once outside your sorting loop (not inside the key function) to avoid redundant compilation.

  import re
  # Compile once at the start
  filename_pattern = re.compile(r"(\d{5})")
  
  def regex_sort_key(file_path):
      match = filename_pattern.search(os.path.basename(file_path))
      return int(match.group(1)) if match else 0
  
  file_paths.sort(key=regex_sort_key)
  

Compression is often a bottleneck—optimize it without losing too much space:

• Use faster compression algorithms: If you don't need maximum compression, switch to faster settings. For example:
  • With zipfile in Python: Use compresslevel=1 (fastest deflate) instead of the default level 6.
  • Use zstandard or lz4 instead of gzip/deflate—these algorithms offer way better speed for similar or better compression ratios. Libraries like zstandard for Python make this easy.
• Batch files into compression commands: If using external tools (like 7z or tar), pass all files in a batch to a single command instead of adding files one by one. This avoids repeated process startup overhead.

  # Faster: pass all files in a batch to 7z
  7z a -mx1 batch_001.7z /path/to/files/*.txt
  

File operations are I/O-bound, so parallelizing work can cut runtime drastically:

• Use multiprocessing for batch compression: Python's concurrent.futures.ProcessPoolExecutor works great here—spawn a few processes (match your CPU core count, e.g., 4-8) to handle separate batches simultaneously.

  from concurrent.futures import ProcessPoolExecutor
  import zipfile
  
  def compress_batch(batch, output_dir, batch_num):
      zip_name = f"batch_{batch_num:03d}.zip"
      with zipfile.ZipFile(os.path.join(output_dir, zip_name), "w", zipfile.ZIP_DEFLATED, compresslevel=1) as zf:
          for file in batch:
              zf.write(file, os.path.basename(file))
      # Optional: Delete original files after compression if needed
  
  # Split files into batches (adjust max_files to your limit)
  max_files = 1000
  batches = [file_paths[i:i+max_files] for i in range(0, len(file_paths), max_files)]
  
  # Run in parallel
  with ProcessPoolExecutor(max_workers=4) as executor:
      for idx, batch in enumerate(batches):
          executor.submit(compress_batch, batch, "/your/output/dir", idx)
  

• Avoid over-parallelizing: Don't spawn more processes than your system can handle—too many concurrent I/O operations will cause disk thrashing and slow everything down. Stick to 1-2 processes per CPU core.

• Disable real-time file monitoring: Tools like Windows Search Indexer or macOS Spotlight can slow down file operations temporarily. Disable them for the duration of your script run if possible.
• Use generators for large file lists: If even storing 200k file paths in memory is a concern, use generators to process files in chunks without loading everything at once.
• Pre-check disk space: Nothing kills performance like a mid-run disk full error—add a quick check at the start to ensure your target drive has enough space.

内容的提问来源于stack exchange，提问作者Processor