如何通过多线程加速大型目录树文件扫描（优化os.walk耗时）

Great question—when dealing with massive directory trees (20k+ folders), IO-bound tasks like file scanning are perfect candidates for parallelization. Python's GIL isn't a bottleneck here because most of the time is spent waiting on disk IO, not CPU. Let's walk through how to implement your hierarchical threading approach to cut down that 6-minute scan time.

Core Idea #

Instead of scanning the entire tree sequentially with os.walk, we'll parallelize the work by spawning threads for each subfolder at every level:

1. Start with your top-level directory (\top).
2. Scan the top directory for target files first.
3. Spin up threads to handle each direct subfolder of \top.
4. Repeat the process recursively in each thread: scan the current folder's files, then launch threads for its subfolders.

Implementation with ThreadPoolExecutor #

We'll use concurrent.futures.ThreadPoolExecutor to manage our threads (it handles thread lifecycle and avoids overloading the system) and os.scandir() (faster than os.listdir() because it caches file metadata, cutting down on extra system calls).

Basic Version (Print Found Files) #

import os
from concurrent.futures import ThreadPoolExecutor

# Tune these based on your system: IO-bound tasks can use more threads
MAX_WORKERS = 20
# Replace with your target file patterns/extensions
TARGET_FILE_PATTERNS = ('.txt', '.log', '.csv')

def scan_directory(path):
    # Scan current directory for target files
    with os.scandir(path) as entries:
        for entry in entries:
            if entry.is_file(follow_symlinks=False) and entry.name.endswith(TARGET_FILE_PATTERNS):
                print(f"Found: {entry.path}")
    
    # Collect all direct subfolders
    subdirs = []
    with os.scandir(path) as entries:
        for entry in entries:
            if entry.is_dir(follow_symlinks=False):
                subdirs.append(entry.path)
    
    # Parallelize scanning of subfolders
    with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
        executor.map(scan_directory, subdirs)

if __name__ == "__main__":
    top_directory = r"\top"  # Replace with your actual top path
    scan_directory(top_directory)

Improved Version (Thread-Safe Result Collection) #

If you need to collect all found file paths instead of just printing them, use a thread-safe container with a lock to avoid race conditions:

import os
import threading
from concurrent.futures import ThreadPoolExecutor

MAX_WORKERS = 20
TARGET_FILE_PATTERNS = ('.txt', '.log', '.csv')
found_files = []
file_lock = threading.Lock()

# Use a global thread pool to avoid repeated setup/teardown
executor = ThreadPoolExecutor(max_workers=MAX_WORKERS)

def scan_directory(path):
    local_matches = []
    
    # Scan current folder for targets
    with os.scandir(path) as entries:
        for entry in entries:
            if entry.is_file(follow_symlinks=False) and entry.name.endswith(TARGET_FILE_PATTERNS):
                local_matches.append(entry.path)
    
    # Safely add local matches to global list
    with file_lock:
        found_files.extend(local_matches)
    
    # Collect subfolders and submit to thread pool
    subdirs = []
    with os.scandir(path) as entries:
        for entry in entries:
            if entry.is_dir(follow_symlinks=False):
                subdirs.append(entry.path)
    
    for subdir in subdirs:
        executor.submit(scan_directory, subdir)

if __name__ == "__main__":
    top_directory = r"\top"
    scan_directory(top_directory)
    executor.shutdown(wait=True)  # Wait for all threads to finish
    
    print(f"Total target files found: {len(found_files)}")
    # Process your found_files list here (e.g., write to a CSV)

Key Optimizations & Notes #

1. Thread Count Tuning: Don't set MAX_WORKERS too high. For mechanical HDDs, 10-20 threads is ideal (disk can't handle too many concurrent reads). For SSDs, you can go higher (30-50), but test to find the sweet spot—too many threads cause IO contention and slow things down.
2. os.scandir() vs os.listdir(): scandir is 2-10x faster for directory traversal because it retrieves file attributes (like whether it's a folder) during the initial scan, avoiding extra os.path calls.
3. Avoid Symlink Loops: The follow_symlinks=False flag prevents the script from following symbolic links, which could lead to infinite loops or scanning unintended directories.
4. Multiprocessing vs Threading: For IO-bound tasks like this, threading is better than multiprocessing. Multiprocessing has higher overhead (process creation, memory duplication) and won't give you better performance than threads here.

Expected Performance Gain #

With this approach, you should see a significant reduction in scan time—how much depends on your disk speed and thread count, but cutting the 6-minute runtime by 50-75% is realistic for most setups.

内容的提问来源于stack exchange，提问作者user9594752