如何限制并行进程？求修改Python脚本实现每次仅启动3个Perl进程

Hey there! I get it—your current Python script fires off all those Perl processes at once, but you want to cap it at 3 running at a time, waiting for each batch to finish before starting the next. Let's get this sorted for you.

First, fix the directory list reading issue #

Your original script reads list.txt line by line, but your list.txt has all directories on a single line. We need to split them into individual entries first to handle each directory correctly.

Solution 1: Manual batch management (close to your original code) #

This approach keeps things familiar, adding logic to track and wait for batches of 3 processes:

import subprocess
from subprocess import Popen

# Read and split the directory list properly
with open('list.txt', 'r') as list_file:
    dirs = list_file.read().strip().split()

batch_size = 3
active_procs = []

for index, dir_name in enumerate(dirs, start=1):
    # Launch a single Perl process
    proc = Popen(['perl', 'test.pl', '-a', dir_name])
    active_procs.append(proc)
    
    # Wait for the batch to finish when we hit the limit, or it's the last entry
    if index % batch_size == 0 or index == len(dirs):
        for proc in active_procs:
            proc.wait()
        active_procs.clear()  # Reset for the next batch

Solution 2: Use concurrent.futures for cleaner code #

If you prefer a more streamlined approach, Python's concurrent.futures module handles concurrency limits automatically:

import subprocess
from concurrent.futures import ThreadPoolExecutor

def run_perl_script(dir_name):
    # Execute the Perl script and wait for completion
    subprocess.run(['perl', 'test.pl', '-a', dir_name], check=False)

# Load the directory list
with open('list.txt', 'r') as list_file:
    dirs = list_file.read().strip().split()

# Run with max 3 concurrent processes
with ThreadPoolExecutor(max_workers=3) as executor:
    executor.map(run_perl_script, dirs)

Quick check on your test.pl #

Just a heads-up: your Perl script captures $ARGV[0] (the -a flag) and $ARGV[1] (the directory name), which perfectly matches how your Python script passes arguments—so that part is working as intended.

内容的提问来源于stack exchange，提问作者kuck