Python周期性读取后台子进程最新输出的最佳实践咨询

Hey there! Let's walk through the best practices for handling your subprocess output needs—your current code is a solid starting point, but we need to adjust it to avoid blocking issues and efficiently grab the latest lines every 3 seconds, while keeping memory usage in check.

Key Issues with Your Current Code #

Your p.stdout.readline() call is blocking—it will wait indefinitely for a new line of output, which means your while True loop won't run every 3 seconds as intended. Additionally, without proper buffering settings, you might not get real-time output from the subprocess, and storing all lines could risk memory bloat (though you're only printing them now, we'll fix that to track only the latest line).

Step-by-Step Solution #

Let's break this down into manageable parts, tailored for your Raspberry Pi/Linux environment.

1. Configure the Subprocess Correctly #

First, we need to launch the subprocess with settings that:

• Enable line-buffered output so we get lines as they're generated
• Merge stdout and stderr to avoid unhandled error output piling up
• Run in the background without tying up your Python terminal

2. Non-Blocking Output Reading #

We'll use the select module (Unix-only, perfect for your Pi) to check if there's new output available without blocking. This lets us read all pending lines, track the latest one, then wait 3 seconds before processing.

3. Cleanup & Error Handling #

We'll add checks to detect if the subprocess exits, and ensure we clean up properly to avoid zombie processes.

Final Working Code (Unix/Linux) #

import subprocess
import os
import time
import select

# Change to your target directory
os.chdir('/home/pi/Tutorials/RIOT/examples/gnrc_networking')

# Command to run
cmd = ["sudo", "BOARD=pba-d-01-kw2x", "make", "all", "flash", "term"]

# Launch subprocess with line buffering, merged stdout/stderr, text mode
p = subprocess.Popen(
    cmd,
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,  # Merge errors into stdout to avoid missing them
    bufsize=1,                 # Line-buffered output
    text=True,                 # Return strings instead of bytes
    start_new_session=True     # Optional: Isolate subprocess from terminal signals
)

try:
    while True:
        latest_line = None
        
        # Check if stdout has data available (non-blocking)
        ready, _, _ = select.select([p.stdout], [], [], 0)
        if ready:
            # Read ALL available lines, keep only the latest one
            for line in iter(p.stdout.readline, ''):
                latest_line = line.rstrip()  # Remove newline characters
        
        # Process the latest line every 3 seconds
        if latest_line is not None:
            print(f"Processing latest output: {latest_line}")
            # Add your custom processing logic here (e.g., parse, store, send)
        
        # Wait 3 seconds before next check
        time.sleep(3)
        
        # Check if the subprocess has exited
        if p.poll() is not None:
            print("Subprocess has finished running.")
            # Read any remaining output
            remaining_output = p.stdout.read()
            if remaining_output:
                print(f"Final remaining output:\n{remaining_output}")
            break

finally:
    # Ensure we clean up the subprocess to avoid zombies
    if p.poll() is None:
        p.terminate()
        p.wait()

What This Does #

• Non-blocking reads: The select call checks if there's new output without waiting, so our loop can stick to the 3-second interval.
• Memory efficiency: We only store the latest line of output, so you don't have to worry about accumulating gigabytes of data.
• Robust cleanup: The finally block ensures the subprocess is terminated if your script exits unexpectedly.
• Error handling: Merging stderr into stdout means you won't miss critical error messages from the make command.

Alternative: Asyncio Approach (Modern & Scalable) #

If you're working with other asynchronous tasks, using asyncio is a cleaner approach:

import asyncio
import subprocess
import os

os.chdir('/home/pi/Tutorials/RIOT/examples/gnrc_networking')
cmd = ["sudo", "BOARD=pba-d-01-kw2x", "make", "all", "flash", "term"]

async def track_latest_output(proc, shared_data):
    """Async task to read subprocess output and update the latest line"""
    while True:
        line = await proc.stdout.readline()
        if not line:
            break
        shared_data['latest_line'] = line.decode().rstrip()

async def main():
    # Launch subprocess
    proc = await asyncio.create_subprocess_exec(
        *cmd,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        start_new_session=True
    )
    
    # Shared object to track the latest line (mutable so task can update it)
    shared_data = {'latest_line': None}
    
    # Start the output tracking task
    output_task = asyncio.create_task(track_latest_output(proc, shared_data))
    
    try:
        while True:
            # Wait 3 seconds
            await asyncio.sleep(3)
            
            # Process the latest line if available
            if shared_data['latest_line'] is not None:
                print(f"Processing latest async output: {shared_data['latest_line']}")
                # Add your processing logic here
            
            # Check if subprocess or tracking task has finished
            if proc.returncode is not None or output_task.done():
                print("Subprocess exited.")
                break
    finally:
        # Cleanup
        if proc.returncode is None:
            proc.terminate()
            await proc.wait()

asyncio.run(main())

Important Notes #

• Sudo permissions: Make sure your user has passwordless sudo access for the make command (add a line like pi ALL=(ALL) NOPASSWD: /usr/bin/make to /etc/sudoers) to avoid your script getting stuck waiting for a password.
• Buffering: Never skip bufsize=1 or text=True—without these, the subprocess will buffer output in large chunks, and you won't get lines every second as expected.
• Zombie processes: Always terminate and wait for the subprocess in a finally block to prevent leftover processes from cluttering your system.

内容的提问来源于stack exchange，提问作者paranoia_do