Python 2.7中Pass_fds替代方案：多内存数据转Unix命令参数

Alright, let's work through this problem. You're stuck on Python 2.7, needing to pass multiple in-memory data blobs to a Unix command that only accepts actual file paths (no stdin support), and you want alternatives to using pass_fds. Here are two practical, tested approaches that fit your constraints:

This is the straightforward go-to method—create temporary files for each in-memory blob, pass their paths to the command, then clean them up afterward. It works on every Unix-like system and requires minimal code complexity.

Example Code: #

import tempfile
import subprocess
import os

def run_cmd_with_mem_files(command, data_blobs):
    temp_paths = []
    try:
        # Create a temp file for each data blob (don't auto-delete yet)
        for blob in data_blobs:
            with tempfile.NamedTemporaryFile(delete=False) as tmp_file:
                tmp_file.write(blob)
                temp_paths.append(tmp_file.name)
        
        # Build the command with temp file paths as arguments
        full_cmd = [command] + temp_paths
        proc = subprocess.Popen(full_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        stdout, stderr = proc.communicate()
        
        return proc.returncode, stdout, stderr
    finally:
        # Clean up all temp files, even if the command fails
        for path in temp_paths:
            try:
                os.unlink(path)
            except OSError:
                # Ignore errors if the file was already deleted for some reason
                pass

Pros & Cons: #

• Pros: Dead simple to implement, no low-level Unix API knowledge needed, works with any command that accepts file paths.
• Cons: Involves disk I/O (not ideal for huge datasets), requires careful cleanup to avoid leaving temp files behind.

If you need to keep everything in memory (no disk writes), you can use Unix pipes paired with manual fork() and execvp() calls. This avoids relying on pass_fds and lets you pass /dev/fd/<fd_number> paths to the command, which it will treat as regular files.

Example Code: #

import os
import errno

def run_cmd_with_pipes(command, data_blobs):
    # Create a pipe for each data blob (read/write pairs)
    pipes = []
    for _ in data_blobs:
        read_fd, write_fd = os.pipe()
        pipes.append((read_fd, write_fd))
    
    pid = os.fork()
    if pid == 0:
        # Child process: close write ends, exec the command with /dev/fd paths
        try:
            for r_fd, w_fd in pipes:
                os.close(w_fd)
            
            # Build command arguments using /dev/fd for each pipe's read end
            cmd_args = [command] + ['/dev/fd/{}'.format(r_fd) for r_fd, _ in pipes]
            os.execvp(command, cmd_args)
        except Exception:
            # Ensure child exits on failure (don't throw exceptions here)
            os._exit(1)
    else:
        # Parent process: write data to pipes, clean up, wait for child
        try:
            for idx, (r_fd, w_fd) in enumerate(pipes):
                os.close(r_fd)
                try:
                    # Write the in-memory blob to the pipe
                    os.write(w_fd, data_blobs[idx])
                finally:
                    os.close(w_fd)
            
            # Wait for the child command to finish and get exit code
            _, status = os.waitpid(pid, 0)
            exit_code = os.WEXITSTATUS(status) if os.WIFEXITED(status) else -1
            return exit_code
        except Exception as e:
            # Clean up pipes if something goes wrong
            for r_fd, w_fd in pipes:
                try:
                    os.close(r_fd)
                except OSError:
                    pass
                try:
                    os.close(w_fd)
                except OSError:
                    pass
            # Kill the child process if it's still running
            try:
                os.kill(pid, 9)
                os.waitpid(pid, 0)
            except OSError as kill_err:
                if kill_err.errno != errno.ESRCH:
                    raise
            # Re-raise the original error
            raise

Pros & Cons: #

• Pros: 100% in-memory, no disk I/O, avoids pass_fds entirely.
• Cons: More complex code, requires understanding Unix fork/exec mechanics, error handling is more verbose.

内容的提问来源于stack exchange，提问作者Ell