非multiprocessing启动的独立进程如何共享队列实现日志复用？

Great question! When you're dealing with independent processes (not spawned via Python's multiprocessing module) that need to log to the same file without race conditions, an in-memory queue won't cut it—each process has its own isolated address space, so they can't share a regular queue.Queue directly.

Instead, you'll need an inter-process communication (IPC) mechanism that works across unrelated processes. Here are three practical, battle-tested approaches you can implement in Python:

1. Named Pipe (FIFO) #

Named pipes are a simple, filesystem-based IPC method that works on Linux/macOS and Windows. The core idea is:

• Spin up a dedicated logging daemon process that listens on the pipe
• All other processes write their log messages to the pipe
• The daemon reads from the pipe and writes to the log file safely (since only one process handles the file write)

Example Code: #

Logging Daemon (log_daemon.py)

import os
import time

PIPE_PATH = "/tmp/my_log_pipe"

# Create the named pipe if it doesn't exist
if not os.path.exists(PIPE_PATH):
    os.mkfifo(PIPE_PATH)

# Open the pipe in read mode (blocks until a writer connects)
with open(PIPE_PATH, "r") as pipe, open("app.log", "a") as log_file:
    print("Log daemon running, waiting for messages...")
    while True:
        # Read line by line (each log message should be a line)
        message = pipe.readline()
        if not message:
            # Pipe was closed, wait a bit and retry
            time.sleep(0.1)
            continue
        # Write to log file with timestamp (optional)
        log_file.write(f"{time.strftime('%Y-%m-%d %H:%M:%S')} - {message}")
        log_file.flush()  # Ensure it's written immediately

Client Process (any independent script)

import os

PIPE_PATH = "/tmp/my_log_pipe"

def log_message(message):
    # Open the pipe in write mode and send the message
    with open(PIPE_PATH, "w") as pipe:
        pipe.write(f"{message}\n")

# Usage
log_message("Hello from process 1!")

Note: On Windows, named pipes use a different path format (e.g., \\.\pipe\my_log_pipe). You'll need libraries like pywin32 for more granular control over Windows pipes, but the core logic remains similar.

2. Redis Queue (Cross-Platform & Scalable) #

If you're okay with running a Redis server, this is a fantastic scalable option. Redis lists act as natural queues, and all processes can connect to the same Redis instance to send/receive log messages. It even works across machines if your processes are distributed.

Example Code: #

Logging Daemon (redis_log_daemon.py)

import redis
import time

r = redis.Redis(host='localhost', port=6379, db=0)
LOG_QUEUE_KEY = "app_log_queue"

print("Redis log daemon running...")
while True:
    # Block until a message is available (BLPOP is blocking)
    _, message = r.blpop(LOG_QUEUE_KEY)
    message = message.decode('utf-8')
    with open("app.log", "a") as log_file:
        log_file.write(f"{time.strftime('%Y-%m-%d %H:%M:%S')} - {message}\n")
        log_file.flush()

Client Process

import redis

r = redis.Redis(host='localhost', port=6379, db=0)
LOG_QUEUE_KEY = "app_log_queue"

def log_message(message):
    r.rpush(LOG_QUEUE_KEY, message)

# Usage
log_message("Hello from a Redis client process!")

Pro tip: You can serialize log messages as JSON (e.g., json.dumps({"level": "INFO", "message": "..."})) for structured logging, then parse them in the daemon before writing to the file.

3. Unix Domain Sockets (Linux/macOS) #

Unix domain sockets are similar to network sockets but work locally on the filesystem, making them fast and secure for inter-process communication. They're a bit more code than named pipes but offer more flexibility (like sending structured data or handling multiple concurrent clients).

Example Code: #

Logging Daemon (socket_log_daemon.py)

import socket
import os
import time

SOCKET_PATH = "/tmp/my_log_socket"

# Clean up old socket if it exists
if os.path.exists(SOCKET_PATH):
    os.unlink(SOCKET_PATH)

# Create a Unix domain socket
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.bind(SOCKET_PATH)
sock.listen(1)

print("Socket log daemon running...")
with open("app.log", "a") as log_file:
    while True:
        conn, _ = sock.accept()
        with conn:
            while True:
                data = conn.recv(1024)
                if not data:
                    break
                message = data.decode('utf-8')
                log_file.write(f"{time.strftime('%Y-%m-%d %H:%M:%S')} - {message}\n")
                log_file.flush()

Client Process

import socket

SOCKET_PATH = "/tmp/my_log_socket"

def log_message(message):
    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    sock.connect(SOCKET_PATH)
    sock.sendall(message.encode('utf-8'))
    sock.close()

# Usage
log_message("Hello from a socket client!")

Final Quick Tips: #

• Start the logging daemon first before any client processes—otherwise, clients will fail to connect or write messages.
• Add error handling (e.g., retry logic if the pipe/socket/Redis is unavailable) to make your setup robust.
• For Windows environments, consider using Windows Named Sockets or the win32pipe library as alternatives to Unix domain sockets.

内容的提问来源于stack exchange，提问作者bb1950328