Python多进程队列能否传递给子进程？父进程发消息存数据可行吗？

Hey there, let's tackle your two questions about Python's multiprocessing Queue one by one!

Absolutely—this is actually one of the standard, recommended ways to handle inter-process communication (IPC) with Python's multiprocessing module. Queues are built to be process-safe, so passing them to child processes works seamlessly without extra hoops.

Here's a quick, straightforward example to show how it's done:

from multiprocessing import Process, Queue

def worker(child_queue):
    # Child process sends data back to parent via the queue
    child_queue.put("Hello from the child process!")

if __name__ == "__main__":
    parent_queue = Queue()
    # Pass the queue as an argument when starting the child process
    child_process = Process(target=worker, args=(parent_queue,))
    child_process.start()
    # Parent process retrieves data from the queue
    print(parent_queue.get())
    child_process.join()

The queue handles all underlying serialization and synchronization automatically, so you don't have to worry about race conditions or data corruption when passing it between processes.

This isn't just possible—it's a far better approach than sending large datasets from child to parent! Moving big chunks of data across processes requires pickling/unpickling, which is slow, memory-heavy, and inefficient. Having the child save its own data directly avoids all that overhead entirely.

Here's a practical example tailored to your data collection scenario:

from multiprocessing import Process, Queue
import time
import pickle
from queue import Empty

def data_collector(control_queue):
    collected_data = []
    # Simulate ongoing data collection loop
    while True:
        # Check for messages from the parent without blocking the collection work
        try:
            msg = control_queue.get(block=False, timeout=0.1)
            if msg == "SAVE_AND_EXIT":
                # Child saves data locally instead of transmitting it to parent
                with open("collected_data.pkl", "wb") as f:
                    pickle.dump(collected_data, f)
                print("Child process saved data successfully and exiting!")
                break
        except Empty:
            # No message available, keep collecting data
            pass
        
        # Simulate collecting a single data point
        collected_data.append(f"Data point {len(collected_data) + 1}")
        time.sleep(0.05)

if __name__ == "__main__":
    control_queue = Queue()
    collector_process = Process(target=data_collector, args=(control_queue,))
    collector_process.start()

    # Let the collector run for a few seconds to gather data
    time.sleep(2)
    # Send the save command to the child process
    control_queue.put("SAVE_AND_EXIT")

    collector_process.join()
    print("Parent process completed!")

A few quick tips to optimize this workflow:

• Use non-blocking get() with a short timeout (or catch the Empty exception) so the child doesn't waste CPU cycles waiting for messages when it should be collecting data.
• If your data collection is CPU-intensive, spawn a small dedicated thread inside the child process to listen for queue messages—this way, the main collection loop isn't interrupted.
• Choose efficient storage formats: Pickle works for Python objects, but for numerical data, numpy.save() or Parquet files are faster and more space-efficient.

内容的提问来源于stack exchange，提问作者Adam