多进程实现远程主机连接与本地网页捕获问题求助

Hey there, let's tackle that tricky final failure you're hitting with your multiprocessing, SSH port forwarding, and Selenium setup. You've already worked through most of the kinks, so let's focus on the common pitfalls that often trip up this exact workflow.

Key Issues & Fixes for Your Scenario #

First, let's recap your setup to make sure we're aligned: you're port-forwarding a remote web server's HTTP port to your local machine, using multiprocessing to handle connections, and trying to capture pages with Selenium. Here's what's likely going wrong, and how to fix it:

1. SSH Connections Aren't Multiprocess-Safe

Paramiko's SSHClient and transport objects don't play nice when shared across processes. If you're creating a single SSH connection in the main process and passing it to child processes, you'll run into connection corruption or unexpected drops.

Fix: Have each child process establish its own SSH connection and port forwarding. This keeps resources isolated and avoids race conditions.

2. Port Conflicts When Forwarding

If multiple processes try to bind to the same local port, you'll get an "address already in use" error—this is a super common hidden issue that can cause silent failures late in the workflow.

Fix: Use dynamic port allocation instead of hardcoding a local port. Let the OS assign an unused port, then retrieve it to pass to Selenium:

transport = ssh.get_transport()
# Bind to port 0 to let the OS pick an available port
channel = transport.request_port_forward('', 0, remote_host, remote_port)
actual_local_port = channel.get_local_port()  # Get the real port we're using

3. PyVirtualDisplay Shared Across Processes

The Display object from PyVirtualDisplay relies on X server resources, which aren't meant to be shared between processes. If you start a single display in the main process, child processes might not have access to it, leading to Selenium failing to launch the browser.

Fix: Start and stop a separate Display instance inside each child process. This ensures each worker has its own isolated display environment.

4. Missing Error Context

A lot of "unexplained" failures happen because exceptions are being swallowed. If your final step is failing, you might not be seeing the actual error (like a timeout connecting to the forwarded port, or a missing ChromeDriver).

Fix: Wrap your Selenium logic in a clear try/except block to capture and print detailed errors. This will tell you exactly why the final step is breaking.

Example Refactored Code #

Here's a cleaned-up version of your workflow that addresses all these issues:

from selenium import webdriver
from pyvirtualdisplay import Display
import paramiko
import multiprocessing

def process_remote_server(task):
    remote_host, remote_http_port = task
    ssh_username = "your-ssh-user"
    ssh_password = "your-ssh-pass"  # Or use SSH keys for better security
    chromedriver_path = "/absolute/path/to/chromedriver"

    # 1. Child process creates its own SSH connection
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    try:
        ssh.connect(remote_host, username=ssh_username, password=ssh_password)
    except Exception as e:
        print(f"Failed to connect to {remote_host}: {str(e)}")
        return

    # 2. Set up dynamic port forwarding
    try:
        transport = ssh.get_transport()
        channel = transport.request_port_forward("", 0, remote_host, remote_http_port)
        local_forwarded_port = channel.get_local_port()
    except Exception as e:
        print(f"Port forwarding failed for {remote_host}: {str(e)}")
        ssh.close()
        return

    # 3. Start child-specific virtual display
    display = Display(visible=0, size=(1920, 1080))
    display.start()

    # 4. Run Selenium capture with error handling
    driver = None
    try:
        driver = webdriver.Chrome(executable_path=chromedriver_path)
        driver.get(f"http://localhost:{local_forwarded_port}")
        # Add your page capture logic here (screenshot, source extraction, etc.)
        driver.save_screenshot(f"{remote_host}_page_capture.png")
        print(f"Successfully captured page for {remote_host}")
    except Exception as e:
        print(f"Final capture failed for {remote_host}: {str(e)}")
    finally:
        # 5. Clean up all resources in order
        if driver:
            driver.quit()
        display.stop()
        ssh.close()

if __name__ == "__main__":
    # List of remote servers to process
    remote_tasks = [("remote-server-01", 80), ("remote-server-02", 80)]
    
    # Use a process pool to handle parallel execution
    with multiprocessing.Pool(processes=2) as pool:
        pool.map(process_remote_server, remote_tasks)

Final Troubleshooting Tips #

• Check OS-specific multiprocessing behavior: On Windows, make sure all your code is wrapped in if __name__ == "__main__" to avoid infinite process spawning.
• Verify ChromeDriver compatibility: Ensure your ChromeDriver version matches the installed Chrome browser version on your machine.
• Test without multiprocessing first: Run the workflow for a single server without multiprocessing to confirm the SSH forwarding and Selenium capture work on their own. This helps isolate whether the issue is truly with multiprocessing.

内容的提问来源于stack exchange，提问作者Danny Watson