Selenium Webdriver Chrome线程过多，求助Python WallBoard WebApp故障排查

Hey Miguel, let's tackle this Selenium Chrome thread bloat issue head-on—super common when folks don't properly manage driver instances, especially in a persistent app like a WallBoard that's running continuously. Let's break this down step by step.

• Unclosed Driver Instances: If your code spins up a new webdriver.Chrome() every time you fetch call data (instead of reusing or properly quitting it), each instance spawns multiple background threads (ChromeDriver itself, browser processes, renderers, etc.) that hang around indefinitely if not terminated.
• Missing Context Management: Failing to use with statements or try/finally blocks means drivers might not get closed if an error occurs mid-operation, leaving stray threads behind.
• Polling Without Cleanup: If your WallBoard refreshes data periodically (which makes total sense for a support tracker), each refresh might be spawning a fresh driver instead of reusing one instance or cleaning up the old one first.

1. Reuse a Single Chrome Driver Instance #

Instead of initializing a new driver every time you pull call data, create one instance when your app starts and reuse it. This eliminates unnecessary thread spawning entirely.

Example code snippet:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# Initialize driver once at app startup
chrome_options = Options()
chrome_options.add_argument("--headless=new")  # Lighter modern headless mode
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-dev-shm-usage")  # Fixes resource limits on some systems

driver = webdriver.Chrome(options=chrome_options)

# Reuse the same driver for all call data fetches
def fetch_calls_waiting():
    driver.get("https://your-call-system-url.com/login")
    # Perform login steps, scrape call data, etc.
    return call_data

# Critical: Clean up when the app shuts down
def shutdown_wallboard():
    driver.quit()  # Kills the driver AND all associated threads/processes

2. Use Context Managers for Temporary Drivers (If Reuse Isn't Feasible) #

If you absolutely need a fresh driver session for each fetch (e.g., your call system enforces session limits), use a with statement to guarantee cleanup, even if an error hits mid-scrape:

def fetch_calls_waiting():
    chrome_options = Options()
    # Add your preferred flags here
    with webdriver.Chrome(options=chrome_options) as driver:
        driver.get("https://your-call-system-url.com/login")
        # Scrape and return data
        return call_data

The with block automatically runs driver.quit() when it exits—no manual cleanup needed.

3. Add Explicit Cleanup in Error Handling #

Wrap your driver usage in try/finally blocks to ensure quit() gets called no matter what goes wrong:

def fetch_calls_waiting():
    driver = None
    try:
        driver = webdriver.Chrome(options=chrome_options)
        driver.get("https://your-call-system-url.com/login")
        # Scrape data
        return call_data
    except Exception as e:
        print(f"Error fetching calls: {e}")
        # Handle the error appropriately
    finally:
        if driver is not None:
            driver.quit()  # Ensures driver is terminated even if an error occurs

4. Tweak Chrome Options for Lower Resource Usage #

Optimize Chrome's startup flags to reduce the number of threads/processes it spawns:

• --headless=new: Uses the lighter, modern headless mode (way better than the old --headless flag).
• --disable-extensions: Cuts out extra threads from unused extensions.
• --disable-plugins: Disables legacy plugins that add unnecessary overhead.
• --single-process: Runs Chrome in one process (not recommended for production, but can drastically reduce thread count—note: less stable).

5. Monitor and Kill Stray Processes (Last Resort) #

If you still see leftover threads after proper driver management, you can explicitly kill Chrome-related processes on shutdown (only use this if driver.quit() isn't working):

import os
import signal

def kill_stray_chrome_processes():
    # For Linux/macOS
    os.system("pkill -f chrome")
    os.system("pkill -f chromedriver")
    # For Windows (uncomment if needed)
    # os.system("taskkill /im chrome.exe /f")
    # os.system("taskkill /im chromedriver.exe /f")

Note: This will kill all Chrome instances on the machine, so use it carefully.

• Audit Polling Logic: Make sure your periodic data refreshes aren't spawning new drivers every time—reuse the same instance or clean up the old one first.
• Verify Database Connections: While your issue is with Selenium threads, double-check that database connections are being closed properly too—unclosed connections add to resource bloat.
• Test Outside VS Code: VS Code might clean up processes when debug sessions end, so run your app as a standalone script to confirm threads are actually persisting in a production-like environment.

If you can share snippets of your current driver initialization code, I can help tweak it further. Hope this gets that thread count under control!

内容的提问来源于stack exchange，提问作者Miguel Vale