Debian11下多线程启动Selenium ChromeDriver失败求助

我将在Windows和Ubuntu上运行正常的代码部署到Debian 11服务器后，使用concurrent.futures多线程启动Selenium ChromeDriver时频繁出现各类错误，最常见的两个错误如下：

常见错误1 #

Message: session not created: Chrome failed to start: exited normally.
  (session not created: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /opt/google/chrome/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Stacktrace:
#0 0x55cacb8b6d33 <unknown>
#1 0x55cacb573f87 <unknown>
#2 0x55cacb5ab87e <unknown>
#3 0x55cacb5a84de <unknown>
#4 0x55cacb5f8a90 <unknown>
#5 0x55cacb5ed0e3 <unknown>
#6 0x55cacb5b5044 <unknown>
#7 0x55cacb5b644e <unknown>
#8 0x55cacb87b861 <unknown>
#9 0x55cacb87f785 <unknown>
#10 0x55cacb869285 <unknown>
#11 0x55cacb88041f <unknown>
#12 0x55cacb84d20f <unknown>
#13 0x55cacb8a4028 <unknown>
#14 0x55cacb8a41f7 <unknown>
#15 0x55cacb8b5ed4 <unknown>
#16 0x1466c9716ea7 start_thread

常见错误2 #

Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/var/www/joyeria/joyeria.jewelry/image/catalog/productsimage/beta.py", line 16, in work
    driver = webdriver.Chrome(service=service, options=driver_options)
  File "/var/www/joyeria/.local/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
    super().__init__(
  File "/var/www/joyeria/.local/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 56, in __init__
    super().__init__(
  File "/var/www/joyeria/.local/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 208, in __init__
    self.start_session(capabilities)
  File "/var/www/joyeria/.local/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 292, in start_session
    response = self.execute(Command.NEW_SESSION, caps)["value"]
  File "/var/www/joyeria/.local/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 347, in execute
    self.error_handler.check_response(response)
  File "/var/www/joyeria/.local/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Failed to create Chrome process.
Stacktrace:
#0 0x564c78203d33 <unknown>
#1 0x564c77ec0f87 <unknown>
#2 0x564c77ef7c70 <unknown>
#3 0x564c77ef54de <unknown>
#4 0x564c77f45a90 <unknown>
#5 0x564c77f3a0e3 <unknown>
#6 0x564c77f02044 <unknown>
#7 0x564c77f0344e <unknown>
#8 0x564c781c8861 <unknown>
#9 0x564c781cc785 <unknown>
#10 0x564c781b6285 <unknown>
#11 0x564c781cd41f <unknown>
#12 0x564c7819a20f <unknown>
#13 0x564c781f1028 <unknown>
#14 0x564c781f11f7 <unknown>
#15 0x564c78202ed4 <unknown>
#16 0x14faadad0ea7 start_thread

当前使用的Driver初始化代码 #

def initialize_driver(proxy):
    driver_options = webdriver.ChromeOptions()
    if proxy:
        pluginfile = 'proxy_auth_plugin.zip'

        with zipfile.ZipFile(pluginfile, 'w') as zp:
            zp.writestr("manifest.json", MANIFEST_JSON)
            zp.writestr("background.js", background_js())
        driver_options.add_extension(pluginfile)
    # driver_options.binary_location = "/var/www/joyeria/joyeria.jewelry/image/catalog/productsimage/chromedriver"
    driver_options.add_argument("--disable-blink-features=AutomationControlled") 
    # driver_options.add_experimental_option("excludeSwitches", ["enable-automation"]) 
    driver_options.add_argument('--enable-automation')
    driver_options.add_experimental_option("useAutomationExtension", False)
    driver_options.add_argument('--no-sandbox')
    driver_options.add_argument('--headless=new')
    driver_options.add_argument('--disable-gpu')
    driver_options.add_argument('--disable-infobars')
    driver_options.add_argument('--disable-dev-shm-usage')
    # driver_options.add_argument('--remote-debugging-pipe')
    # driver_options.add_argument("--dns-prefetch-disable")
    # driver_options.add_argument("--disable-browser-side-navigation")
    # driver_options.add_argument("--remote-debugging-port=9222")
    # driver_options.add_argument(f"crash-dumps-dir={os.path.expanduser('~/tmp/Crashpad')}")
    driver_options.add_argument("--start-maximized")
    # driver_options.add_argument('--disable-application-cache')
    # driver_options.add_argument('--ignore-certificate-errors')
    driver_options.add_argument("--window-size=1920x1080")
    driver_options.add_experimental_option('excludeSwitches', ['enable-logging'])
    driver_options.add_argument("--log-level=3")

    chromedriver_path = "/var/www/joyeria/joyeria.jewelry/image/catalog/productsimage/chromedriver"

    service = Service(chromedriver_path)
    
    driver = webdriver.Chrome(service=service, options=driver_options)

    driver.maximize_window()
    
    return driver

当前线程启动代码 #

threads = []

with concurrent.futures.ThreadPoolExecutor(max_workers=avialable_threads) as executor:
        for store in store_links:
            threads.append(executor.submit(process_main, store, store_links, global_links, global_pass_links, threads_number))
            
        for thread in threads:
            try:
                thread.result()
            except Exception as exception:
                print(exception)

解决方案 #

1. 修正Chrome二进制文件路径配置 #

你当前注释掉了binary_location，但这里应该指向Chrome浏览器的二进制文件，而非chromedriver。Debian上Chrome默认路径通常为/opt/google/chrome/chrome，取消注释并修正：

driver_options.binary_location = "/opt/google/chrome/chrome"

2. 为多线程分配独立的用户数据目录 #

Chrome多线程运行时共享数据目录会导致冲突，需为每个线程创建独立临时目录：

import tempfile
import threading

# 在initialize_driver函数中添加
temp_dir = tempfile.mkdtemp(prefix=f"chrome_thread_{threading.get_ident()}_")
driver_options.add_argument(f"--user-data-dir={temp_dir}")
driver_options.add_argument("--profile-directory=Default")

注意：在driver退出时记得删除临时目录，避免磁盘占用：

# 在process_main函数中driver.close()后添加
import shutil
shutil.rmtree(temp_dir)

3. 调整线程池大小 #

Debian服务器资源（CPU、内存）有限，max_workers不宜过高，建议设置为服务器CPU核心数的1-2倍，避免资源耗尽导致Chrome启动失败。

4. 添加关键启动参数 #

补充以下参数解决DevToolsActivePort问题：

driver_options.add_argument("--remote-debugging-port=0")  # 自动分配空闲端口
driver_options.add_argument("--single-process")  # 单进程模式减少资源占用
driver_options.add_argument("--disable-features=VizDisplayCompositor")  # 适配无头模式

5. 检查权限与依赖 #

• 确保运行代码的用户对/tmp目录有读写权限（Chrome需要写入临时文件）。
• 安装Chrome依赖库：

apt-get install -y libnss3 libgconf-2-4 libxss1 libappindicator1 libindicator7 libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator3-1 libnss3 lsb-release xdg-utils

6. 避免代理插件文件冲突 #

多线程使用相同的proxy_auth_plugin.zip会导致文件占用，需为每个线程生成唯一插件文件：

if proxy:
    # 生成唯一的插件文件名
    pluginfile = f'proxy_auth_plugin_{threading.get_ident()}.zip'
    with zipfile.ZipFile(pluginfile, 'w') as zp:
        zp.writestr("manifest.json", MANIFEST_JSON)
        zp.writestr("background.js", background_js())
    driver_options.add_extension(pluginfile)
    # 后续删除插件文件
    import os
    os.remove(pluginfile)

内容的提问来源于stack exchange，提问作者ArtikTheOnlyOne