Docker部署Selenium Wire/Grid配置代理时遇错误求助

核心问题分析 #

1. 版本不兼容：使用的selenium==4.0.0与Selenium Grid的4.10.0版本差异过大，导致协议交互出错
2. 代理配置冲突：同时通过Chrome的--proxy-server和Selenium Wire的seleniumwire_options配置代理，导致Chrome无法正确路由请求
3. 容器网络限制：Selenium Wire默认只监听本地回环地址，Chrome节点容器无法访问python-app容器内的代理服务

修复步骤 #

1. 对齐Selenium版本 #

修改requirements.txt，将selenium版本升级到与Grid一致的4.10.0：

selenium==4.10.0
selenium-wire==5.1.0
blinker==1.7.0
setuptools==74.0.0
requests
fake_useragent==1.5.1

2. 调整Python代码中的代理配置 #

移除Chrome手动代理参数，配置Selenium Wire允许跨容器访问，并通过CDP命令让Chrome指向python-app容器的代理服务：

print("Docker gestartet") 
import time 
import socket
from seleniumwire import webdriver 
from selenium.webdriver.chrome.options import Options 
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities 
from selenium.webdriver.common.by import By 
import concurrent 
from fake_useragent import UserAgent 
from random import random 
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC

def call_function(): 
    driver = None
    try:
        # 替换为你的代理账号密码
        PROXY = f"http://USER:PW@de.smartproxy.com:20000"    
        
        desired_capabilities = DesiredCapabilities.CHROME.copy()
        chrome_options = Options()
        chrome_options.add_argument('--no-sandbox')
        chrome_options.add_argument('--disable-dev-shm-usage')
        
        # Selenium Wire配置：允许跨容器访问，设置目标代理
        seleniumwire_options = {
            'auto_config': False,
            'addr': '0.0.0.0',  # 监听所有网卡，允许Chrome容器访问
            'proxy': {
                'http': PROXY,
                'https': PROXY,
                'no_proxy': 'selenium-hub'  # 访问Grid Hub时不走代理，避免循环
            }
        }
        
        driver = webdriver.Remote(
            command_executor="http://selenium-hub:4444/wd/hub",
            options=chrome_options, 
            seleniumwire_options=seleniumwire_options,
            desired_capabilities=desired_capabilities
        )
        
        # 获取当前python-app容器的IP，让Chrome连接这个IP的Selenium Wire代理
        python_app_ip = socket.gethostbyname(socket.gethostname())
        driver.execute_cdp_cmd('Network.setProxy', {
            'proxyType': 'manual',
            'httpProxy': f"{python_app_ip}:8080",
            'httpsProxy': f"{python_app_ip}:8080",
            'noProxy': 'selenium-hub'
        })
        
        print("----------------- 当前代理配置 -------------------------------------------")
        print(driver.proxy)
        print("------------------------------------------------------------")
        
        driver.get("https://ip.smartproxy.com/json")
        wait = WebDriverWait(driver, 50)
        pre_element = wait.until(EC.presence_of_element_located((By.TAG_NAME, "pre")))
        res = pre_element.text
        print("----------------- DRIVER 返回结果 -----------------------------------")
        print(res)
        print("------------------------------------------------------------")
        return "", ""
        
    except Exception as e: 
        print(f"错误详情: {str(e)}")
        
    finally: 
        if driver:
            driver.quit()
        
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    futures = []
    for i in range(0, 5):
        print(f"网页索引 {i}")
        future = executor.submit(call_function)
        futures.append(future)
        time.sleep(1)
        
for future in futures:
    try:
        ipv4_value, ipv6_value = future.result()
        print(f"IPv4: {ipv4_value}, IPv6: {ipv6_value}")
    except Exception as e:
        print(f"任务执行失败: {str(e)}")

3. 关键配置说明 #

• addr: 0.0.0.0：解决容器间网络访问问题，让Selenium Wire代理服务能被Chrome节点容器访问
• CDP命令配置代理：在Remote模式下，Selenium Wire的自动代理配置可能失效，通过CDP直接告诉Chrome使用python-app容器的代理
• no_proxy: selenium-hub：避免Chrome访问Grid Hub时走代理，防止请求循环
• 版本对齐：消除Selenium客户端与Grid之间的协议兼容性问题

内容的提问来源于stack exchange，提问作者T0getherAl0ne