如何用undetected-chromedriver+seleniumwire连接ScraperAPI认证代理？

问题分析 #

undetected-chromedriver（简称uc）初始化时会修改Chrome启动参数，容易和seleniumwire的代理配置产生冲突，导致代理未生效；同时ScraperAPI的带认证代理需要特定配置逻辑，才能在uc环境下正常工作。

可行解决方案 #

方案1：调整seleniumwire配置兼容uc #

通过拆分代理认证信息、禁用seleniumwire自动配置，同时给ChromeOptions显式指定代理，避免uc覆盖设置：

API_KEY = 'my_key_here'

proxy_options = {
    'proxy': {
        'http': 'http://proxy-server.scraperapi.com:8001',
        'https': 'http://proxy-server.scraperapi.com:8001',
        'no_proxy': 'localhost,127.0.0.1',
        'username': 'scraperapi',
        'password': API_KEY
    },
    'auto_config': False  # 禁用自动配置，避免和uc冲突
}

chrome_options = uc.ChromeOptions()
chrome_options.add_argument('--ignore-ssl-errors=yes')
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--proxy-server=http://proxy-server.scraperapi.com:8001')  # 显式指定代理

driver = uc.Chrome(
    options=chrome_options,
    version_main=137,
    seleniumwire_options=proxy_options
)

方案2：用Chrome扩展处理代理认证 #

如果seleniumwire和uc冲突无法调和，可通过Chrome扩展实现带认证的代理配置：

1. 创建代理配置文件proxy_auth.json：

{
    "username": "scraperapi",
    "password": "your_api_key_here",
    "proxy": {
        "host": "proxy-server.scraperapi.com",
        "port": 8001
    }
}

2. 代码中生成并加载扩展：

import os
import zipfile
import json
from undetected_chromedriver import Chrome, ChromeOptions

def create_proxy_extension(proxy_auth):
    extension_dir = 'proxy_extension'
    os.makedirs(extension_dir, exist_ok=True)
    
    # 编写manifest文件
    manifest = {
        "version": "1.0.0",
        "manifest_version": 3,
        "name": "Proxy Auth Tool",
        "permissions": ["proxy", "storage", "tabs", "webRequest", "webRequestBlocking"],
        "background": {"service_worker": "background.js"}
    }
    with open(f'{extension_dir}/manifest.json', 'w') as f:
        json.dump(manifest, f)
    
    # 编写背景脚本处理代理和认证
    background_js = f"""
    chrome.proxy.settings.set({{value: {{
        mode: "fixed_servers",
        rules: {{singleProxy: {{scheme: "http", host: "{proxy_auth['proxy']['host']}", port: {proxy_auth['proxy']['port']}}}}}
    }}, scope: "regular"}}, () => {{}});
    
    chrome.webRequest.onAuthRequired.addListener(
        (details, callback) => {{
            callback({{authCredentials: {{username: "{proxy_auth['username']}", password: "{proxy_auth['password']}"}}}});
        }},
        {{urls: ["<all_urls>"]}},
        ["blocking"]
    );
    """
    with open(f'{extension_dir}/background.js', 'w') as f:
        f.write(background_js)
    
    # 打包成zip扩展文件
    zip_path = 'proxy_extension.zip'
    with zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED) as zf:
        zf.write(f'{extension_dir}/manifest.json', 'manifest.json')
        zf.write(f'{extension_dir}/background.js', 'background.js')
    return zip_path

# 加载代理配置
with open('proxy_auth.json', 'r') as f:
    proxy_auth = json.load(f)

# 创建并加载扩展
ext_zip = create_proxy_extension(proxy_auth)
chrome_options = ChromeOptions()
chrome_options.add_argument('--ignore-ssl-errors=yes')
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_extension(ext_zip)

driver = Chrome(options=chrome_options, version_main=137)

方案3：使用ScraperAPI的URL转发模式 #

绕过代理配置，直接用ScraperAPI提供的URL格式访问目标站点：

from undetected_chromedriver import Chrome, ChromeOptions

API_KEY = 'my_key_here'
target_url = 'https://httpbin.org/ip'  # 替换为你的目标URL
scraper_api_url = f'http://api.scraperapi.com?api_key={API_KEY}&url={target_url}'

chrome_options = ChromeOptions()
chrome_options.add_argument('--ignore-ssl-errors=yes')
chrome_options.add_argument('--ignore-certificate-errors')

driver = Chrome(options=chrome_options, version_main=137)
driver.get(scraper_api_url)

验证代理有效性 #

访问https://httpbin.org/ip查看当前IP，确认是否为ScraperAPI提供的代理IP：

driver.get('https://httpbin.org/ip')
print(driver.page_source)

内容的提问来源于stack exchange，提问作者Christian