Selenium4配置代理后启动Firefox请求localhost/session报HTTP400错误

问题现象 #

配置代理连接互联网的环境下，运行Python Selenium自动化脚本时Firefox浏览器无法正常启动。排查日志发现：Selenium向本地geckodriver服务的http://localhost:随机端口/session接口发送新建会话的POST请求时，请求被错误转发到代理服务器，最终返回HTTP 400错误，抛出selenium.common.exceptions.WebDriverException异常。

相关环境信息 #

版本信息 #

python - 3.10.4
selenium - 4.3.0
webdriver-manager - 3.8.1

复现代码 #

from selenium import webdriver
from selenium.webdriver.firefox.service import Service as FirefoxService
from webdriver_manager.firefox import GeckoDriverManager
from pathlib import Path
import logging

folderPath = str(Path( __file__ ).parent.absolute())

logging.basicConfig(format='[%(asctime)s] - [%(filename)s] - [%(levelname)s] - %(message)s', filename = folderPath + '/logs/seleniumAppLog.log', filemode = 'w', level = logging.DEBUG)

driver = webdriver.Firefox(service = FirefoxService(GeckoDriverManager().install()))

driver.get('https://www.google.com/')

运行日志 #

[2022-07-14 11:54:11,735] - [logger.py] - [INFO] - ====== WebDriver manager ======
[2022-07-14 11:54:15,407] - [logger.py] - [INFO] - Get LATEST geckodriver version for 96.0 firefox
[2022-07-14 11:54:15,411] - [connectionpool.py] - [DEBUG] - Starting new HTTPS connection (1): api.github.com:443
[2022-07-14 11:54:16,490] - [connectionpool.py] - [DEBUG] - https://api.github.com:443 "GET /repos/mozilla/geckodriver/releases/latest HTTP/1.1" 200 3015
[2022-07-14 11:54:20,423] - [logger.py] - [INFO] - Getting latest mozilla release info for v0.31.0
[2022-07-14 11:54:20,425] - [connectionpool.py] - [DEBUG] - Starting new HTTPS connection (1): api.github.com:443
[2022-07-14 11:54:21,525] - [connectionpool.py] - [DEBUG] - https://api.github.com:443 "GET /repos/mozilla/geckodriver/releases/tags/v0.31.0 HTTP/1.1" 200 None
[2022-07-14 11:54:21,530] - [logger.py] - [INFO] - About to download new driver from https://github.com/mozilla/geckodriver/releases/download/v0.31.0/geckodriver-v0.31.0-win64.zip
[2022-07-14 11:54:21,535] - [connectionpool.py] - [DEBUG] - Starting new HTTPS connection (1): github.com:443
[2022-07-14 11:54:22,553] - [connectionpool.py] - [DEBUG] - https://github.com:443 "GET /mozilla/geckodriver/releases/download/v0.31.0/geckodriver-v0.31.0-win64.zip HTTP/1.1" 302 0
[2022-07-14 11:54:22,558] - [connectionpool.py] - [DEBUG] - Starting new HTTPS connection (1): objects.githubusercontent.com:443
[2022-07-14 11:54:27,616] - [logger.py] - [INFO] - Driver has been saved in cache [C:\Users\sam\.wdm\drivers\geckodriver\win64\0.31]
[2022-07-14 11:54:28,867] - [remote_connection.py] - [DEBUG] - POST http://localhost:63660/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "firefox", "acceptInsecureCerts": true, "moz:debuggerAddress": true, "pageLoadStrategy": "normal"}}}
[2022-07-14 11:54:28,868] - [connectionpool.py] - [DEBUG] - Starting new HTTP connection (1): <proxy url here>
[2022-07-14 11:54:28,938] - [connectionpool.py] - [DEBUG] - <proxy url> "POST http://localhost:63660/session HTTP/1.1" 400 0
[2022-07-14 11:54:28,938] - [remote_connection.py] - [DEBUG] - Remote response: status=400 | data= | headers=HTTPHeaderDict({'Server': 'BigIP', 'Connection': 'Keep-Alive', 'Content-Length': '0'})
[2022-07-14 11:54:28,938] - [remote_connection.py] - [DEBUG] - Finished Request

控制台报错信息 #

Traceback (most recent call last):
File "C:\Users\sam\Downloads\repo\pendingCM_parser\snowSelenium.py", line 26, in 
driver = webdriver.Firefox(service = FirefoxService(GeckoDriverManager().install()))
File "C:\Program Files\Python\lib\site-packages\selenium\webdriver\firefox\webdriver.py", line 177, in __init__
super().__init__(
File "C:\Program Files\Python\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 277, in __init__
self.start_session(capabilities, browser_profile)
File "C:\Program Files\Python\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 370, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "C:\Program Files\Python\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 435, in execute
self.error_handler.check_response(response)
File "C:\Program Files\Python\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 211, in check_response
raise exception_class(value)
selenium.common.exceptions.WebDriverException: Message:

问题根因 #

Selenium 4.x版本默认继承系统全局代理配置，包括和本地geckodriver服务通信的localhost回环请求也会被转发到代理服务器。而企业代理（日志中显示代理服务为BigIP）无法正确处理指向本地回环地址的请求，直接返回400状态码，导致geckodriver会话创建流程中断，抛出驱动异常。

可行解决方案 #

• 代码层配置代理绕过规则：初始化Firefox驱动时显式指定本地地址不经过代理，示例代码如下：

  from selenium import webdriver
  from selenium.webdriver.firefox.service import Service as FirefoxService
  from webdriver_manager.firefox import GeckoDriverManager
  from selenium.webdriver.common.proxy import Proxy, ProxyType
  
  # 加载系统代理，同时配置本地回环地址绕过
  proxy_config = Proxy()
  proxy_config.proxy_type = ProxyType.SYSTEM
  proxy_config.no_proxy = "localhost,127.0.0.1"
  
  driver = webdriver.Firefox(
      service=FirefoxService(GeckoDriverManager().install()),
      proxy=proxy_config
  )
  
  driver.get('https://www.google.com/')
  

• 系统环境变量配置绕过：运行脚本前配置NO_PROXY环境变量，Selenium底层的HTTP客户端会自动识别该规则，请求本地地址时跳过代理。Windows环境执行set NO_PROXY=localhost,127.0.0.1，Linux/macOS环境执行export NO_PROXY=localhost,127.0.0.1，之后在同一个终端窗口运行脚本即可。
• 临时调整系统代理：启动驱动阶段临时关闭系统全局代理，等浏览器初始化完成后，再按需开启系统代理或单独给Firefox配置代理规则。

内容的提问来源于stack exchange，提问作者Sujin Sam