Ubuntu VDS运行Headless Chrome+Selenium Python代码遇下载报错求助

环境与操作步骤 #

已完成以下操作：

• 更新系统：

  sudo apt update
  sudo apt upgrade
  

• 安装Chrome浏览器：

  wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
  sudo dpkg -i google-chrome-stable_current_amd64.deb
  sudo apt-get install -f
  

• 安装依赖包：

  sudo apt install python3-pip
  pip install selenium
  pip install webdriver_manager
  

• 运行的Python代码：

  from selenium import webdriver
  from selenium.webdriver.chrome.options import Options
  from selenium.webdriver.chrome.service import Service
  from webdriver_manager.chrome import ChromeDriverManager
  
  options = Options()
  options.add_argument('--headless')
  options.add_argument('--no-sandbox')
  options.add_argument('--disable-dev-shm-usage')
  driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
  
  driver.get("https://python.org")
  print(driver.title)
  driver.close()
  

错误信息 #

运行代码时触发以下错误：

Traceback (most recent call last):
  File "/root/test.py", line 24, in <module>
    driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
  File "/root/venv/lib/python3.10/site-packages/webdriver_manager/chrome.py", line 40, in install
    driver_path = self._get_driver_binary_path(self.driver)
  File "/root/venv/lib/python3.10/site-packages/webdriver_manager/core/manager.py", line 40, in _get_driver_binary_path
    file = self._download_manager.download_file(driver.get_driver_download_url(os_type))
  File "/root/venv/lib/python3.10/site-packages/webdriver_manager/core/download_manager.py", line 29, in download_file
    response = self._http_client.get(url)
  File "/root/venv/lib/python3.10/site-packages/webdriver_manager/core/http.py", line 36, in get
    self.validate_response(resp)
  File "/root/venv/lib/python3.10/site-packages/webdriver_manager/core/http.py", line 19, in validate_response
    raise ValueError(
ValueError: response body:
Google-Edge-Cache: forbidden
Error: 7
request url:
https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/116.0.5845.96/linux64/chromedriver-linux64.zip
response headers:
{'content-length': '37', 'content-type': 'text/plain', 'x-request-id': '6bf004ec-4d9e-4bd0-a5d6-a296a27fc6db', 'alt-svc': 'h3=":443"; ma=2592000, h3-29=":443"; ma=2592000', 'date': 'Mon, 21 Aug 2023 08:24:16 GMT', 'server': 'Google-Edge-Cache'}

解决方案 #

1. 手动下载ChromeDriver并指定路径 #

• 第一步：查看已安装的Chrome版本

  google-chrome --version
  

• 第二步：根据版本号下载对应版本的ChromeDriver，解压后得到chromedriver可执行文件
• 第三步：修改Python代码，直接指定chromedriver路径：

  from selenium import webdriver
  from selenium.webdriver.chrome.options import Options
  from selenium.webdriver.chrome.service import Service
  
  options = Options()
  options.add_argument('--headless')
  options.add_argument('--no-sandbox')
  options.add_argument('--disable-dev-shm-usage')
  # 替换为你解压后的chromedriver实际路径
  driver = webdriver.Chrome(service=Service('/usr/local/bin/chromedriver'), options=options)
  
  driver.get("https://python.org")
  print(driver.title)
  driver.quit()
  

2. 使用系统包管理安装chromedriver #

Ubuntu仓库提供chromium的驱动包，可直接安装：

sudo apt install chromium-chromedriver

安装完成后，默认路径为/usr/lib/chromium-browser/chromedriver，可以直接在代码中指定该路径，或者将其添加到系统PATH中：

export PATH=$PATH:/usr/lib/chromium-browser/

之后代码可保持原写法，webdriver_manager会自动检测到已安装的驱动。

3. 为webdriver_manager配置代理（若VDS有可用代理） #

如果VDS需要通过代理访问外部资源，可修改代码为webdriver_manager配置HTTP代理：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.core.http import HttpClient

options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')

# 替换为你的代理地址和端口，支持http/https/socks5
http_client = HttpClient(proxy="http://proxy-ip:proxy-port")
driver = webdriver.Chrome(
    service=Service(ChromeDriverManager(http_client=http_client).install()),
    options=options
)

driver.get("https://python.org")
print(driver.title)
driver.quit()

内容的提问来源于stack exchange，提问作者DoubleW