Databricks中Selenium ChromeDriver版本匹配问题求助

环境与问题背景 #

• 运行环境：Databricks Python脚本，通过Airflow DAG每周调度执行
• 依赖包：seleniumwire、selenium、webdriver_manager、chromedriver_autoinstaller
• 问题现象：此前运行正常，近期出现ChromeDriver版本检测失败，报错对应版本的驱动不存在，推测为Databricks环境或依赖包更新导致

当前代码 #

from seleniumwire import webdriver 
from seleniumwire.utils import decode as sw_decode
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
import chromedriver_autoinstaller

# 配置Chrome选项
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-dev-shm-usage')

preferences = {"download.prompt_for_download": False,
               "download.directory_upgrade": True,
               "directory_upgrade": True,
               "safebrowsing.enabled": True }

chrome_options.add_experimental_option("prefs", preferences)

chrome_driver = "/tmp/chromedriver"

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    options=chrome_options
)

# 访问目标页面
driver.get(url)

报错信息 #

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<command-3775777336036441> in <module>
     16 
     17 driver = webdriver.Chrome(
---> 18     service=Service(ChromeDriverManager().install()),
     19     options=chrome_options
     20 )

/local_disk0/pythonVirtualEnvDirs/virtualEnv-18c7cdb1-838b-427a-a362-0f0baf05cdb9/lib/python3.8/site-packages/webdriver_manager/chrome.py in install(self)
     37 
     38     def install(self) -> str:
---> 39         driver_path = self._get_driver_binary_path(self.driver)
     40         if all(test_os not in driver_path for test_os in ["mac_arm64", "mac_x64"]):
     41             os.chmod(driver_path, 0o755)

/local_disk0/pythonVirtualEnvDirs/virtualEnv-18c7cdb1-838b-427a-a362-0f0baf05cdb9/lib/python3.8/site-packages/webdriver_manager/core/manager.py in _get_driver_binary_path(self, driver)
     31             return binary_path
     32 
---> 33         file = self._download_manager.download_file(driver.get_driver_download_url())
     34         binary_path = self._cache_manager.save_file_to_cache(driver, file)
     35         return binary_path

/local_disk0/pythonVirtualEnvDirs/virtualEnv-18c7cdb1-838b-427a-a362-0f0baf05cdb9/lib/python3.8/site-packages/webdriver_manager/drivers/chrome.py in get_driver_download_url(self)
     43 
     44     def get_driver_download_url(self):
---> 45         driver_version_to_download = self.get_driver_version_to_download()
     46         os_type = self._os_type
     47         # For Mac ARM CPUs after version 106.0.5249.61 the format of OS type changed

/local_disk0/pythonVirtualEnvDirs/virtualEnv-18c7cdb1-838b-427a-a362-0f0baf05cdb9/lib/python3.8/site-packages/webdriver_manager/core/driver.py in get_driver_version_to_download(self)
     49         if not self._driver_to_download_version:
     50             self._driver_to_download_version = self._driver_version if self._driver_version not in (None, "latest") \
---> 51                 else self.get_latest_release_version()
     52         return self._driver_to_download_version
     53 

/local_disk0/pythonVirtualEnvDirs/virtualEnv-18c7cdb1-838b-427a-a362-0f0baf05cdb9/lib/python3.8/site-packages/webdriver_manager/drivers/chrome.py in get_latest_release_version(self)
     77             else f"{self._latest_release_url}_{determined_browser_version}"
     78         )
---> 79         resp = self._http_client.get(url=latest_release_url)
     80         return resp.text.rstrip()
     81 

/local_disk0/pythonVirtualEnvDirs/virtualEnv-18c7cdb1-838b-427a-a362-0f0baf05cdb9/lib/python3.8/site-packages/webdriver_manager/core/http.py in get(self, url, **kwargs)
     34         except exceptions.ConnectionError:
     35             raise ConnectionError(f"Could not reach host. Are you offline?")
---> 36         self.validate_response(resp)
     37         return resp
     38 

/local_disk0/pythonVirtualEnvDirs/virtualEnv-18c7cdb1-838b-427a-a362-0f0baf05cdb9/lib/python3.8/site-packages/webdriver_manager/core/http.py in validate_response(resp)
     13         status_code = resp.status_code
     14         if status_code == 404:
---> 15             raise ValueError(f"There is no such driver by url {resp.url}")
     16         elif status_code == 401:
     17             raise ValueError(f"API Rate limit exceeded. You have to add GH_TOKEN!!!")

ValueError: There is no such driver by url https://chromedriver.storage.googleapis.com/LATEST_RELEASE_108.0.5359.71

已尝试的无效操作 #

1. 尝试指定version="latest"初始化ChromeDriverManager

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager(version="latest").install()), options=chrome_options
)

报错：TypeError: install() got an unexpected keyword argument 'version'

2. 尝试在install()方法中传入version="latest"

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install(version="latest")),
    options=chrome_options
)

报错：TypeError: install() got an unexpected keyword argument 'version'

3. 尝试直接使用无参数的Service

driver = webdriver.Chrome(
        service=Service(),
        options=chrome_options
    )

报错：

The version of chrome cannot be detected. Trying with latest driver version
WebDriverException: Message: Service /root/.cache/selenium/chromedriver/linux64/114.0.5735.90/chromedriver unexpectedly exited. Status code was: 127

解决方案 #

方案1：利用已导入的chromedriver_autoinstaller自动匹配版本 #

你已导入chromedriver_autoinstaller但未使用，该工具可自动检测Chrome版本并安装对应驱动，替换原有ChromeDriverManager逻辑：

from seleniumwire import webdriver 
from seleniumwire.utils import decode as sw_decode
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
import chromedriver_autoinstaller

# 自动安装对应版本的ChromeDriver到/tmp目录（适配Databricks环境）
chromedriver_autoinstaller.install(path='/tmp')

# 配置Chrome选项
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-dev-shm-usage')

preferences = {"download.prompt_for_download": False,
               "download.directory_upgrade": True,
               "directory_upgrade": True,
               "safebrowsing.enabled": True }

chrome_options.add_experimental_option("prefs", preferences)

# 使用自动安装的驱动初始化Chrome
driver = webdriver.Chrome(
    service=Service('/tmp/chromedriver'),
    options=chrome_options
)

# 访问目标页面
driver.get(url)

方案2：升级webdriver_manager到最新版本 #

旧版本webdriver_manager的版本检测逻辑存在缺陷，升级到最新版后，使用正确的参数指定版本：

1. 在Databricks依赖配置中更新webdriver-manager至最新版
2. 修改初始化代码：

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager(driver_version="latest").install()),
    options=chrome_options
)

注：最新版webdriver_manager使用driver_version参数而非version，这是此前尝试失败的核心原因。

方案3：指定Chrome二进制路径帮助版本检测 #

若Databricks环境中Chrome路径不标准，导致工具无法自动检测版本，可手动指定Chrome二进制路径：

from seleniumwire import webdriver 
from seleniumwire.utils import decode as sw_decode
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.core.os_manager import ChromeType

# 配置Chrome选项
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-dev-shm-usage')
# 指定Chrome二进制路径（Databricks环境常见路径为/usr/bin/google-chrome，可根据实际情况调整）
chrome_options.binary_location = "/usr/bin/google-chrome"

preferences = {"download.prompt_for_download": False,
               "download.directory_upgrade": True,
               "directory_upgrade": True,
               "safebrowsing.enabled": True }

chrome_options.add_experimental_option("prefs", preferences)

# 指定Chrome类型，帮助工具正确检测版本
driver = webdriver.Chrome(
    service=Service(ChromeDriverManager(chrome_type=ChromeType.GOOGLE).install()),
    options=chrome_options
)

# 访问目标页面
driver.get(url)

内容的提问来源于stack exchange，提问作者random_name