云函数环境下如何修改webdriver_manager的自定义缓存路径
问题描述
在云函数中搭建无头网页爬虫,使用Selenium结合Webdriver Manager自动化管理驱动,尝试修改缓存路径时触发报错,代码及错误日志如下:
代码示例
import os import logging # selenium 4 os.environ['GH_TOKEN'] = "gkjkjhjkhjhkjhuihjhgjhg" os.environ['WDM_LOG'] = str(logging.NOTSET) os.environ['WDM_LOCAL'] = '1' os.environ['WDM_SSL_VERIFY'] = '0' from selenium import webdriver from selenium.webdriver.chrome.service import Service as ChromeService from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.chrome.options import Options def hello_world(request): """Responds to any HTTP request. Args: request (flask.Request): HTTP request object. Returns: The response text or any set of values that can be turned into a Response object using `make_response <http://flask.pocoo.org/docs/1.0/api/#flask.Flask.make_response>`. """ # instance of Options class allows # us to configure Headless Chrome options = Options() print("options") options.headless = True driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager("2.26", cache_valid_range=1, path = r".\\temp\\Drivers").install() ), options=options) print("driver was initiated") # this parameter tells Chrome that # it should be run without UI (Headless) # initializing webdriver for Chrome with our options # driver = webdriver.Chrome(options=options) # driver = webdriver.Chrome(ChromeDriverManager(path = r"/temp/data").install()) request_json = request.get_json() if request_json and 'url' in request_json: url = request_json['url'] driver.get('https://www.geeksforgeeks.org') print(driver.title) driver.close() return f'Success!' else: return f'Not run'
错误日志
Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.10/site-packages/flask/app.py", line 2525, in wsgi_app response = self.full_dispatch_request() File "/layers/google.python.pip/pip/lib/python3.10/site-packages/flask/app.py", line 1822, in full_dispatch_request rv = self.handle_user_exception(e) File "/layers/google.python.pip/pip/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request rv = self.dispatch_request() File "/layers/google.python.pip/pip/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) File "/layers/google.python.pip/pip/lib/python3.10/site-packages/functions_framework/__init__.py", line 98, in view_func return function(request._get_current_object()) File "/workspace/main.py", line 28, in hello_world driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager("2.26", cache_valid_range=1, path = r".\\temp\\Drivers").install() File "/layers/google.python.pip/pip/lib/python3.10/site-packages/webdriver_manager/chrome.py", line 39, in install driver_path = self._get_driver_path(self.driver) File "/layers/google.python.pip/pip/lib/python3.10/site-packages/webdriver_manager/core/manager.py", line 31, in _get_driver_path binary_path = self.driver_cache.save_file_to_cache(driver, file) File "/layers/google.python.pip/pip/lib/python3.10/site-packages/webdriver_manager/core/driver_cache.py", line 45, in save_file_to_cache archive = save_file(file, path) File "/layers/google.python.pip/pip/lib/python3.10/site-packages/webdriver_manager/core/utils.py", line 38, in save_file os.makedirs(directory, exist_ok=True) File "/layers/google.python.runtime/python/lib/python3.10/os.py", line 215, in makedirs makedirs(head, exist_ok=exist_ok) File "/layers/google.python.runtime/python/lib/python3.10/os.py", line 215, in makedirs makedirs(head, exist_ok=exist_ok) File "/layers/google.python.runtime/python/lib/python3.10/os.py", line 215, in makedirs makedirs(head, exist_ok=exist_ok)
报错核心原因是Webdriver Manager无法写入指定路径,用户尝试用path = r".\\temp\\Drivers"修改缓存路径但配置错误,需调整路径设置方式。
解决方案
1. 确认云函数可写路径
云函数环境中,当前工作目录(.)通常无写入权限,优先使用系统临时目录(如Linux环境的/tmp),这类目录默认开放写入权限。
2. 正确配置缓存路径的两种方式
方式一:通过环境变量全局配置
在导入Webdriver Manager前设置WDM_CACHE_PATH环境变量,全局生效:
import os # 设置可写的临时目录作为缓存路径 os.environ['WDM_CACHE_PATH'] = "/tmp/wdm_drivers" # 其他环境变量设置... from webdriver_manager.chrome import ChromeDriverManager
方式二:通过ChromeDriverManager参数配置
注意:Webdriver Manager的正确参数名是cache_path而非path,修正后的代码如下:
driver = webdriver.Chrome( service=ChromeService( ChromeDriverManager( version="2.26", cache_valid_range=1, cache_path="/tmp/wdm_drivers" # 替换为正确参数名 ).install() ), options=options )
3. 额外注意事项
- 路径格式:Linux环境下使用正斜杠
/作为分隔符,避免Windows风格的反斜杠\\。 - 权限验证:确保指定目录所在路径有写入权限,优先选择系统临时目录。
- 版本兼容:若使用旧版Webdriver Manager,建议升级至最新版以确保
cache_path参数可用:pip install --upgrade webdriver-manager - 资源释放:使用
driver.quit()替代driver.close(),彻底关闭驱动进程,避免资源泄漏。
修改后的完整代码
import os import logging # selenium 4 os.environ['GH_TOKEN'] = "gkjkjhjkhjhkjhuihjhgjhg" os.environ['WDM_LOG'] = str(logging.NOTSET) os.environ['WDM_LOCAL'] = '1' os.environ['WDM_SSL_VERIFY'] = '0' # 设置自定义缓存路径到云函数可写的临时目录 os.environ['WDM_CACHE_PATH'] = "/tmp/wdm_drivers" from selenium import webdriver from selenium.webdriver.chrome.service import Service as ChromeService from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.chrome.options import Options def hello_world(request): options = Options() options.headless = True driver = webdriver.Chrome( service=ChromeService( ChromeDriverManager( version="2.26", cache_valid_range=1, cache_path="/tmp/wdm_drivers" ).install() ), options=options ) print("driver was initiated") request_json = request.get_json() if request_json and 'url' in request_json: url = request_json['url'] driver.get('https://www.geeksforgeeks.org') print(driver.title) driver.quit() return f'Success!' else: driver.quit() return f'Not run'
内容的提问来源于stack exchange,提问作者Gradley bear
相关产品推荐
相关产品推荐

