Linux环境下R selenider爬虫遇--user-data-dir错误求助
问题描述
我尝试使用以下R代码进行网页爬取:
library(selenider) library(rvest) session <- selenider_session("selenium", browser = "chrome") Sys.sleep(3) open_url("https://egamersworld.com/callofduty/matches") elements <- session |> get_page_source() |> html_elements(".item_teams__cKXQT") res <- data.frame( home_team_name = elements |> html_elements(".item_team__evhUQ:nth-child(1) .item_teamName__NSnfH") |> html_text(trim = TRUE), home_team_odds = elements |> html_elements(".item_team__evhUQ:nth-child(1) .item_odd__Lm2Wl") |> html_text(trim = TRUE), away_team_name = elements |> html_elements(".item_team__evhUQ:nth-child(3) .item_teamName__NSnfH") |> html_text(trim = TRUE), away_team_odds = elements |> html_elements(".item_team__evhUQ:nth-child(3) .item_odd__Lm2Wl") |> html_text(trim = TRUE), match_date = elements |> html_elements(".item_scores__Vi7YX .item_date__g4cq_") |> html_text(trim = TRUE), match_time = elements |> html_elements(".item_scores__Vi7YX .item_time__xBia_") |> html_text(trim = TRUE), match_type = elements |> html_elements(".item_scores__Vi7YX .item_bo__u2C9Q") |> html_text(trim = TRUE) )
该代码在Windows 10本地运行正常,但部署到Linux服务器时出现如下错误:
Error in `create_selenium_client_internal()`: ! A Selenium session could not be started Caused by error in `httr2::req_perform()`: ! HTTP 500 Internal Server Error. ✖ Session not created. ✖ Could not start a new session. Error while creating session with the driver service. Stopping driver service: Could not start a new session. Response code 500. Message: probably user data directory is already in use, please specify a unique value for --user-data-dir argument, or don't use --user-data-dir Host info: host: 'Unknown', ip: 'Unknown' Build info: version: '4.29.0', revision: '18ae989' System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-134-generic', java.version: '11.0.26' Driver info: driver.version: unknown Build info: version: '4.29.0', revision: '18ae989' System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-134-generic', java.version: '11.0.26' Driver info: driver.version: unknown
我尝试手动创建并指定目录,但仍出现相同错误:
server_options = selenium_options(server_options = selenium_server_options(extra_args = c("--user-data-dir=/tmp/testing"))) session <- selenider_session( "selenium", browser = "chrome", options = server_options )
我也尝试杀死所有Chrome进程,但无济于事。服务器上有其他Python Selenium脚本可正常运行,且未手动设置--user-data-dir参数,我更擅长R,希望将代码迁移到R环境。
解决方案
针对Linux服务器上的这个问题,可尝试以下几种方法:
1. 配置Chrome生成唯一临时目录并启用无头模式
直接在Chrome选项中生成基于时间戳的唯一用户数据目录,同时启用无头模式适配服务器环境,附加Linux运行必备参数:
library(selenider) library(rvest) # 配置Chrome选项 chrome_opts <- chrome_options( extra_args = c( "--headless=new", "--no-sandbox", "--disable-dev-shm-usage", "--user-data-dir=/tmp/selenium-chrome-$(date +%s)" ) ) # 创建会话 session <- selenider_session( "selenium", browser = "chrome", options = selenium_options(browser_options = chrome_opts) ) # 后续爬取代码保持不变 Sys.sleep(3) open_url("https://egamersworld.com/callofduty/matches") elements <- session |> get_page_source() |> html_elements(".item_teams__cKXQT") res <- data.frame( home_team_name = elements |> html_elements(".item_team__evhUQ:nth-child(1) .item_teamName__NSnfH") |> html_text(trim = TRUE), home_team_odds = elements |> html_elements(".item_team__evhUQ:nth-child(1) .item_odd__Lm2Wl") |> html_text(trim = TRUE), away_team_name = elements |> html_elements(".item_team__evhUQ:nth-child(3) .item_teamName__NSnfH") |> html_text(trim = TRUE), away_team_odds = elements |> html_elements(".item_team__evhUQ:nth-child(3) .item_odd__Lm2Wl") |> html_text(trim = TRUE), match_date = elements |> html_elements(".item_scores__Vi7YX .item_date__g4cq_") |> html_text(trim = TRUE), match_time = elements |> html_elements(".item_scores__Vi7YX .item_time__xBia_") |> html_text(trim = TRUE), match_type = elements |> html_elements(".item_scores__Vi7YX .item_bo__u2C9Q") |> html_text(trim = TRUE) ) # 爬取完成后显式关闭会话释放资源 close_session(session)
2. 检查并匹配Selenium与ChromeDriver版本
确保服务器上的ChromeDriver版本与安装的Chrome浏览器版本完全兼容,Selenium 4.29.0需要对应版本的ChromeDriver。可通过以下命令检查Chrome版本:
google-chrome --version
也可使用wdman包自动管理驱动版本:
library(wdman) chrome() # 自动安装匹配的ChromeDriver
3. 用Docker封装运行环境(可选)
若服务器环境复杂,可使用预配置的Selenium Docker镜像避免依赖问题:
docker run -d -p 4444:4444 selenium/standalone-chrome:latest
然后在R代码中连接远程Selenium服务器:
session <- selenider_session( "remote", browser = "chrome", server_url = "http://localhost:4444/wd/hub" )
内容的提问来源于stack exchange,提问作者Nick Amato
相关产品推荐
相关产品推荐

