使用Kimurai时出现ArgumentError（未知关键字: :driver_path），原因何在？

错误日志 #

WwozScraper.crawl!
I, [2022-08-15 22:43:20 -0400#49087] [M: 3420]  INFO -- wwoz_scraper: Spider: started: wwoz_scraper
D, [2022-08-15 22:43:20 -0400#49087] [M: 3420] DEBUG -- wwoz_scraper: BrowserBuilder (selenium_chrome): created browser instance
D, [2022-08-15 22:43:20 -0400#49087] [M: 3420] DEBUG -- wwoz_scraper: BrowserBuilder (selenium_chrome): enabled native headless_mode
I, [2022-08-15 22:43:20 -0400#49087] [M: 3420]  INFO -- wwoz_scraper: Browser: started get request to: https://www.wwoz.org/programs/playlists
I, [2022-08-15 22:43:20 -0400#49087] [M: 3420]  INFO -- wwoz_scraper: Info: visits: requests: 1, responses: 0
I, [2022-08-15 22:43:20 -0400#49087] [M: 3420]  INFO -- wwoz_scraper: Browser: driver selenium_chrome has been destroyed
F, [2022-08-15 22:43:20 -0400#49087] [M: 3420] FATAL -- wwoz_scraper: Spider: stopped: {:spider_name=>"wwoz_scraper", :status=>:failed, :error=>"#<ArgumentError: unknown keyword: :driver_path>", :environment=>"development", :start_time=>2022-08-15 22:43:20.673033 -0400, :stop_time=>2022-08-15 22:43:20.795284 -0400, :running_time=>"0s", :visits=>{:requests=>1, :responses=>0}, :items=>{:sent=>0, :processed=>0}, :events=>{:requests_errors=>{}, :drop_items_errors=>{}, :custom=>{}}}

爬虫类代码 #

require 'kimurai'

class WwozScraper < Kimurai::Base
  @name= 'wwoz_scraper'
  @start_urls = ['https://www.wwoz.org/programs/playlists']
  @engine = :selenium_chrome

  def scrape_page
      doc = browser.current_response
  end

  def parse(response, url:, data: {})
    scrape_page
  end
end

环境信息 #

• Ruby版本：2.7.4
• Gemfile内容：

source 'https://rubygems.org'
git_source(:github) { |repo| "https://github.com/#{repo}.git" }

ruby '2.7.4'

# Bundle edge Rails instead: gem 'rails', github: 'rails/rails', branch: 'main'
gem 'rails', '~> 6.1.5', '>= 6.1.5.1'
# Use sqlite3 as the database for Active Record
gem 'sqlite3', '~> 1.4'
# Use Puma as the app server
gem 'puma', '~> 5.0'
# Use SCSS for stylesheets
gem 'sass-rails', '>= 6'
# Transpile app-like JavaScript. Read more: https://github.com/rails/webpacker
gem 'webpacker', '~> 5.0'
# Turbolinks makes navigating your web application faster. Read more: https://github.com/turbolinks/turbolinks
gem 'turbolinks', '~> 5'
# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder
gem 'jbuilder', '~> 2.7'
# Use Redis adapter to run Action Cable in production
# gem 'redis', '~> 4.0'
# Use Active Model has_secure_password
# gem 'bcrypt', '~> 3.1.7'
gem 'kimurai'

# Use Active Storage variant
# gem 'image_processing', '~> 1.2'

# Reduces boot times through caching; required in config/boot.rb
gem 'bootsnap', '>= 1.4.4', require: false

group :development, :test do
  # Call 'byebug' anywhere in the code to stop execution and get a debugger console
  gem 'byebug', platforms: [:mri, :mingw, :x64_mingw]
end

group :development do
  # Access an interactive console on exception pages or by calling 'console' anywhere in the code.
  gem 'web-console', '>= 4.1.0'
  # Display performance information such as SQL time and flame graphs for each request in your browser.
  # Can be configured to work on production as well see: https://github.com/MiniProfiler/rack-mini-profiler/blob/master/README.md
  gem 'rack-mini-profiler', '~> 2.0'
  gem 'listen', '~> 3.3'
  # Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
  gem 'spring'
end

group :test do
  # Adds support for Capybara system testing and selenium driver
  gem 'capybara', '>= 3.26'
  gem 'selenium-webdriver', '>= 4.0.0.rc1'
  # Easy installation and use of web drivers to run system tests with browsers
  gem 'webdrivers'
end

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]

解决方法 #

这个错误是Kimurai与高版本selenium-webdriver（你使用的是>=4.0.0.rc1）不兼容导致的——Selenium 4.x已经移除了:driver_path这个关键字参数，但Kimurai的旧版本仍在调用它。

方案一：降级Selenium到3.x系列（推荐） #

1. 修改Gemfile中的selenium-webdriver版本：

   gem 'selenium-webdriver', '~> 3.142.7'
   

2. 执行命令更新依赖：

   bundle update selenium-webdriver
   

3. 重新运行爬虫即可。

方案二：升级Kimurai到最新版本 #

如果Kimurai后续发布了支持Selenium 4的版本，可以尝试升级Kimurai：

1. 修改Gemfile中的Kimurai版本为最新（或直接写gem 'kimurai', github: 'vifreefly/kimuraframework'以获取最新源码）
2. 执行bundle update kimurai更新依赖
3. 测试爬虫是否正常运行

内容的提问来源于stack exchange，提问作者pigusan