使用Watir抓取nfl.com动态JSON内容失败，求技术协助

用Watir抓取nfl.com数据已有一年，近期突然失效，推测是时序问题——页面已加载完成，但JSON驱动的动态内容尚未加载完毕。此前通过js_doc = browser.main(id: "main-content").wait_until(&:present?)等待主内容加载，但目前该逻辑返回过早。尝试等待nfl-o-matchup-group类的div元素却始终超时，求针对动态JSON模块加载的有效等待方案。

环境配置 #

Gemfile.lock相关依赖 #

nokogiri (1.13.9)
  mini_portile2 (~> 2.8.0)
  racc (~> 1.4)
watir (7.1.0)
  regexp_parser (>= 1.2, < 3)
  selenium-webdriver (~> 4.0)
web-console (3.7.0)
  actionview (>= 5.0)
  activemodel (>= 5.0)
  bindex (>= 0.4.0)
  railties (>= 5.0)
webdrivers (5.2.0)
  nokogiri (~> 1.6)
  rubyzip (>= 1.3.0)
  selenium-webdriver (~> 4.0)
selenium-webdriver (4.5.0)
  childprocess (>= 0.5, < 5.0)
  rexml (~> 3.2, >= 3.2.5)
  rubyzip (>= 1.2.2, < 3.0)
  websocket (~> 1.0)

ChromeDriver版本 #

ChromeDriver 90.0.4430.212 (e3cd97fc771b893b7fd1879196d1215b622c2bed-refs/branch-heads/4430@{#1429})

当前代码 #

Watir与Nokogiri调用代码 #

if Rails.env.production?
  args = ['--no-sandbox', '--disable-dev-shm-usage', '--disable-gpu',
          '--remote-debugging-port=9222']
  browser = Watir::Browser.new :chrome, headless: true, options: {args: args}
else
  browser = Watir::Browser.new :chrome, headless: true
end
url_path = "http://www.nfl.com/schedules/" + year + "/REG" + weekNum.to_s
browser.goto(url_path)
js_doc = browser.main(id: "main-content").wait_until(&:present?)
doc = Nokogiri::HTML(js_doc.inner_html)

解析代码（当前返回空值） #

game_date = game_group.css('h2.d3-o-section-title').text

目标HTML结构 #

<main role="main" id="main-content" tabindex="0" aria-label="Page main content">
  <section class="d3-l-section-row d3-l-section-row--no-margin-top">... </section>
  <section class="d3-l-section-row">...</section>
  <section class="d3-l-grid--outer d3-l-section-row">...</section>
  <section class="d3-l-grid--outer d3-l-adv-row">...</section>

  <div data-json-module="{&quot;;Name&quot;;:&quot;;Schedules&quot;;,&quot;;Module&quot;;:{&quot;;seasonFromUrl&quot;:2022,&quot;;SeasonType&quot;;:&quot;;REG9&quot;;,&quot;;WeekFromUrl&quot;:9,&quot;;HeaderCountryCode&quot;;:&quot;;US&quot;;,&quot;;TimeZoneID&quot;;:&quot;;America/Chicago&quot;;,&quot;;PreSeasonPlacement&quot;:0,&quot;;RegularSeasonPlacement&quot;:0,&quot;;PostSeasonPlacement&quot;:0}}" data-require="modules/scheduleByWeek" data-require-loaded="true">

    <span style="display:none">...</span>
    <section class="d3-l-grid--outer d3-l-section-row nfl-o-matchup-group">
      <div class="d3-l-grid--inner">
        <div class="d3-l-col__col-12">
            <h2 class="d3-o-section-title">Thursday, November 3rd</h2>
          <div class="nfl-c-matchup-strip nfl-c-matchup-strip--post-game">
           <a class="nfl-c-matchup-strip__left-area" href="/games/eagles-at-texans-2022-reg-9" aria-label="Texans vs Eagles game page">
            <div class="nfl-c-matchup-strip__game-info">
            <p class="nfl-c-matchup-strip__period">FINAL</p>
          </div>
          <div class="nfl-c-matchup-strip__game">
            <div class="nfl-c-matchup-strip__team nfl-c-matchup-strip__team--opponent">
            <div class="nfl-c-matchup-strip__team-score" data-require="modules/displayScores" data-score="29" data-require-loaded="true">29</div>
              <p class="nfl-c-matchup-strip__team-name">
                <span class="nfl-c-matchup-strip__team-logo">
                  <picture>... </picture>
                </span>
                <span class="nfl-c-matchup-strip__team-abbreviation"> PHI </span>
                <span class="nfl-c-matchup-strip__team-fullname"> Eagles </span>
              </p>
              <div class="nfl-c-matchup-strip__record">(8-0)</div>
            </div>
            <div class="nfl-c-matchup-strip__team-separator">
              <span class="nfl-o-icon nfl-o-icon--medium">... </span>
            </div>
            <div class="nfl-c-matchup-strip__team">
              <div class="nfl-c-matchup-strip__team-score" data-require="modules/displayScores" data-score="17" data-require-loaded="true">17</div>
                <p class="nfl-c-matchup-strip__team-name">
                  <span class="nfl-c-matchup-strip__team-logo">...</span>
                  <span class="nfl-c-matchup-strip__team-abbreviation"> HOU </span>
                  <span class="nfl-c-matchup-strip__team-fullname"> Texans </span>
                </p>
                <div class="nfl-c-matchup-strip__record">(1-6-1)</div>
              </div>
            </div>
          </a>
          <div class="nfl-c-matchup-strip__right-area">
            <a class="nfl-o-cta nfl-o-cta--link" href="/games/eagles-at-texans-2022-reg-9" aria-label="Replay">
              <span class="nfl-o-icon nfl-o-icon--medium">...</span>
              <span>Replay</span>
            </a>
          </div>
        </div>
      </div>
    </div>
  </section>

解决方案 #

方案1：直接等待目标数据元素可见 #

既然最终需要的是h2.d3-o-section-title的文本，直接等待该元素可见比等待父元素更精准，能确保动态内容已渲染：

# 替换原等待逻辑
browser.h2(class: "d3-o-section-title").wait_until(&:visible?)
# 再获取完整页面内容
doc = Nokogiri::HTML(browser.html)

也可以等待nfl-o-matchup-group元素内包含子元素：

browser.section(class: "nfl-o-matchup-group").wait_until { |el| el.h2.present? }

方案2：等待JSON模块加载完成 #

观察到目标内容依赖data-require="modules/scheduleByWeek"的div模块，该模块加载完成后会添加data-require-loaded="true"属性，可等待这个状态：

browser.div(data_require: "modules/scheduleByWeek").wait_until { |el| el.data_require_loaded == "true" }

方案3：修复ChromeDriver版本兼容性 #

当前ChromeDriver为90版本，与selenium-webdriver 4.5.0可能存在版本不匹配问题。建议用webdrivers gem自动管理Driver版本，确保与本地Chrome浏览器版本一致：

# 在代码开头添加
require 'webdrivers'
Webdrivers.cache_time = 86400 # 缓存Driver一天，避免重复下载

方案4：辅助调整隐式等待时间 #

全局设置隐式等待，让Watir在查找元素时自动等待一段时间（不建议单独依赖，需结合显式等待）：

browser.driver.manage.timeouts.implicit_wait = 10 # 单位：秒

内容的提问来源于stack exchange，提问作者Robert Reynolds