R语言抓取Wayback Machine不等长表格，缺失节点填NA求助

原代码核心问题 #

• 未定义parseMatch函数，直接调用触发报错
• 未区分4列/7列表格的结构差异，硬编码提取数据导致内容错位
• map_df(html, getBasicInfo)逻辑冗余，单页HTML无需遍历处理
• CSS选择器依赖固定列位置，在不同结构表格中匹配错误内容
• 缺失对无Worlds列场景的处理，无法自动填充NA

修正后的代码 #

library(rvest); library(tidyverse)
library(RSelenium); library(netstat)

# 定义文本提取函数
parseMatch <- function(html_node, css_selector, is_single = FALSE) {
  elements <- html_node %>% html_elements(css = css_selector) %>% html_text2()
  if (is_single) {
    return(ifelse(length(elements) > 0, elements[1], NA_character_))
  } else {
    return(elements)
  }
}

# 目标快照链接
links2 = c("https://web.archive.org/web/20220000000000*/https://www.bjjcompsystem.com/tournaments/1869/categories/2053146",
           "https://web.archive.org/web/20220000000000*/https://www.bjjcompsystem.com/tournaments/1869/categories/2053225")

# 启动Selenium服务
remote_driver = rsDriver(browser = 'firefox', verbose = F, port = free_port())
rd = remote_driver$client
rd$open()
rd$maxWindowSize()

# 初始化空数据框
all.ranks = tibble()

# 循环抓取每个链接
for (link in links2) {
  rd$navigate(link)
  Sys.sleep(10)
  
  # 点击最新快照
  date_element = rd$findElement(using = 'css', '.captures-range-info a:last-of-type')
  date_element$clickElement()
  Sys.sleep(10)
  
  # 获取页面HTML
  page_html = read_html(rd$getPageSource()[[1]])
  
  # 获取表格行（排除表头）
  table_rows = page_html %>% html_elements("table tr:not(:first-child)")
  
  # 遍历每行提取数据
  row_data <- map_dfr(table_rows, function(row) {
    # 提取基础列
    rank <- parseMatch(row, '.prioriry-number', is_single = TRUE)
    name <- parseMatch(row, '.competitor-name', is_single = TRUE)
    gym <- parseMatch(row, 'td:nth-child(2)', is_single = TRUE)
    grand_slam_pts <- parseMatch(row, 'td:nth-child(3)', is_single = TRUE)
    overall_pts <- parseMatch(row, 'td:nth-child(4)', is_single = TRUE)
    
    # 判断是否为7列表格
    has_worlds_cols <- length(row %>% html_elements("td")) >= 5
    
    # 初始化Worlds列为NA
    adult_world <- NA_character_
    m1_world <- NA_character_
    m2_world <- NA_character_
    
    # 7列表格提取对应数据
    if (has_worlds_cols) {
      adult_world <- parseMatch(row, 'td:nth-child(5)', is_single = TRUE)
      m1_world <- parseMatch(row, 'td:nth-child(6)', is_single = TRUE)
      m2_world <- parseMatch(row, 'td:nth-child(7)', is_single = TRUE)
    }
    
    # 返回当前行数据
    tibble(
      rank = rank,
      name = name,
      gym = gym,
      grand_slam_pts = grand_slam_pts,
      overall_pts = overall_pts,
      AdultWorldChamp = adult_world,
      M1WorldChamp = m1_world,
      M2WorldChamp = m2_world
    )
  })
  
  # 合并到总数据框
  all.ranks <- bind_rows(all.ranks, row_data)
}

# 关闭Selenium服务
rd$close()
remote_driver$server$stop()

关键改进点 #

1. 补全parseMatch函数，统一处理文本提取逻辑，支持单元素/多元素场景
2. 通过每行<td>数量区分表格结构，针对性提取数据
3. 使用tibble和bind_rows自动对齐列，缺失列自动填充NA
4. 遍历表格行而非整页HTML，避免结构差异导致的批量提取错误
5. 明确列位置对应关系，避免CSS选择器在不同结构表格中失效

内容的提问来源于stack exchange，提问作者bandcar