使用R的rvest爬取WCA网站，如何过滤比赛数据仅保留个人PR

问题背景 #

使用R语言的rvest工具爬取世界魔方协会（WCA）选手页面时，目标是获取各项目的个人最佳单次成绩（Single）和平均成绩（Average），但当前代码会同时返回无关的比赛数据，无法只保留个人纪录（PRs）。

原代码 #

#' get_person_data
#'
#' Gets person data from WCA website using the ID provided.
#'
#' @importFrom rvest read_html html_node html_text html_attr
#' @importFrom dplyr %>%
#' @param id ID of the person to be searched.
#' @param export_csv Whether or not the data should be exported. Set to false by default.
#' @export
get_person_data <- function(id, export_csv=FALSE, directory=NULL) {
  html <- rvest::read_html(paste("https://www.worldcubeassociation.org/persons/", id, sep = ""))

  name <- html %>%
    rvest::html_node("div.text-center h2") %>%
    rvest::html_text()

  image_url <- html %>%
    rvest::html_node("div.text-center img.avatar") %>%
    rvest::html_attr("src")

   country <- html %>%
    rvest::html_node(".country") %>%
    rvest::html_text()

   gender <- html %>%
     rvest::html_node("table.table tbody td:nth-child(3)") %>%
     rvest::html_text()

   comps <- html %>%
     rvest::html_node("table.table tbody td:nth-child(4)") %>%
     rvest::html_text()

   comp_solves <- html %>%
     rvest::html_node("table.table tbody td:nth-child(5)") %>%
     rvest::html_text()

   events <- html %>%
     rvest::html_nodes("table.table tbody tr")

   event_data <- lapply(events, function(event) {
     event_name <- event %>%
       rvest::html_node("td:nth-child(1)") %>%
       rvest::html_text(trim=TRUE)

     single <- event %>%
       rvest::html_node("td:nth-child(5)") %>%
       rvest::html_text(trim=TRUE)

     average <- event %>%
       rvest::html_node("td:nth-child(6)") %>%
       rvest::html_text(trim=TRUE)

     data.frame(
       Event = event_name,
       Single = single,
       Average = average,
       stringsAsFactors = FALSE
     )
   })

   event_data <- do.call(rbind, event_data)

   if(export_csv) {
     key_data <- data.frame(
       name,
       id,
       image_url,
       country,
       gender,
       comps,
       comp_solves
     )
     names(key_data) <- c("Name", "ID", "Avatar", "Country", "Gender", "Competitions", "Completed Solves")

     final_data <- cbind(key_data, event_data)

     write.csv(final_data, file=paste0(directory, "/person_data.csv"), fileEncoding = "UTF-8")
     print("Saved to directory")
   }

  cat("Name:", name, "\n")
  cat("ID:", id, "\n")
  cat("Image URL:", image_url, "\n")
  cat("Country:", country, "\n")
  cat("Gender:", gender, "\n")
  cat("Competitions:", comps, "\n")
  cat("Completed Solves:", comp_solves, "\n")
  cat("Event Data:\n")
  print(event_data)
}

当前问题 #

运行代码后，event_data会混入无关的比赛数据，输出示例：

Event Data:
                     Event Single Average
1               3x3x3 Cube  15.02   17.47
2               3x3x3 Cube   <NA>    <NA>
3  Avenida Chile VIII 2024          18.06
4                                   17.47
5 Monterrey Bogotá VI 2024          30.20

此前尝试用以下代码定位目标表格，但返回NULL：

event_table <- html %>%
    html_node(xpath = "//table[contains(@class, 'table table-striped table-condensed')][1]") %>%
    html_node("tbody") %>%
    html_nodes("tr")

解决方案 #

问题核心是原代码选中了页面所有表格的行，包括选手信息表、比赛记录表等冗余内容。需要精准定位个人最佳成绩（Personal Bests）对应的表格，同时优化基础信息的获取逻辑，避免结构变化导致的错误。

修正后的完整代码 #

#' get_person_data
#'
#' Gets person data from WCA website using the ID provided.
#'
#' @importFrom rvest read_html html_node html_text html_attr html_nodes
#' @importFrom dplyr %>%
#' @param id ID of the person to be searched.
#' @param export_csv Whether or not the data should be exported. Set to false by default.
#' @export
get_person_data <- function(id, export_csv=FALSE, directory=NULL) {
  html <- rvest::read_html(paste("https://www.worldcubeassociation.org/persons/", id, sep = ""))

  # 获取基础信息
  name <- html %>%
    rvest::html_node("div.text-center h2") %>%
    rvest::html_text(trim = TRUE)

  image_url <- html %>%
    rvest::html_node("div.text-center img.avatar") %>%
    rvest::html_attr("src")

  country <- html %>%
    rvest::html_node(".country") %>%
    rvest::html_text(trim = TRUE)

  # 精准定位选手信息表格，避免结构变化出错
  info_table <- html %>% 
    rvest::html_node(xpath = "//h3[text()='Person Information']/following-sibling::table[1]") %>% 
    rvest::html_nodes("td")
  
  gender <- info_table[3] %>% rvest::html_text(trim = TRUE)
  comps <- info_table[4] %>% rvest::html_text(trim = TRUE)
  comp_solves <- info_table[5] %>% rvest::html_text(trim = TRUE)

  # 精准定位个人最佳成绩表格的行
  pr_rows <- html %>% 
    rvest::html_node(xpath = "//h3[text()='Personal Bests']/following-sibling::table[1]") %>% 
    rvest::html_nodes("tbody tr")

  # 提取PR数据
  event_data <- lapply(pr_rows, function(row) {
    event_name <- row %>%
      rvest::html_node("td:nth-child(1)") %>%
      rvest::html_text(trim=TRUE)

    single <- row %>%
      rvest::html_node("td:nth-child(5)") %>%
      rvest::html_text(trim=TRUE)

    average <- row %>%
      rvest::html_node("td:nth-child(6)") %>%
      rvest::html_text(trim=TRUE)

    data.frame(
      Event = event_name,
      Single = ifelse(single == "", NA, single),
      Average = ifelse(average == "", NA, average),
      stringsAsFactors = FALSE
    )
  })

  event_data <- do.call(rbind, event_data)
  # 过滤空行（如果有的话）
  event_data <- event_data[!is.na(event_data$Event) & event_data$Event != "", ]

   if(export_csv) {
     key_data <- data.frame(
       Name = name,
       ID = id,
       Avatar = image_url,
       Country = country,
       Gender = gender,
       Competitions = comps,
       Completed_Solves = comp_solves,
       stringsAsFactors = FALSE
     )

     # 避免cbind重复复制基础数据，确保每条PR对应一条基础信息
     final_data <- cbind(key_data[rep(1, nrow(event_data)), ], event_data)

     write.csv(final_data, file=paste0(directory, "/person_data.csv"), fileEncoding = "UTF-8", row.names = FALSE)
     print("Saved to directory")
   }

  # 输出信息
  cat("Name:", name, "\n")
  cat("ID:", id, "\n")
  cat("Image URL:", image_url, "\n")
  cat("Country:", country, "\n")
  cat("Gender:", gender, "\n")
  cat("Competitions:", comps, "\n")
  cat("Completed Solves:", comp_solves, "\n")
  cat("Personal Bests:\n")
  print(event_data)
}

关键修改点 #

1. 精准定位PR表格：使用XPath//h3[text()='Personal Bests']/following-sibling::table[1]，通过标题"Personal Bests"找到对应的表格，避免class匹配失败的问题。
2. 优化基础信息获取：同样通过标题定位选手信息表格，避免依赖nth-child的脆弱选择器。
3. 处理空值：将空字符串转换为NA，并过滤空行，让数据更整洁。
4. 优化CSV导出：修正cbind导致的基础数据重复问题，确保每条PR数据对应一条基础信息。

内容的提问来源于stack exchange，提问作者Lucas E