使用while循环在R中爬取NBA数据时遇HTTP 429错误及内存问题

问题描述 #

我编写while循环爬取2003-2017年NBA常规赛所有月份的数据，遇到两个核心问题：

1. 网页真实存在且可访问时，仍持续返回错误：Error in open.connection(x, "rb") : HTTP error 429；虽用try处理无比赛月份的异常，但未达到跳过无效月份继续执行的效果。
2. 循环执行时内存占用率高达95%，不确定是否会影响循环执行及目标数据表生成。

原代码如下：

library(rvest)
library(dplyr)
mnths = c("october","november","december","january","february","march","april","may")
#list of months to cycle through
yrs = seq(2003,2017)
#list of years to cycle through
url_base = "https://www.basketball-reference.com/leagues/NBA_"
#beginning of webpage URL
#https://www.basketball-reference.com/leagues/NBA_2003_games-october.html
#above is the final webpage formatted for the 1st run of the loop as an example
i = 1
j = 1

while(i<=length(yrs)){
#begin loop to cycle through each year

  while(j <= length(mnths)){
  #begin subloop to cycle through each month in a year

    webpage = paste(paste(paste(paste(url_base,yrs[i],sep = ""),"_games-",sep = ""),mnths[j],sep = ""),".html",sep = "")
    #string variable of webpage with specific month and year in loop
    webpageexists = try(read_html(webpage) %>% html_node(), silent = TRUE)
    #try variable to check if webpage exists

    if(webpageexists == "try-error"){
    #if statement to check if webpage exists, if not variable "webpageexists" will be a try-error and the month will be incremented and subloop continues
      j = j + 1
      rm(webpageexists)
      #removing try variable from memory
    }else if(exists("tb")){
      tbx = as.data.frame(read_html(webpage) %>% html_nodes("table") %>% html_table())
      #table created to contain new data from specific webpage in loop
      tb = rbind(tb,tbx)
      #table holding all data from all runs of loop
      j = j + 1
      rm(webpageexists)
      #removing try variable from memory
    }else{
      tb = as.data.frame(read_html(webpage) %>% html_nodes("table") %>% html_table())
      #table that is created that all new tables will be merged into
      #this else statement is only used on the very first run of the loop
      j = j + 1
      rm(webpageexists)
      #removing try variable from memory
    }
    
  }
  #end subloop to cycle through each month in a year

  j = 1
  #j reset to 1 so that the next year starts at the first month in the "mnths" list
  i = i + 1
  #i is incremented by 1 to move to the next year in the "yrs" list
}
#end loop to cycle through each year

问题分析与解决 #

1. HTTP 429错误的解决 #

HTTP 429是网站因请求频率过高触发的限流机制，同时你的try判断逻辑存在错误：

• 修复try判断逻辑：原代码用webpageexists == "try-error"是错误的，正确判断try错误的方式是inherits(webpageexists, "try-error")。
• 避免重复请求：原代码在判断网页存在后，又重新调用read_html请求一次，既浪费资源又加剧限流，应该复用已经读取的HTML对象。
• 添加请求延迟：每次请求后添加Sys.sleep()，控制请求间隔（建议2-5秒），降低触发反爬的概率。

2. 内存占用过高的优化 #

内存占用95%会显著拖慢程序，甚至导致崩溃，核心原因是每次用rbind合并数据框时会复制整个数据集，内存开销随循环次数线性增长：

• 改用列表存储子数据框：先将每个月份的数据存到列表中，最后一次性合并成总数据框，避免频繁复制大对象。
• 减少临时变量冗余：清理不必要的临时变量，可在每处理完一年后调用gc()主动触发垃圾回收，释放内存。

优化后的代码 #

library(rvest)
library(dplyr)

mnths <- c("october","november","december","january","february","march","april","may")
yrs <- seq(2003,2017)
url_base <- "https://www.basketball-reference.com/leagues/NBA_"

# 用列表存储所有子数据框，避免频繁rbind
data_list <- list()
list_index <- 1

for(i in seq_along(yrs)) {
  current_year <- yrs[i]
  for(j in seq_along(mnths)) {
    current_month <- mnths[j]
    webpage <- paste0(url_base, current_year, "_games-", current_month, ".html")
    
    # 尝试读取网页，捕获错误
    html_obj <- try(read_html(webpage), silent = TRUE)
    
    if(inherits(html_obj, "try-error")) {
      next
    }
    
    # 提取表格数据
    tbx <- html_obj %>% html_nodes("table") %>% html_table() %>% as.data.frame()
    # 添加年份和月份标识，方便后续数据处理
    tbx$year <- current_year
    tbx$month <- current_month
    
    data_list[[list_index]] <- tbx
    list_index <- list_index + 1
    
    # 添加请求延迟，避免限流
    Sys.sleep(3)
  }
  
  # 每处理完一年，触发一次垃圾回收
  gc()
}

# 合并所有子数据框
tb <- bind_rows(data_list)

内容的提问来源于stack exchange，提问作者Omar123456789