爬取ESPN NBA球队赛程数据时遭遇HTTP 403错误的排查与解决咨询

我完全理解你的困惑——明明和别人用一样的代码，自己运行却一直弹出Error in open.connection(x, "rb") : HTTP error 403，这本质是ESPN的反爬机制在阻止你的请求。403错误代表服务器识别出你的请求来自自动化工具（默认配置的rvest很容易被标记），所以直接拒绝了访问。下面来拆解原因和修复方案：

错误核心原因 #

ESPN的服务器会校验请求的User-Agent头信息，rvest默认的标识类似libcurl/7.79.1 r-curl/4.3.2 httr/1.4.2，一眼就能被反爬系统判定为爬虫。别人的代码能运行，要么是他们的环境默认配置了浏览器级别的请求头，要么是ESPN的反爬规则在他们使用后更新了。

修复方案：模拟浏览器请求头 #

我们只需要给请求加上浏览器的User-Agent标识，让服务器误以为请求来自普通用户。具体步骤如下：

1. 加载httr包用于自定义请求头
2. 定义模拟浏览器的User-Agent字符串
3. 修改请求逻辑，先用带请求头的GET()获取页面，再传给read_html()解析

修改后的完整代码 #

library(rvest)
library(lubridate)
library(tidyverse)
library(stringr)
library(zoo)
library(h2o)
library(httr) # 新增用于自定义请求头的包

teams<-c("tor", "mil", "den", "gs", "ind", "phi", "okc", "por", "bos", "hou", "lac", "sa", "lal", "utah", "mia", "sac", "min", "bkn", "dal", "no", "cha", "mem", "det", "orl", "wsh", "atl", "phx", "ny", "chi", "cle")
teams_fullname<-c("Toronto", "Milwaukee", "Denver", "Golden State", "Indiana", "Philadelphia", "Oklahoma City","Portland", "Boston", "Houston", "LA", "San Antonio", "Los Angeles", "Utah", "Miami", "Sacramento", "Minnesota", "Brooklyn", "Dallas", "New Orleans", "Charlotte", "Memphis", "Detroit", "Orlando", "Washington", "Atlanta", "Phoenix", "New York", "Chicago", "Cleveland")

# 模拟Chrome浏览器的User-Agent，也可以换成你自己浏览器的真实标识
user_agent <- "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"

by_team<-{}
for (i in 1:length(teams)) {
 url<-paste0("http://www.espn.com/nba/team/schedule/_/name/", teams[i])
 
 # 带自定义请求头发起请求
 response <- GET(url, add_headers(`User-Agent` = user_agent))
 # 检查请求是否成功，避免单个失败中断整个循环
 if (http_status(response)$category != "Success") {
   warning(paste("请求", url, "失败，状态码：", response$status_code))
   next
 }
 webpage <- read_html(response)
 
 team_table <- html_nodes(webpage, 'table')
 team_c <- html_table(team_table, fill=TRUE, header = TRUE)[[1]]
 team_c<-team_c[1:which(team_c$RESULT=="TIME")-1,]
 team_c$URLTeam<-toupper(teams[i])
 team_c$FullURLTeam<-(teams_fullname[i])
 by_team<-rbind(by_team, team_c)
 
 # 增加1秒延迟，避免请求过于频繁触发反爬限制
 Sys.sleep(1)
}
# 移除延期比赛
by_team<-by_team%>%filter(RESULT!='Postponed')

额外注意事项 #

• 更换真实User-Agent：如果还是报错，可以打开自己的浏览器（比如Chrome）按F12，在Network标签随便找一个请求，复制Request Headers里的User-Agent替换代码中的值
• 控制请求频率：循环中的Sys.sleep(1)可以根据情况调整，避免短时间内大量请求导致IP被临时封禁
• 遵守爬取规则：可以查看ESPN的robots.txt文件，确认赛程数据是否允许爬取，尊重网站的爬取规范

内容的提问来源于stack exchange，提问作者Cooper Ristau