使用Rvest爬取美国国务院网站遭遇403访问禁止问题求助

问题背景 #

此前定期使用Rvest爬取美国国务院新闻稿，突然收到403访问禁止响应，更换多台电脑及云平台尝试后结果一致，且无法访问该站点的RSS订阅源https://www.state.gov/rss-feed/press-releases/feed/，核心代码如下：

options(stringsAsFactors = F)         # no automatic data transformation
options("scipen" = 999, "digits" = 4) # suppress math annotation
library(tidyverse)
library(rvest)
library(readtext)
library(webdriver)
library(gsubfn)
library(lubridate)
library(stringr)
library(readxl)
library(urltools)
library(crul)
############################********Department_of_State*****##################################

HEADERS <- list(
  "User-Agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
  
)


US_Gov=tibble(url="https://www.state.gov/press-releases/page/",page=1)
US_Gov=rbind(US_Gov, US_Gov[rep(1, 4), ])
US_Gov=US_Gov %>% mutate(page=row_number())
US_Gov$page=paste0(US_Gov$url,US_Gov$page,"/")

df=US_Gov$page


ttt=lapply(df,function(x){
  
  HttpClient$new(x, header=HEADERS)$get()
})



t4=lapply(ttt,function(x){
  read_html(x$parse())
})

links=lapply(t4,function(x){
  html_attr(html_nodes(x, "a"), "href")
  
})

class=lapply(t4,function(x){
  html_attr(html_nodes(x, "a"), "class")
  
})



US_Links=plyr::ldply(links, cbind)
US_Class=plyr::ldply(class, cbind)


US_Data=bind_cols(US_Links,US_Class)

解决方案 #

1. 完善请求头字段 #

单一User-Agent易被识别为爬虫，补充更多真实浏览器的请求头信息，模拟正常用户访问：

HEADERS <- list(
  "User-Agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
  "Accept" = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
  "Accept-Language" = "en-US,en;q=0.5",
  "Referer" = "https://www.state.gov/",
  "DNT" = "1",
  "Connection" = "keep-alive",
  "Upgrade-Insecure-Requests" = "1"
)

2. 添加请求延迟 #

频繁无间隔的请求会触发反爬机制，在每次请求间添加随机延迟：

ttt <- lapply(df, function(x){
  Sys.sleep(runif(1, min = 1, max = 3)) # 随机延迟1-3秒
  HttpClient$new(x, header=HEADERS)$get()
})

3. 使用Session维持会话状态 #

改用crul的Session对象维持会话，模拟用户连续访问的行为，避免每次请求都新建连接：

session <- Session$new(url = "https://www.state.gov", headers = HEADERS)
ttt <- lapply(df, function(x){
  Sys.sleep(runif(1, 1, 3))
  session$get(x)
})

4. 优化RSS源访问 #

针对RSS源，添加适配的Accept请求头，确保请求格式匹配：

rss_headers <- list(
  "User-Agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
  "Accept" = "application/rss+xml,application/xml;q=0.9,*/*;q=0.8"
)
rss_response <- HttpClient$new("https://www.state.gov/rss-feed/press-releases/feed/", header = rss_headers)$get()

# 若请求成功，解析RSS内容
if(rss_response$status_code == 200){
  library(xml2)
  rss_xml <- read_xml(rss_response$parse())
  titles <- xml_text(xml_find_all(rss_xml, "//item/title"))
  links <- xml_text(xml_find_all(rss_xml, "//item/link"))
}

5. 排查IP封禁问题 #

若上述方法无效，可能IP段被网站封禁：

• 查看网站robots.txt（访问https://www.state.gov/robots.txt）确认爬取权限
• 尝试使用合规代理IP更换访问IP（注意遵守网站使用条款）
• 联系网站管理员确认是否存在误封禁情况

内容的提问来源于stack exchange，提问作者Mark Peter