使用R的httr包请求US BLS接口返回403，Python requests则返回200

尝试从美国劳工统计局（BLS）获取CPI发布日期时，用R的httr包请求https://www.bls.gov/bls/news-release/cpi.htm返回403状态码，但用Python requests包发起相同User-Agent的请求却返回200。

问题代码示例 #

R代码（返回403） #

library(httr)

p = GET(
  "https://www.bls.gov/bls/news-release/cpi.htm",
  user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36")
)
status_code(p)
# [1] 403

Python代码（返回200） #

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36'
}

p = requests.get("https://www.bls.gov/bls/news-release/cpi.htm", headers=headers)

p.status_code
# 200

问题原因 #

差异核心在于httr默认携带的请求头与Python requests的默认请求头不一致，BLS的服务器反爬机制识别到httr的请求特征后进行了拦截。比如：

• httr默认的Accept头为*/*，而Python requests默认的Accept头更贴近真实浏览器的取值
• httr缺少Accept-Language、Upgrade-Insecure-Requests等浏览器常用请求头

解决方案 #

手动添加完整的请求头，模拟Python requests的默认请求头集合，让R的请求更贴近浏览器或Python请求的格式：

修正后的R代码 #

library(httr)

# 模拟Python requests默认的请求头组合
request_headers = add_headers(
  `User-Agent` = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36",
  `Accept` = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
  `Accept-Language` = "en-US,en;q=0.5",
  `Accept-Encoding` = "gzip, deflate, br",
  `Connection` = "keep-alive",
  `Upgrade-Insecure-Requests` = "1"
)

p = GET(
  "https://www.bls.gov/bls/news-release/cpi.htm",
  request_headers
)

status_code(p)
# 预期返回200

验证思路 #

可以通过str(p$headers)查看httr默认发送的请求头，对比Python中print(p.request.headers)输出的请求头，精准定位差异字段，针对性补充即可解决403拦截问题。

内容的提问来源于stack exchange，提问作者Mislav Sagovac