如何使用R爬取stats.nba.com球队高阶数据表格 现有方案失效

问题原因 #

你两次尝试失败、以及拿到res后找不到表格的核心原因是：NBA stats站点的表格是前端JavaScript动态渲染生成的，原始HTML中没有现成的<table>节点，数据是预先内嵌在页面的<script>标签中、或者通过单独的接口请求拉取的。你之前直接调用API无响应是因为缺少API要求的请求头校验字段。

可行解决方案 #

方案1：调用官方统计API（更稳定，推荐） #

你之前抓的API地址是对的，只需要补全对应的请求头就能正常返回数据，不需要带复杂的cookie：

library(httr)
library(jsonlite)

# 补全API要求的请求头，核心校验字段为Host和Referer
api_headers <- c(
  `Host` = "stats.nba.com",
  `Referer` = "https://www.nba.com/",
  `User-Agent` = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36",
  `Accept` = "application/json, text/plain, */*"
)

# 调用官方统计接口
api_res <- GET(
  url = "https://stats.nba.com/stats/leaguedashteamstats",
  query = list(
    Conference = "",
    DateFrom = "",
    DateTo = "",
    Division = "",
    GameScope = "",
    GameSegment = "",
    LastNGames = 0,
    LeagueID = "00",
    Location = "",
    MeasureType = "Advanced",
    Month = 0,
    OpponentTeamID = 0,
    Outcome = "",
    PORound = 0,
    PaceAdjust = "N",
    PerMode = "PerGame",
    Period = 0,
    PlayerExperience = "",
    PlayerPosition = "",
    PlusMinus = "N",
    Rank = "N",
    Season = "2020-21",
    SeasonSegment = "",
    SeasonType = "Regular Season",
    ShotClockRange = "",
    StarterBench = "",
    TeamID = 0,
    TwoWay = 0,
    VsConference = "",
    VsDivision = ""
  ),
  add_headers(.headers = api_headers)
)

# 解析返回JSON生成结构化表格
raw_data <- fromJSON(content(api_res, as = "text"))
advanced_table <- as.data.frame(raw_data$resultSets$rowSet[[1]])
colnames(advanced_table) <- raw_data$resultSets$headers[[1]]

方案2：从你已拿到的res中提取数据 #

如果不想调用API，直接解析你已经拿到的页面响应，提取内嵌的JSON数据即可：

library(rvest)
library(stringr)
library(jsonlite)

# 从res中读取HTML文本
page_content <- content(res, as = "text")

# 匹配页面内嵌的初始数据JSON
json_str <- str_match(page_content, 'window.__INITIAL_STATE__ = (.*?);\\s*<\\/script>')[,2]
page_data <- fromJSON(json_str)

# 提取目标表格数据
advanced_table <- as.data.frame(page_data$stats$teams$advanced$table$rows)
colnames(advanced_table) <- page_data$stats$teams$advanced$table$columns$field

以上两种方案都可以稳定拿到2020-21赛季常规赛球队进阶统计表格。

内容的提问来源于stack exchange，提问作者Canovice