如何用Python爬取2023版PGA Tour网站数据

背景与需求 #

PGA Tour官网在2023年2月7日完成更新，彻底导致我之前的数据爬取方法失效。此前我通过开发者工具的Network标签找到隐藏URL，结合Python的Requests库拉取数据表。

现在需要实现两个爬取需求：

• 爬取任意赛事任意年份/赛季的历史数据（新站示例为赛事过往结果页）
• 爬取任意统计项任意年份/赛季的数据（新站示例为统计详情页）

我的初始尝试只能拉取当前页面的表格（无法获取往年数据），且部分数据显示为格式代码而非文本，初始代码如下：

import requests
import pandas as pd
tournament_url = 'https://www.pgatour.com/tournaments/2023/fortinet-championship/R2023464/past-results'
headers = {'User-Agent': 'Mozilla/5.0'}
t = pd.read_html(requests.get(tournament_url, headers=headers).text)[0]
t

编辑1：发现GraphQL接口 #

从回复中得知网站采用GraphQL，我在Network标签的graphql项里看到Payload包含变量：{ "tournamentPastResultsId": "R2023464", "year": 2022 }。这些变量包含赛事ID和年份，理论上修改变量就能爬取任意赛事任意年份的数据，但我不清楚怎么把它集成到爬取逻辑里，打算研究用Selenium传递这些变量。

编辑2：统计数据爬取成功实现 #

下方回复给出了赛事数据的爬取方法，我修改代码后成功获取了统计数据（示例为某统计项详情页），代码如下（感谢@Jurakin！）：

import pandas as pd
from numpy import NaN
import requests
# 请求头中需要固定的x-api-key令牌
X_API_KEY = "da2-gsrx5bibzbb4njvhl7t37wqyl4"
YEAR = 2022  # 统计赛季
STAT_ID = "02567"  # 统计项ID
# 构造请求负载
payload = {
    "operationName": "StatDetails",
    "variables": {
        "tourCode": "R",
        "statId": STAT_ID,
        "year": YEAR,
        "eventQuery": None
    },
    "query": "query StatDetails($tourCode: TourCode!, $statId: String!, $year: Int, $eventQuery: StatDetailEventQuery) {\n  statDetails(\n    tourCode: $tourCode\n    statId: $statId\n    year: $year\n    eventQuery: $eventQuery\n  ) {\n    tourCode\n    year\n    displaySeason\n    statId\n    statType\n    tournamentPills {\n      tournamentId\n      displayName\n    }\n    yearPills {\n      year\n      displaySeason\n    }\n    statTitle\n    statDescription\n    tourAvg\n    lastProcessed\n    statHeaders\n    statCategories {\n      category\n      displayName\n      subCategories {\n        displayName\n        stats {\n          statId\n          statTitle\n        }\n      }\n    }\n    rows {\n      ... on StatDetailsPlayer {\n        __typename\n        playerId\n        playerName\n        country\n        countryFlag\n        rank\n        rankDiff\n        rankChangeTendency\n        stats {\n          statName\n          statValue\n          color\n        }\n      }\n      ... on StatDetailTourAvg {\n        __typename\n        displayName\n        value\n      }\n    }\n  }\n}"  
  }
# 发送POST请求
page = requests.post("https://orchestrator.pgatour.com/graphql", json=payload, headers={"x-api-key": X_API_KEY})
# 检查响应状态码
page.raise_for_status()
# 提取数据
data = page.json()["data"]["statDetails"]["rows"]
# print(data)
# 格式化为网页中的表格结构
table = map(lambda item: {
    "rank": item["rank"],
    "player": item["playerName"],
    "average": item["stats"][0]["statValue"],
}, data)
# 转换为DataFrame
s = pd.DataFrame(table)
s

编辑3：后续问题 - 3位字符Stat ID的适配问题 #

上述统计数据爬取代码适用于5位字符的Stat ID，但对于3位字符的Stat ID（示例为Birdie Average的156），数据能正常获取，但表格映射部分失败，我找不到原因，对应代码如下：

import pandas as pd
from numpy import NaN
import requests
# 请求头中需要固定的x-api-key令牌
X_API_KEY = "da2-gsrx5bibzbb4njvhl7t37wqyl4"
YEAR = 2022  # 统计赛季
# STAT_ID = "02567"  # 统计项ID SGOTT
STAT_ID = "156"  # Birdie Average - 三位数字的统计项无法正常工作，找不到原因
# 构造请求负载
payload = {
    "operationName": "StatDetails",
    "variables": {
        "tourCode": "R",
        "statId": STAT_ID,
        "year": YEAR,
        "eventQuery": None
    },
    "query": "query StatDetails($tourCode: TourCode!, $statId: String!, $year: Int, $eventQuery: StatDetailEventQuery) {\n  statDetails(\n    tourCode: $tourCode\n    statId: $statId\n    year: $year\n    eventQuery: $eventQuery\n  ) {\n    tourCode\n    year\n    displaySeason\n    statId\n    statType\n    tournamentPills {\n      tournamentId\n      displayName\n    }\n    yearPills {\n      year\n      displaySeason\n    }\n    statTitle\n    statDescription\n    tourAvg\n    lastProcessed\n    statHeaders\n    statCategories {\n      category\n      displayName\n      subCategories {\n        displayName\n        stats {\n          statId\n          statTitle\n        }\n      }\n    }\n    rows {\n      ... on StatDetailsPlayer {\n        __typename\n        playerId\n        playerName\n        country\n        countryFlag\n        rank\n        rankDiff\n        rankChangeTendency\n        stats {\n          statName\n          statValue\n          color\n        }\n      }\n      ... on StatDetailTourAvg {\n        __typename\n        displayName\n        value\n      }\n    }\n  }\n}"  
  }
# 发送POST请求
page = requests.post("https://orchestrator.pgatour.com/graphql", json=payload, headers={"x-api-key": X_API_KEY})
# 检查响应状态码
page.raise_for_status()
# 提取数据
data = page.json()["data"]["statDetails"]["rows"]
print(data)
# 格式化为网页中的表格结构
table = map(lambda item: {
    "RANK": item["rank"],
    "PLAYER": item["playerName"],
    "AVERAGE": item["stats"][0]["statValue"],
}, data)
# 转换为DataFrame
s = pd.DataFrame(table)
s

内容的提问来源于Stack Exchange，提问作者Ryan Miller