求助：如何用Scrapy提取Fangraphs表格中的球员姓名？

问题核心 #

Fangraphs的球员数据表格是JavaScript动态渲染的，直接请求静态HTML拿不到有效数据，同时你使用的CSS选择器也没有命中正确的表格行元素。

可行解决方案 #

1. 直接请求API接口（推荐，效率更高） #

Fangraphs的榜单数据是通过XHR接口返回的JSON格式，无需解析HTML，直接请求接口更高效：

import scrapy
import json

class FangraphsPlayersSpider(scrapy.Spider):
    name = 'fangraphs_players'
    # 直接调用数据接口，参数与原页面一致
    start_urls = [
        'https://www.fangraphs.com/api/leaders/splits?splitArr=&splitArrPitch=&position=B&autoPt=true&splitTeams=false&statType=player&statgroup=1&startDate=2023-03-01&endDate=2023-11-01&players=&filter=&groupBy=season&wxTemperature=&wxPressure=&wxAirDensity=&wxElevation=&wxWindSpeed=&sort=22,1&pageitems=30&page=1'
    ]

    def parse(self, response):
        data = json.loads(response.text)
        # 遍历JSON数据提取姓名
        for player in data['data']:
            yield {
                'player_name': player['Name'],
                # 可按需提取其他字段，比如球队、WAR值等
                'team': player['Team'],
                'war': player['WAR']
            }

2. 用Playwright渲染动态HTML（适合必须解析页面的场景） #

如果一定要通过解析HTML提取数据，需要结合scrapy-playwright处理JavaScript渲染：

• 先安装依赖：pip install scrapy-playwright
• 在项目settings.py中添加配置：

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
PLAYWRIGHT_LAUNCH_OPTIONS = {"headless": True}

• 编写爬虫代码：

import scrapy

class FangraphsHtmlSpider(scrapy.Spider):
    name = 'fangraphs_html'
    start_urls = [
        'https://www.fangraphs.com/leaders/splits-leaderboards?splitArr=&splitArrPitch=&position=B&autoPt=true&splitTeams=false&statType=player&statgroup=1&startDate=2023-03-01&endDate=2023-11-01&players=&filter=&groupBy=season&wxTemperature=&wxPressure=&wxAirDensity=&wxElevation=&wxWindSpeed=&sort=22,1'
    ]

    def start_requests(self):
        for url in self.start_urls:
            # 启用Playwright渲染
            yield scrapy.Request(url, meta={"playwright": True})

    def parse(self, response):
        # 正确定位表格行与姓名元素：数据行类名为player-row，姓名在td下的a标签中
        rows = response.css('tr.player-row')
        for row in rows:
            player_name = row.css('td a::text').get()
            if player_name:
                yield {'player_name': player_name.strip()}

原选择器失效原因 #

• tr.is-selected_invalid是标记选中无效行的样式类，并非普通数据行的类名
• td.alight-left.fixed存在拼写错误（应为align-left），且即使正确，也不如直接定位到包含姓名的<a>标签精准

内容的提问来源于stack exchange，提问作者Troy Wakimoto