如何用Scrapy获取ESPN NBA球员统计全量数据？

我正在学习使用Scrapy进行数据爬取，想要获取ESPN NBA球员统计页面的全量数据，页面地址为：

• https://www.espn.com/nba/stats/player
• https://www.espn.com/nba/stats/player/_/season/2023/seasontype/2

访问页面可见数据底部有show more按钮，目前仅能爬取50条数据，该按钮的属性仅为href=#。

一、用户提供的Scrapy+Playwright代码 #

def start_requests(self):
        yield scrapy.Request(
            url='https://www.espn.com/nba/stats/player/_/season/2023/seasontype/2',
            meta= dict(
                playwright = True,
                playwright_include_page=True,
                playwright_page_coroutines = [
                    PageMethod('wait_for_selector','//a[@class="AnchorLink loadMore__link"]'),
                    PageMethod('click','//a[@class="AnchorLink loadMore__link"]'),
                ]
            ),
            callback=self.parse,
        )

    async def parse(self, response):
        page = response.meta['playwright_page']

        button = response.meta['playwright_page_coroutines'][0]
        if button:
            await button.click()

            resp = response.body
            player_list = sel.xpath(
                "//table[@class='Table Table--align-right Table--fixed Table--fixed-left']//tbody//tr")
            stats_list = sel.xpath(
                "//div[@class='Table__ScrollerWrapper relative overflow-hidden']/div[@class='Table__Scroller']/table/tbody/tr")

            await page.wait_for_selector(player_list)
            
            sel = Selector(text=resp)

            for player, stat in zip(player_list, stats_list):
                    player_name = player.xpath(".//a/text()").get()
                    position = stat.xpath(".//td/div/text()").get()
                    team_name = player.xpath(".//span/text()").get()
                    game_played = stat.xpath(".//td[2]/text()").get()
                    minutes_per_minute = stat.xpath(".//td[3]/text()").get()
                    points_per_game = stat.xpath(".//td[4]/text()").get()
                    fields_goal_made = stat.xpath(".//td[5]/text()").get()
                    fields_goal_attempted = stat.xpath(".//td[6]/text()").get()
                    field_goal_percentage = stat.xpath(".//td[7]/text()").get()
                    three_point_goal_made = stat.xpath(".//td[8]/text()").get()

            yield {
                        "player_name": player_name,
                        "player_position": position,
                        "team_name": team_name,
                        "game_played": game_played,
                        "minutes_per_minute": minutes_per_minute,
                        "points_per_game": points_per_game,
                        "fields_goal_made": fields_goal_made,
                        "fields_goal_attempted": fields_goal_attempted,
                        "field_goal_percentage": field_goal_percentage,
                        "three_point_goal_made": three_point_goal_made,
                    }

二、用户提供的仅Scrapy代码 #

def start_requests(self):
        yield scrapy.Request(
            url='https://www.espn.com/nba/stats/player/_/season/2023/seasontype/2',
            callback=self.parse,
        )
 def parse(self, response):
        sel = Selector(text=response.body)

        player_list = sel.xpath("//table[@class='Table Table--align-right Table--fixed Table--fixed-left']//tbody//tr")
        stats_list = sel.xpath("//div[@class='Table__ScrollerWrapper relative overflow-hidden']/div[@class='Table__Scroller']/table/tbody/tr")

        for player,stat in zip(player_list,stats_list):
            player_name = player.xpath(".//a/text()").get()
            position = stat.xpath(".//td/div/text()").get()
            team_name = player.xpath(".//span/text()").get()
            game_played = stat.xpath(".//td[2]/text()").get()
            minutes_per_minute= stat.xpath(".//td[3]/text()").get()
            points_per_game = stat.xpath(".//td[4]/text()").get()
            fields_goal_made = stat.xpath(".//td[5]/text()").get()
            fields_goal_attempted = stat.xpath(".//td[6]/text()").get()
            field_goal_percentage = stat.xpath(".//td[7]/text()").get()
            three_point_goal_made = stat.xpath(".//td[8]/text()").get()

            yield {
                "player_name": player_name,
                "player_position":position,
                "team_name": team_name,
                "game_played": game_played,
                "minutes_per_minute": minutes_per_minute,
                "points_per_game": points_per_game,
                "fields_goal_made":fields_goal_made,
                "fields_goal_attempted":fields_goal_attempted,
                "field_goal_percentage":field_goal_percentage,
                "three_point_goal_made":three_point_goal_made,
            }

用户疑问：
请问我的代码是否存在错误？另外点击show more按钮时会出现对应的API接口，我可以通过该API爬取数据，但目前希望通过HTML的XPath方式获取全量数据。

代码错误分析与修正 #

一、仅Scrapy代码的问题 #

仅用Scrapy无法获取全量数据，因为show more按钮是通过前端JS动态加载数据的，Scrapy默认只能抓取初始HTML里的50条数据，无法触发JS事件加载更多内容，必须用Playwright这类无头浏览器工具辅助。

二、Scrapy+Playwright代码的错误 #

1. 重复点击按钮：playwright_page_coroutines里已经设置了点击操作，会在请求完成前自动执行，无需在parse方法里重复调用。
2. Selector初始化顺序错误：先调用了sel.xpath但未初始化sel，应该先获取页面最新HTML再初始化Selector。
3. wait_for_selector用法错误：该方法需要传入选择器字符串，而非xpath查询结果。
4. yield位置错误：将yield放在循环外，只会输出最后一条数据，需放在循环内部。
5. 未处理多次点击：一次show more仅加载50条，需循环点击直到按钮消失才能获取全量数据。

修正后的Scrapy+Playwright代码 #

from scrapy import Spider, Selector
from scrapy_playwright.page import PageMethod

class NBAPlayerStatsSpider(Spider):
    name = "nba_player_stats"

    def start_requests(self):
        yield scrapy.Request(
            url='https://www.espn.com/nba/stats/player/_/season/2023/seasontype/2',
            meta=dict(
                playwright=True,
                playwright_include_page=True,
            ),
            callback=self.parse,
        )

    async def parse(self, response):
        page = response.meta['playwright_page']
        show_more_selector = '//a[@class="AnchorLink loadMore__link"]'

        # 循环点击show more直到按钮消失
        while True:
            try:
                await page.wait_for_selector(show_more_selector, timeout=3000)
                await page.click(show_more_selector)
                await page.wait_for_timeout(1000)  # 等待数据加载，可替换为更精准的等待逻辑
            except:
                break

        # 获取页面完整HTML并解析
        page_html = await page.content()
        sel = Selector(text=page_html)

        player_list = sel.xpath("//table[@class='Table Table--align-right Table--fixed Table--fixed-left']//tbody//tr")
        stats_list = sel.xpath("//div[@class='Table__ScrollerWrapper relative overflow-hidden']/div[@class='Table__Scroller']/table/tbody/tr")

        # 遍历输出所有数据
        for player, stat in zip(player_list, stats_list):
            player_name = player.xpath(".//a/text()").get()
            position = stat.xpath(".//td/div/text()").get()
            team_name = player.xpath(".//span/text()").get()
            game_played = stat.xpath(".//td[2]/text()").get()
            minutes_per_game = stat.xpath(".//td[3]/text()").get()
            points_per_game = stat.xpath(".//td[4]/text()").get()
            fields_goal_made = stat.xpath(".//td[5]/text()").get()
            fields_goal_attempted = stat.xpath(".//td[6]/text()").get()
            field_goal_percentage = stat.xpath(".//td[7]/text()").get()
            three_point_goal_made = stat.xpath(".//td[8]/text()").get()

            yield {
                "player_name": player_name,
                "player_position": position,
                "team_name": team_name,
                "game_played": game_played,
                "minutes_per_game": minutes_per_game,
                "points_per_game": points_per_game,
                "fields_goal_made": fields_goal_made,
                "fields_goal_attempted": fields_goal_attempted,
                "field_goal_percentage": field_goal_percentage,
                "three_point_goal_made": three_point_goal_made,
            }

        await page.close()

说明 #

1. 循环点击按钮直到消失，确保加载所有数据。
2. 点击后等待短暂时间让数据加载完成，也可替换为等待表格行数增加等更精准的逻辑。
3. 从页面获取最新HTML解析，保证拿到全量数据。
4. 将yield放在循环内，确保每条数据都被输出。
5. 修正变量名minutes_per_minute为minutes_per_game，更符合数据含义。

内容的提问来源于stack exchange，提问作者YouCanCallMe Syarif