如何用Scrapy-Playwright点击「show more」获取ESPN NBA全部球员数据

抓取ESPN 2023赛季NBA球员数据页面（https://www.espn.com/nba/stats/player/_/season/2023/seasontype/2）时，仅能获取50条数据，需要循环点击「show more」按钮直到按钮消失来获取全部数据。使用Scrapy+Playwright，已定位到按钮但仅执行一次点击，且页面刷新后数据回到50条，求实现循环点击并获取全量数据的方法。

核心问题是当前代码只执行了一次按钮点击，且直接使用初始响应的body解析数据（此时数据还没完全加载）。需要在parse方法里实现循环点击+等待加载+实时解析页面的逻辑：

• 循环判断「show more」按钮是否存在且可点击，直到按钮消失
• 每次点击后等待数据加载完成（比如等待按钮状态变化或表格更新）
• 不要使用初始的response.body，而是从Playwright的page中获取最新的HTML内容解析

from scrapy import Selector, Request
from scrapy_playwright.page import PageMethod

class NBAPlayerStatsSpider(scrapy.Spider):
    name = "nba_player_stats"

    def start_requests(self):
        yield Request(
            url='https://www.espn.com/nba/stats/player/_/season/2023/seasontype/2',
            meta=dict(
                playwright=True,
                playwright_include_page=True,
                # 初始仅等待页面基础加载完成，点击逻辑放到parse中循环执行
                playwright_page_coroutines=[
                    PageMethod('wait_for_selector', "//table[@class='Table Table--align-right Table--fixed Table--fixed-left']//tbody//tr"),
                ]
            ),
            callback=self.parse,
        )

    async def parse(self, response):
        page = response.meta["playwright_page"]
        button_locator = page.locator("//a[@class='AnchorLink loadMore__link']")

        while True:
            # 检查按钮是否存在
            button_count = await button_locator.count()
            if button_count == 0:
                break
            
            # 点击按钮，等待数据加载完成（按钮隐藏代表加载完成）
            await button_locator.click()
            await button_locator.wait_for(state="hidden", timeout=10000)
            # 额外延迟确保DOM完全更新
            await page.wait_for_timeout(1000)

        # 所有数据加载完成后，获取最新页面HTML
        page_html = await page.content()
        sel = Selector(text=page_html)

        player_list = sel.xpath(
            "//table[@class='Table Table--align-right Table--fixed Table--fixed-left']//tbody//tr")
        stats_list = sel.xpath(
            "//div[@class='Table__ScrollerWrapper relative overflow-hidden']/div[@class='Table__Scroller']/table/tbody/tr")

        for player, stat in zip(player_list, stats_list):
            player_name = player.xpath(".//a/text()").get()
            position = stat.xpath(".//td/div/text()").get()
            team_name = player.xpath(".//span/text()").get()
            game_played = stat.xpath(".//td[2]/text()").get()
            minutes_per_game = stat.xpath(".//td[3]/text()").get()
            points_per_game = stat.xpath(".//td[4]/text()").get()
            fields_goal_made = stat.xpath(".//td[5]/text()").get()
            fields_goal_attempted = stat.xpath(".//td[6]/text()").get()
            field_goal_percentage = stat.xpath(".//td[7]/text()").get()
            three_point_goal_made = stat.xpath(".//td[8]/text()").get()

            yield {
                "player_name": player_name,
                "player_position": position,
                "team_name": team_name,
                "game_played": game_played,
                "minutes_per_game": minutes_per_game,
                "points_per_game": points_per_game,
                "fields_goal_made": fields_goal_made,
                "fields_goal_attempted": fields_goal_attempted,
                "field_goal_percentage": field_goal_percentage,
                "three_point_goal_made": three_point_goal_made,
            }
        
        # 关闭页面释放资源
        await page.close()

• 循环逻辑：通过while True循环，每次检查按钮数量，为0时退出循环
• 加载等待：点击按钮后等待按钮隐藏（说明数据加载完成），再加1秒延迟确保DOM完全更新
• 数据解析时机：所有按钮点击完成后，才获取最新的页面HTML进行解析，避免只拿到初始的50条数据
• 变量修正：原代码中minutes_per_minute变量名错误，修正为minutes_per_game更准确

内容的提问来源于stack exchange，提问作者YouCanCallMe Syarif