使用Playwright时Scrapy无返回结果？JS网站多页爬取遇问题

问题概述 #

爬取JavaScript驱动的多页网站内容时，使用Scrapy+Playwright组合无任何返回结果，但仅使用Scrapy时运行正常。

原代码 #

import scrapy
from scrapy_playwright.page import PageMethod

class PositionsSpider(scrapy.Spider):
    name = "positions"
    allowed_domains = ["https://trafigura.com/"]
    start_urls = ["https://careers.trafigura.com/TrafiguraCareerSite/search?"]


    def start_requests(self):
        yield scrapy.Request(
            self.start_urls[0],
            meta=dict(
                playwright = True,
                playwright_page_methods = [
                    PageMethod("wait_for_selector", 'section#results div[role="list"]')
                ]

            )
        )

    async def parse(self, response):
        for job in response.css('section#results div[role=list] div[role="listitem]'):
            yield{
                'title': job.css('a::text').get()
            }

错误分析 #

1. 域名配置错误：allowed_domains不能包含https://前缀，且目标站点是careers.trafigura.com（主域为trafigura.com），原配置会触发Scrapy的域名过滤机制，直接拦截请求。
2. CSS选择器语法错误：div[role="listitem]缺少闭合双引号，导致选择器完全失效，无法匹配任何元素。
3. 页面等待策略不足：仅等待列表容器加载，无法确保列表项这类动态渲染的内容完全生成，页面可能还处于加载状态就开始解析。
4. 异步方法风格不统一：start_requests为同步方法，但parse是异步方法，可能导致Playwright的异步交互逻辑出现异常。

修复后的代码 #

import scrapy
from scrapy_playwright.page import PageMethod

class PositionsSpider(scrapy.Spider):
    name = "positions"
    allowed_domains = ["trafigura.com"]  # 修正域名格式，覆盖子域名
    start_urls = ["https://careers.trafigura.com/TrafiguraCareerSite/search?"]

    # 改为异步start_requests，与parse方法风格统一
    async def start_requests(self):
        yield scrapy.Request(
            self.start_urls[0],
            meta=dict(
                playwright=True,
                playwright_page_methods=[
                    # 先等待网络空闲，确保页面资源加载完成
                    PageMethod("wait_for_load_state", "networkidle"),
                    # 直接等待列表项元素，确保内容渲染完成
                    PageMethod("wait_for_selector", 'section#results div[role="list"] div[role="listitem"]')
                ],
                # 设置超时时间，避免无限等待
                playwright_timeout=30000
            )
        )

    async def parse(self, response):
        # 修正CSS选择器的引号问题
        for job in response.css('section#results div[role="list"] div[role="listitem"]'):
            yield {
                'title': job.css('a::text').get().strip() if job.css('a::text').get() else None
            }

额外配置要求 #

确保Scrapy项目的settings.py中添加以下Playwright相关配置：

PLAYWRIGHT_ENABLED = True
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

内容的提问来源于stack exchange，提问作者Anish Thapa