Scrapy Playwright后续请求未触发，parse_next函数未调用求助

我编写了如下Scrapy Playwright示例代码：

import json

import scrapy
import scrapy_playwright
from scrapy.selector import Selector
from scrapy_playwright.handler import Page, PageMethod


class GreetingsSpider(scrapy.Spider):
    name = "greetings"
    allowed_domains = ["example.com"]

    custom_settings = {
        "LOG_LEVEL": "INFO",
    }

    def start_requests(self):
        url = "https://www.example.com"
        yield scrapy.Request(
            url,
            callback=self.parse,
            meta={"playwright": True, "playwright_include_page": True},
        )


    async def parse(self, response):
        print("Hello ")
        page: Page = response.meta["playwright_page"]
        await page.close()
        print("Hello from parse next")
        yield scrapy.Request(
            response.url,
            callback=self.parse_next,
            meta={"playwright": True},
            errback=self.errback_close_page,
        )
        print("Hello from second parse next")
        

    def parse_next(self, response):
        print(response.url)

运行代码后，parse_next函数从未被调用，print(response.url)始终未执行。程序输出日志如下：

Hello 
Hello from parse next
Hello from second parse next
2022-11-03 07:26:03 [scrapy.core.engine] INFO: Closing spider (finished)
2022-11-03 07:26:03 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 193,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 260537,
...
}
2022-11-03 07:26:03 [scrapy.core.engine] INFO: Spider closed (finished)
2022-11-03 07:26:03 [scrapy-playwright] INFO: Closing download handler
2022-11-03 07:26:03 [scrapy-playwright] INFO: Closing download handler
2022-11-03 07:26:03 [scrapy-playwright] INFO: Closing browser

日志显示仅发起1次请求爬虫就结束了，查阅官方文档后仍未找到问题所在，求分析解决。

问题出在异步parse函数的处理逻辑以及Scrapy Playwright的配置缺失上，具体原因和修复步骤如下：

1. 缺少Scrapy Playwright必要配置 #

你的爬虫未启用Scrapy Playwright的下载中间件和处理器，导致第二个请求的meta={"playwright": True}无法生效，请求未被正确处理。需要在custom_settings中添加以下配置：

custom_settings = {
    "LOG_LEVEL": "INFO",
    "DOWNLOAD_HANDLERS": {
        "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    },
    "DOWNLOADER_MIDDLEWARES": {
        "scrapy_playwright.middleware.PlaywrightMiddleware": 543,
    },
}

2. 未定义errback函数 #

你在第二个请求中指定了errback=self.errback_close_page，但未实现该函数，会导致请求出错被丢弃。需要补充该函数：

async def errback_close_page(self, failure):
    page = failure.request.meta.get("playwright_page")
    if page:
        await page.close()

3. 重复请求被过滤 #

第二个请求和第一个请求URL相同，Scrapy默认会过滤重复请求。需要在第二个请求中添加dont_filter=True参数，避免被过滤：

yield scrapy.Request(
    response.url,
    callback=self.parse_next,
    meta={"playwright": True},
    errback=self.errback_close_page,
    dont_filter=True  # 新增参数
)

修复后的完整代码 #

import json

import scrapy
import scrapy_playwright
from scrapy.selector import Selector
from scrapy_playwright.handler import Page, PageMethod


class GreetingsSpider(scrapy.Spider):
    name = "greetings"
    allowed_domains = ["example.com"]

    custom_settings = {
        "LOG_LEVEL": "INFO",
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        "DOWNLOADER_MIDDLEWARES": {
            "scrapy_playwright.middleware.PlaywrightMiddleware": 543,
        },
    }

    def start_requests(self):
        url = "https://www.example.com"
        yield scrapy.Request(
            url,
            callback=self.parse,
            meta={"playwright": True, "playwright_include_page": True},
        )

    async def parse(self, response):
        print("Hello ")
        page: Page = response.meta["playwright_page"]
        await page.close()
        print("Hello from parse next")
        yield scrapy.Request(
            response.url,
            callback=self.parse_next,
            meta={"playwright": True},
            errback=self.errback_close_page,
            dont_filter=True
        )
        print("Hello from second parse next")

    def parse_next(self, response):
        print(response.url)

    async def errback_close_page(self, failure):
        page = failure.request.meta.get("playwright_page")
        if page:
            await page.close()

运行修复后的代码，parse_next函数会被正常调用，日志中会显示第二次请求的记录，print(response.url)也会输出对应内容。

内容的提问来源于stack exchange，提问作者asfand hikmat