You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

Scrapy-Playwright:为每个请求创建新上下文的并发限制问题求助

问题:Scrapy-Playwright 单上下文限制下无法生成后续新上下文

我最初尝试为每个请求修改上下文的User-Agent但遇到困难,转而尝试在Scrapy-Playwright中为每个请求创建新上下文。当PLAYWRIGHT_MAX_CONTEXTS设置为1时,仅第一个上下文能成功创建并处理请求,后续上下文无法生成,导致爬虫持续运行但无后续操作。恳请提供解决思路或指导,提前致谢!

当前代码

import scrapy
from .function.general import *

load_dotenv()

class TestSpider(scrapy.Spider):
    name = "test"
    custom_settings = {
        "PLAYWRIGHT_MAX_CONTEXTS" : 1
    }

    def start_requests(self):
        url = "https://bot.sannysoft.com"
        
        for i in range(2):
            yield scrapy.Request(
                url=url,
                callback=self.parse,
                dont_filter=True,
                meta=dict(
                    url=url,
                    playwright=True,
                    playwright_include_page=True,
                    playwright_page_init_callback=init_page,
                    number=i,
                    playwright_context= f"new-{i}",
                    playwright_context_kwargs= get_context_args()
                ),
            )

    async def parse(self, response):
        page = response.meta["playwright_page"]
        number = response.meta["number"]
        await page.screenshot(path=f"headless-test-result-{number}.png")
        await page.close()

内容的提问来源于stack exchange,提问作者MisterFox

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.07.16 00:40:55