Scrapy-Playwright:为每个请求创建新上下文的并发限制问题求助
问题:Scrapy-Playwright 单上下文限制下无法生成后续新上下文
我最初尝试为每个请求修改上下文的User-Agent但遇到困难,转而尝试在Scrapy-Playwright中为每个请求创建新上下文。当PLAYWRIGHT_MAX_CONTEXTS设置为1时,仅第一个上下文能成功创建并处理请求,后续上下文无法生成,导致爬虫持续运行但无后续操作。恳请提供解决思路或指导,提前致谢!
当前代码
import scrapy from .function.general import * load_dotenv() class TestSpider(scrapy.Spider): name = "test" custom_settings = { "PLAYWRIGHT_MAX_CONTEXTS" : 1 } def start_requests(self): url = "https://bot.sannysoft.com" for i in range(2): yield scrapy.Request( url=url, callback=self.parse, dont_filter=True, meta=dict( url=url, playwright=True, playwright_include_page=True, playwright_page_init_callback=init_page, number=i, playwright_context= f"new-{i}", playwright_context_kwargs= get_context_args() ), ) async def parse(self, response): page = response.meta["playwright_page"] number = response.meta["number"] await page.screenshot(path=f"headless-test-result-{number}.png") await page.close()
内容的提问来源于stack exchange,提问作者MisterFox
相关产品推荐
相关产品推荐

