Scrapy-playwright多start_urls无法生成请求问题求助

你代码里的核心问题是循环遍历了错误的变量：在start_requests方法中，你把CSV里的URL读取到了局部变量start_urls，但后续循环用的是self.start_urls——这个类属性你从未赋值，所以循环根本没执行，自然不会生成任何请求。

修复步骤 #

1. 将循环对象从self.start_urls改为刚读取的局部变量start_urls
2. 同步/异步parse方法的选择：如果不需要在解析阶段操作Playwright页面，用同步方法即可；若需要异步操作，再保留async并确保配置支持

修改后的完整代码 #

import scrapy
from scrapy_playwright.page import PageMethod

class MySpider(scrapy.Spider):
    name = "Forum01"
    allowed_domains = ["example.com"]

    def start_requests(self):
        with open('FullLink.csv') as file:
            start_urls = [line.strip() for line in file]
        print(start_urls)
        
        # 遍历读取到的局部变量start_urls
        for u in start_urls:    
            yield scrapy.Request(
                u,
                meta=dict(
                    playwright=True,
                    playwright_include_page=False,
                    playwright_page_methods=[
                        PageMethod("wait_for_selector", "div.modal-body > p")
                    ],
                ),
                callback=self.parse
            )

    def parse(self, response):
        for item in response.css('div.modal-content'):
            yield{
                'title': item.css('h1::text').get(),
                'info': item.css('.row+ p::text').get(),
            }   

额外注意事项 #

• 确保Scrapy项目的settings.py中正确配置Scrapy-playwright：

# 启用Playwright下载处理器
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

# Playwright启动配置（按需调整）
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,
}

• 检查CSV中的URL是否在allowed_domains范围内，若涉及其他域名，修改allowed_domains或直接删除该限制
• 若需要在parse中操作Playwright页面，需将playwright_include_page设为True，并使用异步解析方法：

async def parse(self, response):
    page = response.meta["playwright_page"]
    # 执行异步页面操作
    await page.close()  # 记得关闭页面

内容的提问来源于stack exchange，提问作者ReCodeRa