Scrapy-playwright爬虫response.meta缺失page/playwright_page问题求助

从运行日志的Enabled downloader middlewares部分可以看到：Scrapy并未加载scrapy_playwright.middleware.PlaywrightMiddleware，请求仍由默认下载器处理，因此response.meta中不会包含Playwright的Page对象。下面给出两种解决方案：

修复现有Scrapy-Playwright代码 #

1. 补充核心配置项 #

在Spider的custom_settings中添加DOWNLOAD_HANDLERS，替换默认的HTTP/HTTPS下载器为Playwright实现，确保请求由Playwright处理：

custom_settings = {
    'CONCURRENT_REQUESTS': 1,
    'DOWNLOAD_DELAY': 3,
    'COOKIES_ENABLED': False,
    'PLAYWRIGHT_BROWSER_TYPE': 'chromium',
    # 新增：替换默认下载器
    'DOWNLOAD_HANDLERS': {
        "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    },
    'MIDDLEWARES': {
        'scrapy_playwright.middleware.PlaywrightMiddleware': 800,
    },
}

2. 确保Playwright浏览器已安装 #

执行命令安装Chromium：

playwright install chromium

3. 修正CrawlerProcess初始化（可选） #

如果自定义设置仍不生效，可将Spider配置直接传入CrawlerProcess：

def main(domain, stop, user_agent):
    settings = GoogleSpider.custom_settings.copy()
    settings['USER_AGENT'] = user_agent
    process = CrawlerProcess(settings=settings)
    process.crawl(GoogleSpider, domain=domain, stop=stop)
    process.start()

4. 处理异步兼容性 #

在执行文件开头添加代码，确保Scrapy使用支持异步的Reactor：

import asyncio
from twisted.internet.asyncioreactor import AsyncioSelectorReactor
AsyncioSelectorReactor(asyncio.get_event_loop())

纯Playwright替代方案（更简洁易调试） #

如果不想依赖Scrapy，直接用Playwright编写独立脚本，逻辑与原代码一致，且无需处理Scrapy的配置与中间件问题：

from playwright.async_api import async_playwright
import asyncio

async def crawl_google(domain, stop_count, user_agent):
    search_url = f'https://www.google.com/search?q=intitle%3A%28%22Data+Scientist%22+OR+%22Data+Engineer%22+OR+%22Machine+Learning%22+OR+%22Data+Analyst%22+OR+%22Software+Engineer%22%29+Remote+-%22Director%22+-%22Principal%22+-%22Staff%22+-%22Frontend%22+-%22Front+End%22+-%22Full+Stack%22+site%3A{domain}%2F%2A+after%3A2023-03-27'
    
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(user_agent=user_agent)
        page = await context.new_page()
        
        await page.goto(search_url, wait_until='networkidle')
        
        collected_urls = []
        current_length = 0
        
        while len(collected_urls) < stop_count:
            # 提取搜索结果链接
            urls = await page.locator('.yuRUbf>a').evaluate_all('nodes => nodes.map(n => n.href)')
            # 过滤目标域名且未收集的链接
            new_urls = [url for url in urls if domain in url and url not in collected_urls]
            collected_urls.extend(new_urls)
            
            # 达到停止数量则截断
            if len(collected_urls) >= stop_count:
                collected_urls = collected_urls[:stop_count]
                break
            
            # 滚动加载更多
            if len(urls) > current_length:
                current_length = len(urls)
                await page.evaluate('window.scrollTo(0, document.body.scrollHeight)')
                await page.wait_for_timeout(1500)
                await page.wait_for_selector('.yuRUbf>a', state='visible')
            else:
                # 无更多结果，退出循环
                break
        
        # 输出结果
        print(f'收集到 {len(collected_urls)} 个URL：')
        for url in collected_urls:
            print(url)
        
        await browser.close()

if __name__ == '__main__':
    domain = 'jobs.lever.co'
    stop = 25
    user_agent3 = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.2; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)"
    asyncio.run(crawl_google(domain, stop, user_agent3))

内容的提问来源于stack exchange，提问作者reesh19