使用scrapy-playwright爬取网站时遇TypeError错误求助

问题场景 #

跟随教程进行Magicbricks网站爬取，代码如下：

import scrapy
from scrapy_playwright.page import PageMethod

class magicbrick_spider(scrapy.Spider):
    name = 'spy'
    
    async def start_requests(self):
        yield scrapy.Request(
            url = 'https://www.magicbricks.com/property-for-sale/residential-real-estate?bedroom=&proptype=Multistorey-Apartment,Builder-Floor-Apartment,Penthouse,Studio-Apartment,Residential-House,Villa&cityName=Mumbai',
            meta = {
                "playwright" : True,
                "playwright_include_page" : True,
                "playwright_page_methods" : [
                    PageMethod("wait_for_selector", ".mb-srp__card__container:nth-child(10)")
                ]
            },
            errback=self.close_page
        )

    async def parse(self, response):
        page = response.meta['playwright_page']
        for i in range(2, 11):
            containers = 30 * i
            await page.wait_for_selector(f".mb-srp__card__container:nth-child({containers})")
            await page.evaluate("window.scrollBy(0,document.body.scrollHeight)")
        html = await page.content()
        await page.close()

        title = html.css(".mb-srp__card--title::text").get()
        yield {"title" : title}

    async def close_page(self, failure):
        page = failure.request.meta['playwright_page']
        await page.close()

运行后触发错误：

2022-12-30 12:37:28 [scrapy.middleware] INFO: Enabled item pipelines:
[]
Unhandled error in Deferred:
2022-12-30 12:37:28 [twisted] CRITICAL: Unhandled error in Deferred:

Traceback (most recent call last):
  File "C:\Users\amanr\miniconda3\envs\Scraping\lib\site-packages\scrapy\crawler.py", line 205, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "C:\Users\amanr\miniconda3\envs\Scraping\lib\site-packages\scrapy\crawler.py", line 209, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "C:\Users\amanr\miniconda3\envs\Scraping\lib\site-packages\twisted\internet\defer.py", line 1905, in unwindGenerator    
    return _cancellableInlineCallbacks(gen)
  File "C:\Users\amanr\miniconda3\envs\Scraping\lib\site-packages\twisted\internet\defer.py", line 1815, in _cancellableInlineCallbacks
    _inlineCallbacks(None, gen, status)
--- <exception caught here> ---
  File "C:\Users\amanr\miniconda3\envs\Scraping\lib\site-packages\twisted\internet\defer.py", line 1660, in _inlineCallbacks   
    result = current_context.run(gen.send, result)
  File "C:\Users\amanr\miniconda3\envs\Scraping\lib\site-packages\scrapy\crawler.py", line 102, in crawl
    start_requests = iter(self.spider.start_requests())
builtins.TypeError: 'async_generator' object is not iterable

2022-12-30 12:37:28 [twisted] CRITICAL:
Traceback (most recent call last):
  File "C:\Users\amanr\miniconda3\envs\Scraping\lib\site-packages\twisted\internet\defer.py", line 1660, in _inlineCallbacks   
    result = current_context.run(gen.send, result)
  File "C:\Users\amanr\miniconda3\envs\Scraping\lib\site-packages\scrapy\crawler.py", line 102, in crawl
    start_requests = iter(self.spider.start_requests())
TypeError: 'async_generator' object is not iterable

补充说明：scrapy-playwright 对 Windows 缺乏原生支持，存在相关使用限制。

解决方案 #

1. 核心错误修复：移除start_requests的async修饰符 #

Scrapy的start_requests方法要求是普通生成器，不能定义为异步生成器（async def），这是触发报错的直接原因。

2. 修复HTML解析逻辑 #

page.content()返回的是字符串，无法直接调用Scrapy的.css()方法，需要将其转换为Scrapy的Selector对象。

3. 优化滚动加载逻辑（可选） #

原代码硬算容器数量不够灵活，可改为滚动直到页面不再加载新内容。

修改后的完整代码：

import scrapy
from scrapy_playwright.page import PageMethod
from scrapy.selector import Selector

class magicbrick_spider(scrapy.Spider):
    name = 'spy'
    
    # 移除async修饰符，改为普通生成器
    def start_requests(self):
        yield scrapy.Request(
            url = 'https://www.magicbricks.com/property-for-sale/residential-real-estate?bedroom=&proptype=Multistorey-Apartment,Builder-Floor-Apartment,Penthouse,Studio-Apartment,Residential-House,Villa&cityName=Mumbai',
            meta = {
                "playwright" : True,
                "playwright_include_page" : True,
                "playwright_page_methods" : [
                    PageMethod("wait_for_selector", ".mb-srp__card__container:nth-child(10)")
                ]
            },
            errback=self.close_page
        )

    async def parse(self, response):
        page = response.meta['playwright_page']
        
        # 优化滚动：循环滚动直到底部无新内容
        last_height = await page.evaluate("document.body.scrollHeight")
        while True:
            await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
            await page.wait_for_timeout(2000)  # 等待加载
            new_height = await page.evaluate("document.body.scrollHeight")
            if new_height == last_height:
                break
            last_height = new_height
        
        html = await page.content()
        await page.close()

        # 将字符串转换为Selector对象再解析
        selector = Selector(text=html)
        # 爬取所有标题，而非单个
        for title in selector.css(".mb-srp__card--title::text").getall():
            yield {"title": title.strip()}

    async def close_page(self, failure):
        page = failure.request.meta['playwright_page']
        await page.close()

额外注意事项 #

由于scrapy-playwright在Windows上无原生支持，需确保已按照官方文档配置WSL或其他兼容环境，避免后续出现运行时问题。

内容的提问来源于stack exchange，提问作者Aman Rao