如何在Scrapy Python中异步运行Playwright？报错求助

同步版本的Scrapy+Playwright代码可正常解析所有起始URL并输出目标文本，但将代码改为异步执行（调用call_playwright()函数而非yield方式）时，出现RuntimeError: no running event loop错误。

同步代码 #

import scrapy
from scrapy.crawler import CrawlerProcess
from playwright.sync_api import sync_playwright


class ToscrapeSpider(scrapy.Spider):
    name = 'toscrape'
    allowed_domains = ['example.com']
    start_urls = [
        'https://example.com/?1',
        'https://example.com/?2',
        'https://example.com/?3',
    ]

    def parse(self, response):
        link = response.url
        self.call_playwright(link)

    
    def call_playwright(self, url):
        with sync_playwright() as playwright:
            browser = playwright.chromium.launch(headless=True)
            page = browser.new_page()
            page.goto(url)
            page.wait_for_selector('h1', timeout=5000)
            tag = page.locator('h1')
            print(tag.inner_text())
            browser.close()
        
        
if __name__ == "__main__":
    process = CrawlerProcess()
    process.crawl(ToscrapeSpider)
    process.start()

异步尝试代码 #

import scrapy
from scrapy.crawler import CrawlerProcess
from playwright.async_api import async_playwright


class ToscrapeSpider(scrapy.Spider):
    name = 'toscrape'
    allowed_domains = ['example.com']
    start_urls = [
        'https://example.com/?1',
        'https://example.com/?2',
        'https://example.com/?3',
    ]


    async def parse(self, response):
        link = response.url
        await self.call_playwright(link)

    
    async def call_playwright(self, url):
        async with async_playwright() as playwright:
            browser = await playwright.chromium.launch(headless=True)
            page = await browser.new_page()
            await page.goto(url)
            await page.wait_for_selector('h1', timeout=5000)
            tag = page.locator('h1')
            print(tag.inner_text())
            await browser.close()
        
        
if __name__ == "__main__":
    process = CrawlerProcess()
    process.crawl(ToscrapeSpider)
    process.start()

错误信息 #

Traceback (most recent call last):
  File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\internet\defer.py", line 1697, in _inlineCallbacks   
    result = context.run(gen.send, result)
  File "h:\spiders\getprices.py", line 21, in parse
    await self.call_playwright(link)
  File "h:\spiders\getprices.py", line 25, in call_playwright
    async with async_playwright() as playwright:
  File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\site-packages\playwright\async_api\_context_manager.py", line 31, in __aenter__
    loop = asyncio.get_running_loop()
RuntimeError: no running event loop

环境：Python 3.10.7、Scrapy 2.8.0、Playwright 1.34.0

错误根源是Scrapy基于Twisted框架，使用Twisted事件循环，而Playwright异步API依赖asyncio事件循环，两者默认不兼容。需用Scrapy提供的工具将asyncio协程适配到Twisted环境中，具体修改如下：

修改后的完整代码 #

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.utils.defer import async_to_deferred
from playwright.async_api import async_playwright


class ToscrapeSpider(scrapy.Spider):
    name = 'toscrape'
    allowed_domains = ['example.com']
    start_urls = [
        'https://example.com/?1',
        'https://example.com/?2',
        'https://example.com/?3',
    ]

    def parse(self, response):
        link = response.url
        # 用async_to_deferred将异步协程转为Twisted可处理的Deferred对象
        return async_to_deferred(self.call_playwright)(link)

    async def call_playwright(self, url):
        async with async_playwright() as playwright:
            browser = await playwright.chromium.launch(headless=True)
            page = await browser.new_page()
            await page.goto(url)
            await page.wait_for_selector('h1', timeout=5000)
            tag = page.locator('h1')
            # inner_text是异步方法，必须添加await
            text = await tag.inner_text()
            print(text)
            await browser.close()


if __name__ == "__main__":
    process = CrawlerProcess()
    process.crawl(ToscrapeSpider)
    process.start()

关键改动说明 #

1. 导入async_to_deferred：这是Scrapy官方提供的工具，负责将asyncio协程转换为Twisted的Deferred对象，自动处理两个事件循环的适配，解决no running event loop问题。
2. 修改parse方法：不再直接用await调用异步方法，而是返回async_to_deferred包装后的结果，让Scrapy的Twisted引擎能正确调度异步任务。
3. 修复inner_text()调用：原代码中遗漏了await，tag.inner_text()是异步方法，必须加await才能获取实际文本内容。

内容的提问来源于stack exchange，提问作者Space Pilot 3000