在Windows系统PyCharm中运行Scrapy-Playwright时遇NotImplementedError

报错信息 #

Traceback (most recent call last):
File "c:\users\brainbox\appdata\local\programs\python\python38-32\lib\site-packages\playwright\_impl\_connection.py", line 166, in run
await self._transport.run()
File "c:\users\brainbox\appdata\local\programs\python\python38-32\lib\site-packages\playwright\_impl\_transport.py", line 60, in run
proc = await asyncio.create_subprocess_exec(
File "c:\users\brainbox\appdata\local\programs\python\python38-32\lib\asyncio\subprocess.py", line 236, in create_subprocess_exec
transport, protocol = await loop.subprocess_exec(
File "c:\users\brainbox\appdata\local\programs\python\python38-32\lib\asyncio\base_events.py", line 1630, in subprocess_exec
transport = await self._make_subprocess_transport(
File "c:\users\brainbox\appdata\local\programs\python\python38-32\lib\asyncio\base_events.py", line 491, in _make_subprocess_transport
raise NotImplementedError
NotImplementedError
2023-05-13 09:47:41 [asyncio] ERROR: Task exception was never retrieved
future: <Task finished name='Task-4' coro=<Connection.run() done, defined at c:\users\brainbox\appdata\local\programs\python\python38-32\lib\site-packages\playwright\_impl\_connection.py:16
3> exception=NotImplementedError()>
Traceback (most recent call last):
File "c:\users\brainbox\appdata\local\programs\python\python38-32\lib\site-packages\playwright\_impl\_connection.py", line 166, in run
await self._transport.run()
File "c:\users\brainbox\appdata\local\programs\python\python38-32\lib\site-packages\playwright\_impl\_transport.py", line 60, in run
proc = await asyncio.create_subprocess_exec(
File "c:\users\brainbox\appdata\local\programs\python\python38-32\lib\asyncio\subprocess.py", line 236, in create_subprocess_exec
transport, protocol = await loop.subprocess_exec(
File "c:\users\brainbox\appdata\local\programs\python\python38-32\lib\asyncio\base_events.py", line 1630, in subprocess_exec
transport = await self._make_subprocess_transport(
File "c:\users\brainbox\appdata\local\programs\python\python38-32\lib\asyncio\base_events.py", line 491, in _make_subprocess_transport
raise NotImplementedError
NotImplementedError
2023-05-13 09:48:41

爬虫代码 #

import scrapy
from quotes_js_scraper.items import QuoteItem


class QuotesSpider(scrapy.Spider):
    name = 'quotes'

    def start_requests(self):
        url = "https://quotes.toscrape.com/js/"
        yield scrapy.Request(url, meta={'playwright': True})

    def parse(self, response):
            yield{
                response.text
            }

配置文件（settings.py） #

# Scrapy settings for quotes_js_scraper project

BOT_NAME = 'quotes_js_scraper'

SPIDER_MODULES = ['quotes_js_scraper.spiders']
NEWSPIDER_MODULE = 'quotes_js_scraper.spiders'
# Obey robots.txt rules
ROBOTSTXT_OBEY = True

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
PLAQYWRIGHT_LAUNCH_OPTIONS = {
    "headless": False
}

解决步骤 #

1. 修复配置拼写错误 #

配置中PLAQYWRIGHT_LAUNCH_OPTIONS存在拼写错误，正确应为PLAYWRIGHT_LAUNCH_OPTIONS，修正后才能让Playwright读取启动参数。

2. 切换Windows兼容的Asyncio事件循环 #

Windows下Python 3.8默认的SelectorEventLoop不支持子进程操作，而Playwright需要启动浏览器子进程，因此必须切换到WindowsProactorEventLoopPolicy。

在settings.py开头添加以下代码：

import asyncio
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())

3. 更换为64位Python环境 #

你当前使用的是32位Python 3.8，Playwright对64位环境支持更稳定，建议卸载32位版本，安装64位Python 3.8及以上版本。

4. 重装Playwright浏览器驱动 #

若浏览器驱动损坏，执行以下命令重装：

playwright install

5. 修正爬虫代码的返回格式 #

parse方法中直接返回response.text不符合Scrapy的Item格式要求，改为键值对形式：

def parse(self, response):
    yield {
        'html_content': response.text
    }

内容的提问来源于stack exchange，提问作者Daniel