使用scrapy-playwright爬取Coursera页面显示爬取0页，求排查问题

以下是导致‘Crawled 0 pages’的常见原因及修复方案：

1. 未正确配置Scrapy-Playwright中间件 #

这是最常见的问题——如果settings.py里没启用Playwright相关的下载处理器和中间件，Scrapy会继续用默认下载器处理请求，无法渲染JS页面。

在settings.py中添加以下配置：

# 启用Playwright下载处理器
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

# 启用Playwright中间件
SPIDER_MIDDLEWARES = {
    "scrapy_playwright.middleware.PlaywrightMiddleware": 543,
}

# 可选：配置Playwright启动参数，解决无头模式可能的权限问题
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": False,  # 先关闭无头模式，直观检查页面加载情况
    "args": ["--no-sandbox", "--disable-dev-shm-usage"],
}

2. 等待选择器错误或无效 #

你代码中等待的ul.cds-71可能不存在、class名动态变化，或者页面加载逻辑导致该元素迟迟不出现。

修复方案： #

• 替换为更稳定的选择器：比如Coursera搜索结果的容器通常是div[data-testid="search-results"]
• 改用等待页面网络空闲状态，确保所有资源加载完成：

meta={
    "playwright": True,
    "playwright_page_methods": [
        PageMethod("wait_for_load_state", "networkidle"),  # 等待网络空闲（无请求持续500ms）
        PageMethod("wait_for_selector", "div[data-testid='search-results']"),  # 再等待结果容器出现
    ],
}

3. 被Coursera反爬机制拦截 #

Coursera会检测自动化工具，未模拟真实用户环境的请求可能被拦截或返回空页面。

修复方案： #

在请求meta中添加浏览器视口和真实User-Agent：

meta={
    "playwright": True,
    "playwright_page_methods": [
        PageMethod("wait_for_load_state", "networkidle"),
        PageMethod("wait_for_selector", "div[data-testid='search-results']"),
    ],
    "playwright_context_kwargs": {
        "viewport": {"width": 1920, "height": 1080},  # 模拟桌面端视口
        "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
    },
}

4. 排查请求状态与日志 #

在parse函数中添加状态打印，确认请求是否成功：

def parse(self, response):
    self.logger.info(f"Response status: {response.status}")
    yield{'text':response.text}

同时在settings.py中开启Playwright日志，查看页面加载细节：

PLAYWRIGHT_LOG_LEVEL = "DEBUG"

内容的提问来源于stack exchange，提问作者Amen Aziz