使用Scrapy与Scrapy-Playwright加载页面报错：需启用JavaScript

针对你遇到的WSJ反爬检测和超时问题，核心原因是网站识别出了自动化工具特征，同时你的等待逻辑没有适配页面的检测流程。以下是具体修复方案：

关键优化点 #

• 隐藏Playwright自动化标识：禁用浏览器的自动化检测特征，让浏览器更接近真实用户环境
• 模拟真实浏览器配置：设置合理的User-Agent、视口尺寸
• 优化页面等待逻辑：替换固定超时为等待网络空闲或关键元素加载，避免提前获取未完成渲染的页面
• 处理广告拦截检测：通过页面交互绕过检测提示

修改后的完整代码 #

import scrapy
from scrapy.selector import Selector
from scrapy_playwright.page import PageMethod


class WsjNewsJSSpider(scrapy.Spider):
    name = 'wsj_newsJS_BACKUP'
    start_urls = ['https://www.wsj.com']

    custom_settings = {
        "DOWNLOAD_HANDLERS": {
            'http': 'scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler',
            'https': 'scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler',
        },
        "TWISTED_REACTOR": 'twisted.internet.asyncioreactor.AsyncioSelectorReactor',
        "PLAYWRIGHT_BROWSER_TYPE": "chromium",
        "PLAYWRIGHT_LAUNCH_OPTIONS": {
            "headless": False,
            # 禁用自动化标识
            "args": [
                "--disable-blink-features=AutomationControlled",
                "--start-maximized"  # 最大化窗口，模拟真实用户
            ]
        },
        # 设置默认请求头，模拟真实浏览器
        "DEFAULT_REQUEST_HEADERS": {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
        }
    }

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={
                    'playwright': True,
                    "playwright_page_methods": [
                        # 设置视口尺寸
                        PageMethod("set_viewport_size", width=1920, height=1080),
                        # 等待网络空闲（500ms内没有新请求），确保页面加载完成
                        PageMethod("wait_for_load_state", state="networkidle"),
                        # 若出现广告拦截提示，尝试点击允许/关闭按钮（根据实际页面调整选择器）
                        PageMethod("click", "button:has-text('允许')", timeout=3000),
                    ],
                    # 增加请求超时时间
                    'download_timeout': 30,
                },
                callback=self.parse
            )

    def parse(self, response):
        sel = Selector(response)
        # 验证是否成功加载页面，比如获取标题
        page_title = sel.xpath('//title/text()').get()
        print(f"页面标题: {page_title}")
        print('页面加载成功')

额外注意事项 #

• 如果仍有检测，可尝试切换为Firefox或WebKit浏览器（修改PLAYWRIGHT_BROWSER_TYPE）
• 避免频繁请求，添加随机延迟（可通过DOWNLOAD_DELAY配置）
• 部分情况下可能需要启用Playwright的context.add_init_script来进一步隐藏自动化特征，比如：

  PageMethod("context.add_init_script", script="""
      Object.defineProperty(navigator, 'webdriver', {get: () => undefined})
  """)
  

内容的提问来源于stack exchange，提问作者jay queue