使用Scrapy爬取MarketWatch遇302重定向问题求助

重定向的核心原因 #

• 缺失浏览器特征请求头：MarketWatch的反爬机制会校验请求是否来自真实浏览器，缺少关键头信息会被判定为爬虫，强制重定向到搜索页。
• 会话Cookie未建立：部分页面需要先通过访问首页/股票基础页获取会话Cookie，直接请求详情页会被拦截重定向。
• 爬虫行为特征触发拦截：请求频率过高、请求顺序不符合正常用户路径，也可能触发重定向逻辑。

具体解决方案 #

1. 配置真实浏览器级请求头 #

在Scrapy的settings.py中设置全局默认请求头，模拟Chrome浏览器的请求特征：

DEFAULT_REQUEST_HEADERS = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Accept-Encoding': 'gzip, deflate, br',
    'Referer': 'https://www.marketwatch.com/',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1'
}

若全局设置无效，可在单个Request中单独指定headers：

def start_requests(self):
    tickers = ['abbv']
    for ticker in tickers:
        url = f'https://www.marketwatch.com/investing/{ticker}/financials'
        yield scrapy.Request(
            url=url,
            callback=self.parse_financials,
            headers={
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
                'Referer': f'https://www.marketwatch.com/investing/{ticker}'
            }
        )

2. 先建立会话再请求详情页 #

先请求目标股票的基础页面（如https://www.marketwatch.com/investing/abbv）获取网站会话Cookie，再携带Cookie请求金融数据页：

def start_requests(self):
    tickers = ['abbv']
    for ticker in tickers:
        base_url = f'https://www.marketwatch.com/investing/{ticker}'
        yield scrapy.Request(
            url=base_url,
            callback=self.parse_base,
            meta={'ticker': ticker}
        )

def parse_base(self, response):
    ticker = response.meta['ticker']
    financials_url = f'https://www.marketwatch.com/investing/{ticker}/financials'
    # 复用基础页请求获取的Cookie
    yield scrapy.Request(
        url=financials_url,
        callback=self.parse_financials,
        cookies=response.cookies
    )

3. 调整Scrapy反爬相关设置 #

在settings.py中添加以下配置，降低被识别为爬虫的概率：

# 限制重定向次数，避免无限循环
REDIRECT_MAX_TIMES = 3
# 设置下载延迟，模拟用户浏览节奏
DOWNLOAD_DELAY = 2
# 启用自动限速扩展，动态调整请求间隔
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 5

4. 处理JavaScript动态渲染（若必要） #

如果上述方法均无效，可能页面内容是通过JS动态加载的，可结合Scrapy-Playwright渲染页面：

• 安装依赖：pip install scrapy-playwright
• 在settings.py中启用扩展：

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,
    "args": ["--no-sandbox"],
}

• 在Request中指定启用Playwright：

yield scrapy.Request(
    url=financials_url,
    callback=self.parse_financials,
    meta={'playwright': True}
)

内容的提问来源于stack exchange，提问作者Josep Cubedo Macian