Scrapy爬取Lowe's产品页面返回400响应，添加请求头仍未解决

问题说明 #

• 目标爬取页面：https://www.lowes.com/pd/ZLINE-KITCHEN-BATH-Professional-7-Burners-4-cu-ft-2-cu-ft-Double-Oven-Convection-Dual-Fuel-Range-Stainless-Steel-Common-48-in-Actual-48-in/1000514227
• 原有Scrapy爬虫之前运行正常，近期突然失效，推测网站新增了反爬机制
• 尝试添加从开发者工具提取的请求头后，运行爬虫返回400错误响应

现有爬虫代码 #

# -*- coding: utf-8 -*-
import scrapy
from ..items import LowesItem
import re
import pandas as pd
import requests
import json
from scrapy.http import Request
from datetime import date


class LowesSpider(scrapy.Spider):
    name = 'Lowes'

    def start_requests(self):
        HEADERS = {
            'method': 'GET',
            'scheme': 'https',
            'authority': 'content.syndigo.com',
            'Accept': '*/*',
            'Content-Type': 'text/plain',
            'Origin': 'https://lowes.com',
            'Accept-Language': 'en-US,en;q=0.9',
            'Host': 'content.syndigo.com',
            'User-Agent': ' Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15',
            'Referer': 'https://www.lowes.com/',
            'Accept-Encoding': 'gzip, deflate, br',
            'Connection': 'keep-alive',
            'Cookie': 'sn=0321'
        }

        start_urls = ['https://www.lowes.com/pd/ZLINE-KITCHEN-BATH-Professional-7-Burners-4-cu-ft-2-cu-ft-Double-Oven-Convection-Dual-Fuel-Range-Stainless-Steel-Common-48-in-Actual-48-in/1000514227']

        for url in start_urls:
            yield Request(url,
                        headers=HEADERS,
                        meta={'dont_merge_cookies': True,
                          'url':url}) 

    def parse(self, response):
        for item in self.parseLowes(response):
            yield item
        pass

    def parseLowes(self, response):
        item = LowesItem() #items from items.py
        
        script_tag = response.xpath('//script[@type=\"application/ld+json\"]/text()').get() #get js container
        productPrice = json.loads(script_tag)[2][\"offers\"][\"price\"]

        productURL = response.url
        url = response.meta['url']
        productSKU = url.split(\"=\")[-1]

        scrapedDate = date.today()

        #item['productName'] = productName #display product name
        item['productOMS'] = productSKU
        item['productPrice'] = productPrice #display price and assign to variable
        item['productURL'] = productURL #displayURL
        item['scrapedDate'] = scrapedDate
        yield item

问题根源 #

你当前使用的请求头是针对content.syndigo.com域名的，但实际请求的是lowes.com的页面，请求头里的Host、Origin等核心字段和目标站点不匹配，直接触发了400错误。

修复方案 #

1. 修正请求头 #

删掉多余字段，调整为匹配lowes.com的请求头：

HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.9',
    'Host': 'www.lowes.com',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15',
    'Referer': 'https://www.lowes.com/',
    'Accept-Encoding': 'gzip, deflate, br',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1'
}

• 移除method、scheme、authority：Scrapy会自动处理这些属性，手动设置反而可能出错
• 修正Host为www.lowes.com，匹配目标域名
• 删除Content-Type：请求HTML页面不需要该字段
• 去掉User-Agent开头的多余空格
• 添加Upgrade-Insecure-Requests模拟浏览器默认行为

2. 调整Cookie设置 #

删掉meta里的dont_merge_cookies: True，让Scrapy自动管理Cookie，避免手动设置的Cookie过期失效。

3. 额外优化 #

• 在settings.py中设置DOWNLOAD_DELAY = 2，降低请求频率，减少被封禁的风险
• 可以使用scrapy-user-agents库随机切换User-Agent，进一步模拟真实用户请求

内容的提问来源于stack exchange，提问作者chrisHG