Scrapy中移除Connection: close请求头遇阻，自定义请求报错求助

问题分析与解决方案 #

首先，你遇到的报错核心原因是**start_requests方法需要生成scrapy.Request对象，而你返回的是requests.get()的Response对象**。Scrapy的调度器、中间件都是围绕自身的Request/Response模型工作的，所以会抛出AttributeError——因为requests库的Response没有Scrapy Request的meta、dont_filter这些属性。

你完全不需要用requests替代Scrapy的Request，通过以下两种方式就能移除Connection: close请求头：

方案1：子类化scrapy.Request移除指定请求头 #

如果只需要给特定爬虫或请求移除Connection: close，可以自定义Request类来处理：

import scrapy

class NoCloseConnectionRequest(scrapy.Request):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        # 移除默认添加的Connection: close
        if 'Connection' in self.headers:
            del self.headers['Connection']

class AdidasSpider(scrapy.Spider):
    name = "adidas"
    def start_requests(self):
        url = 'http://www.adidas.com/us/men-shoes'
        headers = {
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
            "Accept-Encoding": "gzip, deflate",
            "Accept-Language": "en-US,en;q=0.9",
            "Host": "www.adidas.com",
            "Upgrade-Insecure-Requests": "1",
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
        }
        # 使用自定义Request类
        yield NoCloseConnectionRequest(url, headers=headers, callback=self.parse)

    def parse(self, response):
        # 验证请求头是否已移除Connection: close
        print("当前请求头:", response.request.headers)
        # 后续正常处理响应即可
        pass

方案2：用下载中间件全局移除请求头 #

如果希望所有爬虫的请求都去掉Connection: close，更推荐用下载中间件，配置一次全局生效：

1. 在项目的middlewares.py中添加中间件代码：

class RemoveCloseConnectionMiddleware:
    def process_request(self, request, spider):
        # 移除Connection: close请求头
        if 'Connection' in request.headers:
            del request.headers['Connection']
        return None

2. 在settings.py中启用该中间件（注意优先级数字要合理，确保在默认的头处理中间件之后执行）：

DOWNLOADER_MIDDLEWARES = {
    'your_project_name.middlewares.RemoveCloseConnectionMiddleware': 543,
    # 保留其他默认中间件配置，比如Scrapy的UserAgentMiddleware等
}

3. 你的爬虫就可以用普通scrapy.Request了：

import scrapy

class AdidasSpider(scrapy.Spider):
    name = "adidas"
    def start_requests(self):
        url = 'http://www.adidas.com/us/men-shoes'
        headers = {
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
            "Accept-Encoding": "gzip, deflate",
            "Accept-Language": "en-US,en;q=0.9",
            "Host": "www.adidas.com",
            "Upgrade-Insecure-Requests": "1",
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
        }
        yield scrapy.Request(url, headers=headers, callback=self.parse)

    def parse(self, response):
        print("当前请求头:", response.request.headers)
        pass

额外提示 #

不要用requests库替代Scrapy的Request，这会完全丢掉Scrapy的异步处理、自动重试、调度队列、去重等核心优势，反而增加开发复杂度。

内容的提问来源于stack exchange，提问作者Biswajit Chopdar