Scrapy爬取savings.gov.pk返回500错误，Requests可正常访问

我在使用Scrapy爬取目标页面时，无法获取响应，运行爬虫会收到HTTP 500内部服务器错误，但用Python的Requests模块能正常获取响应。

我的基础Spider代码 #

import scrapy

class SavingsGov(scrapy.Spider):
    name        = 'savings'
    start_urls  = [
        'https://savings.gov.pk/download-draws/'
    ]

    def parse(self, response):
        for option in response.css('select option'):
            yield {
                'url': option.css('::attr(value)').get()
            }

错误日志（已将settings.py重试次数设为10） #

2023-08-26 16:30:22 [scrapy.core.engine] INFO: Spider opened
2023-08-26 16:30:22 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-08-26 16:30:22 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2023-08-26 16:30:24 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/robots.txt> (failed 1 times): 500 Internal Server Error
2023-08-26 16:30:25 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/robots.txt> (failed 2 times): 500 Internal Server Error
2023-08-26 16:30:27 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/robots.txt> (failed 3 times): 500 Internal Server Error
2023-08-26 16:30:28 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/robots.txt> (failed 4 times): 500 Internal Server Error
2023-08-26 16:30:30 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/robots.txt> (failed 5 times): 500 Internal Server Error
2023-08-26 16:30:31 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/robots.txt> (failed 6 times): 500 Internal Server Error
2023-08-26 16:30:33 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/robots.txt> (failed 7 times): 500 Internal Server Error
2023-08-26 16:30:35 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/robots.txt> (failed 8 times): 500 Internal Server Error
2023-08-26 16:30:37 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/robots.txt> (failed 9 times): 500 Internal Server Error
2023-08-26 16:30:39 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/robots.txt> (failed 10 times): 500 Internal Server Error
2023-08-26 16:30:40 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://savings.gov.pk/robots.txt> (failed 11 times): 500 Internal Server Error
2023-08-26 16:30:40 [scrapy.core.engine] DEBUG: Crawled (500) <GET https://savings.gov.pk/robots.txt> (referer: None)
2023-08-26 16:30:40 [protego] DEBUG: Rule at line 1 without any user agent to enforce it on.
2023-08-26 16:30:41 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/download-draws/> (failed 1 times): 500 Internal Server Error
2023-08-26 16:30:43 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/download-draws/> (failed 2 times): 500 Internal Server Error
2023-08-26 16:30:44 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/download-draws/> (failed 3 times): 500 Internal Server Error
2023-08-26 16:30:46 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/download-draws/> (failed 4 times): 500 Internal Server Error
2023-08-26 16:30:47 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/download-draws/> (failed 5 times): 500 Internal Server Error
2023-08-26 16:30:49 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/download-draws/> (failed 6 times): 500 Internal Server Error
2023-08-26 16:30:50 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/download-draws/> (failed 7 times): 500 Internal Server Error
2023-08-26 16:30:52 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/download-draws/> (failed 8 times): 500 Internal Server Error
2023-08-26 16:30:53 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/download-draws/> (failed 9 times): 500 Internal Server Error
2023-08-26 16:30:55 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://savings.gov.pk/download-draws/> (failed 10 times): 500 Internal Server Error
2023-08-26 16:30:56 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://savings.gov.pk/download-draws/> (failed 11 times): 500 Internal Server Error
2023-08-26 16:30:56 [scrapy.core.engine] DEBUG: Crawled (500) <GET https://savings.gov.pk/download-draws/> (referer: None)
2023-08-26 16:30:56 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <500 https://savings.gov.pk/download-draws/>: HTTP status code is not handled or not allowed
2023-08-26 16:30:56 [scrapy.core.engine] INFO: Closing spider (finished)

Requests测试代码（可正常获取响应） #

import requests

response = requests.get('https://savings.gov.pk/download-draws/')
print(response.text)

我的疑问 #

不清楚问题所在，猜测可能是Scrapy.Request的问题。是否可以用Requests发送请求后将响应传给Scrapy？更优的方案是如何调试Scrapy.Request？作为Scrapy新手，若有对问题的误解，也请指出。

问题解决及调试方案 #

1. 核心原因：请求头差异 #

Scrapy默认请求头（尤其是User-Agent）和Requests不同，目标网站可能校验请求头，拒绝了Scrapy的默认请求。

2. 快速修复：自定义请求头 #

在settings.py中添加浏览器风格的User-Agent：

USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36'

也可以在Spider中手动指定请求头：

import scrapy

class SavingsGov(scrapy.Spider):
    name = 'savings'
    start_urls = ['https://savings.gov.pk/download-draws/']

    def start_requests(self):
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36'
        }
        for url in self.start_urls:
            yield scrapy.Request(url, headers=headers, callback=self.parse)

    def parse(self, response):
        for option in response.css('select option'):
            yield {
                'url': option.css('::attr(value)').get()
            }

3. 禁用Robots协议检查 #

日志显示Scrapy先请求robots.txt并返回500，虽然不影响最终请求，但可在settings.py中禁用：

ROBOTSTXT_OBEY = False

4. 调试Scrapy请求的方法 #

• 对比请求头：用scrapy shell查看默认请求头，和Requests的请求头对比：

  scrapy shell https://savings.gov.pk/download-draws/
  # 在shell中执行：
  from scrapy import Request
  req = Request(url)
  print(req.headers)
  

  再对比Requests的请求头：

  import requests
  req = requests.get(url)
  print(req.request.headers)
  

  找出差异后调整Scrapy的请求头。
• 启用详细日志：在settings.py中设置日志级别为DEBUG，查看请求的完整细节：

  LOG_LEVEL = 'DEBUG'
  

• 使用下载中间件：如果网站有反爬机制，可通过自定义下载中间件统一处理请求头、Cookies等。

5. 关于用Requests传递响应给Scrapy #

不推荐这种做法，会破坏Scrapy的异步架构和中间件机制。如果确实需要临时方案，可以构造Scrapy响应对象：

import scrapy
import requests
from scrapy.http import HtmlResponse

class SavingsGov(scrapy.Spider):
    name = 'savings'
    start_urls = ['https://savings.gov.pk/download-draws/']

    def start_requests(self):
        for url in self.start_urls:
            resp = requests.get(url)
            scrapy_resp = HtmlResponse(url=url, body=resp.text, encoding='utf-8')
            yield scrapy_resp

    def parse(self, response):
        # 正常处理响应
        pass

但此方式会失去Scrapy的重试、异步等特性，仅作为临时应急方案。

内容的提问来源于stack exchange，提问作者Hammad Ahmed