Scrapy无法获取页面但curl/浏览器可访问的问题求助

问题重现 #

用户编写的Scrapy爬虫代码如下：

import scrapy
from scrapy.crawler import CrawlerProcess

class ScraperSpider(scrapy.Spider):
    name = "scraper"

    def start_requests(self):
        urls = [
            'https://api.ipify.org?format=json',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        self.logger.info('================Request: %s, IP address: %s' % (response.request, response.text))

if __name__ == "__main__":
    process = CrawlerProcess()
    process.crawl(ScraperSpider)
    process.start()

运行时返回400错误：

2023-12-18 23:56:34 [scrapy.core.engine] DEBUG: Crawled (400) <GET https://api.ipify.org?format=json> (referer: None)
2023-12-18 23:56:34 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <400 https://api.ipify.org?format=json>: HTTP status code is not handled or not allowed
2023-12-18 23:56:34 [scrapy.core.engine] INFO: Closing spider (finished)

但该URL可通过curl或浏览器正常访问。

问题原因 #

api.ipify.org会拦截Scrapy默认的User-Agent（格式类似Scrapy/2.x.x (+https://scrapy.org)），这类标识容易被网站判定为爬虫请求，从而返回400错误。

解决方案 #

通过设置自定义User-Agent模拟浏览器请求即可解决，有两种实现方式：

方式1：为单个请求添加headers

修改start_requests方法中的请求，添加浏览器的User-Agent：

def start_requests(self):
    urls = [
        'https://api.ipify.org?format=json',
    ]
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36'
    }
    for url in urls:
        yield scrapy.Request(url=url, callback=self.parse, headers=headers)

方式2：全局设置User-Agent

在CrawlerProcess初始化时配置全局User-Agent，所有请求都会使用该标识：

if __name__ == "__main__":
    process = CrawlerProcess(settings={
        'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36'
    })
    process.crawl(ScraperSpider)
    process.start()

验证效果 #

修改后运行爬虫，即可正常获取IP信息，日志会输出类似内容：

2023-12-19 00:00:00 [scraper] INFO: ================Request: <GET https://api.ipify.org?format=json>, IP address: {"ip":"xxx.xxx.xxx.xxx"}

内容的提问来源于stack exchange，提问作者Charles Ju