如何解决Scrapy爬取book24.ru时出现的520错误？

安全阿岳

2026-8-3

Book24.ru Scrapy爬虫520拦截问题及解决尝试

问题背景

我编写的Scrapy爬虫一年前可正常抓取book24.ru内容，现在网站会拦截爬虫请求，返回520错误。

Scrapy Shell返回的520错误日志

2023-01-20 22:10:23 [scrapy.utils.log] INFO: Scrapy 2.7.1 started (bot: bestsellers)
2023-01-20 22:10:23 [scrapy.utils.log] INFO: Versions: lxml 4.9.2.0, libxml2 2.9.14, cssselect 1.2.0, parsel 1.7.0, w3lib 2.1.1, Twisted 22.10.0, Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0], pyOpenSSL 23.0.0 (OpenSSL 3.0.7 1 Nov 2022), cryptography 39.0.0, Platform Linux-5.15.0-58-generic-x86_64-with-glibc2.35
2023-01-20 22:10:23 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'bestsellers',
 'NEWSPIDER_MODULE': 'bestsellers.spiders',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'SPIDER_MODULES': ['bestsellers.spiders'],
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'}
2023-01-20 22:10:23 [asyncio] DEBUG: Using selector: EpollSelector
2023-01-20 22:10:23 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2023-01-20 22:10:23 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop
2023-01-20 22:10:23 [scrapy.extensions.telnet] INFO: Telnet Password: cb384b87b1e215df
2023-01-20 22:10:23 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2023-01-20 22:10:23 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2023-01-20 22:10:23 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2023-01-20 22:10:23 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2023-01-20 22:10:23 [scrapy.core.engine] INFO: Spider opened
2023-01-20 22:10:23 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-01-20 22:10:23 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2023-01-20 22:10:23 [filelock] DEBUG: Attempting to acquire lock 140382385887728 on /home/user/.cache/python-tldextract/3.10.6.final__first_scrapy-rekTiTXv__400173__tldextract-3.4.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-01-20 22:10:23 [filelock] DEBUG: Lock 140382385887728 acquired on /home/user/.cache/python-tldextract/3.10.6.final__first_scrapy-rekTiTXv__400173__tldextract-3.4.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-01-20 22:10:23 [filelock] DEBUG: Attempting to release lock 140382385887728 on /home/user/.cache/python-tldextract/3.10.6.final__first_scrapy-rekTiTXv__400173__tldextract-3.4.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-01-20 22:10:23 [filelock] DEBUG: Lock 140382385887728 released on /home/user/.cache/python-tldextract/3.10.6.final__first_scrapy-rekTiTXv__400173__tldextract-3.4.0/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-01-20 22:10:23 [scrapy.core.engine] DEBUG: Crawled (520) &lt;GET https://book24.ru/product/transhumanism-inc-6015821/&gt; (referer: None)
2023-01-20 22:10:23 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;520 https://book24.ru/product/transhumanism-inc-6015821/&gt;: HTTP status code is not handled or not allowed
2023-01-20 22:10:23 [scrapy.core.engine] INFO: Closing spider (finished)
2023-01-20 22:10:23 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 243,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 7640,
 'downloader/response_count': 1,
 'downloader/response_status_count/520': 1,
 'elapsed_time_seconds': 0.364996,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2023, 1, 20, 19, 10, 23, 800623),
 'httperror/response_ignored_count': 1,
 'httperror/response_ignored_status_count/520': 1,
 'log_count/DEBUG': 8,
 'log_count/INFO': 11,
 'memusage/max': 65613824,
 'memusage/startup': 65613824,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2023, 1, 20, 19, 10, 23, 435627)}
2023-01-20 22:10:23 [scrapy.core.engine] INFO: Spider closed (finished)

解决尝试

第一次尝试：添加请求头

尝试添加自定义headers伪装成浏览器请求，仍返回520错误。代码如下：

import scrapy

class BookSpider(scrapy.Spider):
    name = 'book24-2'
    start_urls = ['https://book24.ru/knigi-bestsellery/']
    
    def parse(self, response):
        headers = {
        'user-agent' : 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'
        }
        
        for link in response.css('div.product-card__image-holder a::attr(href)', headers=headers).get():
            yield response.follow(link, callback=self.parse_book, headers=headers)

        for i in range(1, 25):
            next_page = f'https://book24.ru/knigi-bestsellery/page-{i}/'
            yield response.follow(next_page, callback=self.parse, headers=headers)

    def parse_book(self, response):
        yield{
            'name': response.css('h1.product-detail-page__title::text').get(),
            'buy': response.css('p.product-detail-page__purchased-text::text').get().split()[1],
            'type': response.css('div.product-characteristic__value a::attr(title)')[2].get()
        }

第二次尝试：关闭ROBOTSTXT_OBEY

将settings.py中的ROBOTSTXT_OBEY = True改为False，520错误依旧存在。

第三次尝试：使用fake_useragent库

引入fake_useragent库随机生成User-Agent，问题未解决。代码如下：

import scrapy
from fake_useragent import UserAgent

class BookSpider(scrapy.Spider):
    name = 'book24-2'
    start_urls = ['https://book24.ru/knigi-bestsellery/']
    
    def parse(self, response):
        headers = {
        'user-agent' : UserAgent().random
        }
        
        for link in response.css('div.product-card__image-holder a::attr(href)', headers=headers).get():
            yield response.follow(link, callback=self.parse_book, headers=headers)

        for i in range(1, 25):
            next_page = f'https://book24.ru/knigi-bestsellery/page-{i}/'
            yield response.follow(next_page, callback=self.parse, headers=headers)

    def parse_book(self, response):
        headers = {
        'user-agent' : UserAgent().random
        }
        
        yield{
            'name': response.css('h1.product-detail-page__title::text', headers=headers).get(),
            'buy': response.css('p.product-detail-page__purchased-text::text', headers=headers).get().split()[1],
            'type': response.css('div.product-characteristic__value a::attr(title)', headers=headers)[2].get()
        }

内容的提问来源于stack exchange，提问作者4www44

如何解决Scrapy爬取book24.ru时出现的520错误？

问题背景 #

Scrapy Shell返回的520错误日志 #

解决尝试 #

第一次尝试：添加请求头 #

第二次尝试：关闭ROBOTSTXT_OBEY #