Scrapy爬取Yelp API遭遇brotli解压错误求助

问题描述 #

使用Scrapy抓取Yelp意大利站餐厅四星评价时，通过餐厅页面的yelp-biz-id构造API请求链接，代码如下：

bizId = response.xpath("//meta[@name='yelp-biz-id']/@content").extract_first()
api_url = 'https://www.yelp.it/biz/' + bizId + '/review_feed?rr=' + str(n_star_filter)
yield response.follow(url=api_url, callback = self.parse_yelp_restaurant_api)

多数请求会触发Brotli解压错误，报错信息如下：

2023-10-27 15:57:39 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.yelp.it/biz/78t73jTxdUw5C-v44lj4Iw/review_feed?rr=4>
Traceback (most recent call last):
  File "/Users/mauri/anaconda3/lib/python3.11/site-packages/twisted/internet/defer.py", line 1697, in _inlineCallbacks
    result = context.run(gen.send, result)
  File "/Users/mauri/anaconda3/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 64, in process_response
    method(request=request, response=response, spider=spider)
  File "/Users/mauri/anaconda3/lib/python3.11/site-packages/scrapy/downloadermiddlewares/httpcompression.py", line 63, in process_response
    decoded_body = self._decode(response.body, encoding.lower())
  File "/Users/mauri/anaconda3/lib/python3.11/site-packages/scrapy/downloadermiddlewares/httpcompression.py", line 102, in _decode
    body = brotli.decompress(body)
  File "/Users/mauri/anaconda3/lib/python3.11/site-packages/brotli/brotli.py", line 90, in decompress
    d.finish()
  File "/Users/mauri/anaconda3/lib/python3.11/site-packages/brotli/brotli.py", line 464, in finish
    raise Error("Decompression error: incomplete compressed stream.")
brotli.brotli.Error: Decompression error: incomplete compressed stream.

错误本质 #

这个错误核心是服务器返回的Brotli压缩数据流不完整，常见触发场景：

• Yelp反爬机制：高频请求、无合法请求头的请求会被返回截断响应，以此识别并拦截爬虫。
• 网络波动：请求过程中数据包丢失，导致接收的压缩数据不完整。
• Scrapy压缩中间件兼容性：默认HttpCompressionMiddleware处理某些异常压缩响应时，无法正确识别数据流完整性。

解决方案 #

1. 完善请求头，模拟浏览器行为 #

Yelp会校验请求头合法性，添加完整字段可降低拦截概率：

def start_requests(self):
    headers = {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
        'Accept-Encoding': 'gzip, deflate, br',
        'Accept-Language': 'it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3',
        'Referer': 'https://www.yelp.it/'
    }
    yield scrapy.Request(url=your_restaurant_url, headers=headers, callback=self.parse)

def parse(self, response):
    bizId = response.xpath("//meta[@name='yelp-biz-id']/@content").extract_first()
    api_url = 'https://www.yelp.it/biz/' + bizId + '/review_feed?rr=' + str(n_star_filter)
    # 继承原请求的headers，保证请求一致性
    yield response.follow(url=api_url, callback=self.parse_yelp_restaurant_api, headers=response.request.headers)

2. 禁用Brotli，改用gzip/deflate压缩 #

如果Brotli问题频繁，可强制服务器返回其他压缩格式：

• 修改请求头的Accept-Encoding字段，移除br：

headers['Accept-Encoding'] = 'gzip, deflate'

• 或直接禁用Scrapy的HttpCompressionMiddleware（不推荐，会增加流量消耗），在settings.py中添加：

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': None,
}

3. 添加重试机制，处理不完整响应 #

针对解压失败的请求，配置自动重试：
在settings.py中设置：

RETRY_ENABLED = True
RETRY_TIMES = 3
# 把200加入重试码，因为错误发生在200响应的解压阶段
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429, 200]

也可自定义重试中间件，专门捕获Brotli错误：

from scrapy.downloadermiddlewares.retry import RetryMiddleware
import brotli

class BrotliRetryMiddleware(RetryMiddleware):
    def process_exception(self, request, exception, spider):
        if isinstance(exception, brotli.brotli.Error):
            reason = "Brotli decompression failed"
            return self._retry(request, reason, spider) or super().process_exception(request, exception, spider)

然后在settings.py启用：

DOWNLOADER_MIDDLEWARES = {
    'your_project_name.middlewares.BrotliRetryMiddleware': 550,
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
}

4. 控制请求频率，规避反爬 #

Yelp对请求频率敏感，添加延迟和随机间隔：
在settings.py中配置：

DOWNLOAD_DELAY = 2
RANDOMIZE_DOWNLOAD_DELAY = True

注意事项 #

• Yelp网页端API有使用限制，频繁抓取可能导致IP封禁，建议使用代理IP池分散请求来源。
• 需严格遵守Yelp服务条款，避免过度抓取引发法律风险。

内容的提问来源于stack exchange，提问作者Rodolfo