Scraping Yelp遇503 Service Unavailable及后续防封禁咨询

问题描述 #

使用Scrapy抓取Yelp平台的餐厅及热门菜品信息，初期抓取正常，后续出现503 Service Unavailable错误，无法访问网站。已尝试以下操作但未解决问题：

• 设置自定义User Agent
• 将ROBOTSTXT_OBEY设为False
• 使用fake-useragent包轮换User Agent
  已向Yelp团队发送邮件申请访问权限，但担心后续大量抓取时再次被封禁，且当前User Agent轮换策略无效。

附当前配置及代码片段：

settings.py #

from fake_useragent import UserAgent
ua = UserAgent()
BOT_NAME = "yelp"

SPIDER_MODULES = ["yelp.spiders"]
NEWSPIDER_MODULE = "yelp.spiders"

# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
#USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"
USER_AGENT = ua.random
# Obey robots.txt rules
ROBOTSTXT_OBEY = False

Spider核心代码 #

def parse(self, response):
        pagination_text = response.css('.css-1aq64zd .css-chan6m::text').get()
        if pagination_text:
            total_pages = int(pagination_text.split(' of ')[-1])

            for page_number in range(total_pages):
                start_value = page_number * 10
                #next_page_url = f"https://www.yelp.com/search?find_desc=&find_loc=New+York%2C+NY&start={start_value}"
                #next_page_url = f"https://www.yelp.com/search?find_desc=Restaurants&find_loc=San+Francisco%2C+CA&start={start_value}"
                next_page_url = response.url + "&start={}".format(start_value)
                yield response.follow(next_page_url, callback=self.parse_page, headers={"User-Agent":ua.random})

核心问题分析 #

1. User Agent轮换未真正生效：当前settings.py中USER_AGENT = ua.random仅在爬虫启动时生成一次，所有请求共用同一个UA，并未实现每次请求轮换。
2. IP被精准封禁：Yelp反爬机制不仅识别UA，更会针对高频请求的IP直接拦截，单纯更换UA无法绕过IP封禁。
3. 请求模式过于规律：分页请求按固定顺序、固定间隔发送，极易被识别为爬虫行为。

可行解决方案 #

1. 实现真正的UA动态轮换 #

在Scrapy下载中间件中每次请求动态生成UA，替代settings中的一次性设置：

• 在middlewares.py中添加自定义中间件：

from fake_useragent import UserAgent

class RandomUserAgentMiddleware:
    def __init__(self):
        self.ua = UserAgent()

    def process_request(self, request, spider):
        request.headers['User-Agent'] = self.ua.random

• 在settings.py中启用该中间件并禁用默认UA中间件：

DOWNLOADER_MIDDLEWARES = {
    'yelp.middlewares.RandomUserAgentMiddleware': 400,
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
}

同时删除settings.py中USER_AGENT = ua.random这一行，避免冲突。

2. 启用IP代理轮换 #

Yelp对IP的检测极为严格，必须通过代理池分散请求来源：

• 使用scrapy-rotating-proxies库，安装后在settings.py中配置：

DOWNLOADER_MIDDLEWARES = {
    'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
    'rotating_proxies.middlewares.BanDetectionMiddleware': 620,
    'yelp.middlewares.RandomUserAgentMiddleware': 400,
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
}

ROTATING_PROXIES_LIST = [
    'http://proxy1:port',
    'http://proxy2:port',
    # 建议使用付费代理池保证可用性，免费代理稳定性极差
]

3. 严格控制请求频率 #

• 在settings.py中添加延迟配置：

DOWNLOAD_DELAY = 3  # 基础延迟3秒
RANDOMIZE_DOWNLOAD_DELAY = True  # 随机在0.5-1.5倍基础延迟间波动
CONCURRENT_REQUESTS = 2  # 降低并发请求数

4. 完善请求头信息 #

模拟真实浏览器的完整请求头，补充UA外的关键字段：
修改UA中间件的process_request方法：

def process_request(self, request, spider):
    request.headers['User-Agent'] = self.ua.random
    request.headers['Accept'] = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
    request.headers['Accept-Language'] = 'en-US,en;q=0.5'
    request.headers['Referer'] = 'https://www.yelp.com/'

5. 打乱请求顺序 #

避免按页码顺序爬取，随机打乱分页URL的请求顺序：

def parse(self, response):
    pagination_text = response.css('.css-1aq64zd .css-chan6m::text').get()
    if pagination_text:
        total_pages = int(pagination_text.split(' of ')[-1])
        page_urls = []
        for page_number in range(total_pages):
            start_value = page_number * 10
            next_page_url = response.url + "&start={}".format(start_value)
            page_urls.append(next_page_url)
        # 随机打乱URL列表
        import random
        random.shuffle(page_urls)
        for url in page_urls:
            yield response.follow(url, callback=self.parse_page)

6. 优先使用官方API #

如果Yelp批准了访问权限，优先使用官方API获取数据，这是最安全合规的方式，完全规避爬虫封禁风险。

内容的提问来源于stack exchange，提问作者helloworldIwanttocode