Scrapy爬虫配置UA与代理中间件仍遇403错误求解决方案

已经实现随机UA和代理IP中间件，但特定节点仍返回403 Forbidden，下面针对你提供的代码分析问题并给出修复方案：

一、现有中间件的问题分析 #

1. ScrapeOpsFakeUserAgentMiddleware的逻辑缺陷 #

• _scrapeops_fake_user_agents_enabled方法逻辑错误：不管API key是否有效、功能是否开启，最后都会强制把scrapeops_fake_user_agents_active设为True，可能导致无效场景下仍尝试使用该中间件，进而获取不到有效UA列表。
• 未处理UA获取失败的情况：调用ScrapeOps API时如果网络出错、API key无效，user_agents_list会为空，后续取随机UA时会触发索引越界错误，导致请求不带UA就发送，直接触发反爬。
• 仅设置User-Agent：很多网站会校验Accept、Accept-Language、Referer等头部，单一UA不足以绕过反爬规则。

2. RandomProxyMiddleware的不足 #

• 无代理有效性校验：随机选的代理可能已经失效或被目标网站拉黑，直接使用会返回403。
• 未处理代理协议匹配：如果目标站点是HTTPS，用HTTP代理可能出现协议不兼容问题；没有区分代理的HTTP/HTTPS类型。
• 缺少代理认证支持：如果代理需要账号密码，当前代码没处理认证逻辑，请求会无法通过代理。

二、修复后的中间件代码 #

修复ScrapeOpsFakeUserAgentMiddleware #

import requests
from urllib.parse import urlencode
import random

class ScrapeOpsFakeUserAgentMiddleware:
    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings)

    def __init__(self, settings):
        self.scrapeops_api_key = settings.get('SCRAPEOPS_API_KEY')
        self.scrapeops_endpoint = settings.get('SCRAPEOPS_FAKE_USER_AGENT_ENDPOINT', 'http://headers.scrapeops.io/v1/user-agents?') 
        self.scrapeops_fake_user_agents_active = settings.get('SCRAPEOPS_FAKE_USER_AGENT_ENABLED', False)
        self.scrapeops_num_results = settings.get('SCRAPEOPS_NUM_RESULTS', 100)
        self.user_agents_list = []
        # 先校验启用条件，再获取UA列表
        if self._scrapeops_fake_user_agents_enabled():
            self._get_user_agents_list()

    def _get_user_agents_list(self):
        try:
            payload = {'api_key': self.scrapeops_api_key}
            if self.scrapeops_num_results is not None:
                payload['num_results'] = self.scrapeops_num_results
            response = requests.get(self.scrapeops_endpoint, params=urlencode(payload), timeout=10)
            response.raise_for_status()  # 触发HTTP错误异常
            json_response = response.json()
            self.user_agents_list = json_response.get('result', [])
        except Exception as e:
            print(f"获取UA列表失败: {str(e)}")
            # 备用UA，避免空列表报错
            self.user_agents_list = ['Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36']

    def _get_random_user_agent(self):
        if not self.user_agents_list:
            return 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
        return random.choice(self.user_agents_list)

    def _scrapeops_fake_user_agents_enabled(self):
        if not self.scrapeops_api_key or self.scrapeops_api_key.strip() == '' or not self.scrapeops_fake_user_agents_active:
            self.scrapeops_fake_user_agents_active = False
        else:
            self.scrapeops_fake_user_agents_active = True
        return self.scrapeops_fake_user_agents_active

    def process_request(self, request, spider):        
        if not self.scrapeops_fake_user_agents_active:
            return
        
        random_user_agent = self._get_random_user_agent()
        request.headers['User-Agent'] = random_user_agent
        # 添加其他关键请求头，模拟真实浏览器
        request.headers['Accept'] = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
        request.headers['Accept-Language'] = 'zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3'
        request.headers['Accept-Encoding'] = 'gzip, deflate, br'
        request.headers['Connection'] = 'keep-alive'
        # 自动设置Referer，模拟从首页跳转
        if not request.headers.get('Referer'):
            request.headers['Referer'] = spider.start_urls[0] if spider.start_urls else 'https://www.example.com'

修复RandomProxyMiddleware #

import random
from scrapy.exceptions import NotConfigured
import base64

class RandomProxyMiddleware:
    def __init__(self, settings):
        self.proxies = settings.get('ROTATING_PROXY_LIST', [])
        self.proxy_auth = settings.get('ROTATING_PROXY_AUTH', None)  # 格式：'username:password'
        
        if not self.proxies:
            raise NotConfigured("未配置代理列表")

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings)

    def process_request(self, request, spider):
        proxy = random.choice(self.proxies)
        request.meta['proxy'] = proxy
        
        # 处理代理账号密码认证
        if self.proxy_auth:
            auth_encoded = base64.b64encode(self.proxy_auth.encode()).decode()
            request.headers['Proxy-Authorization'] = f'Basic {auth_encoded}'
    
    def process_response(self, request, response, spider):
        # 代理失效时更换重试
        if response.status in [403, 407, 500, 502]:
            if request.meta['proxy'] in self.proxies:
                self.proxies.remove(request.meta['proxy'])
            return request.copy()
        return response
    
    def process_exception(self, request, exception, spider):
        # 代理连接失败时更换重试
        if 'proxy' in request.meta and request.meta['proxy'] in self.proxies:
            self.proxies.remove(request.meta['proxy'])
        return request.copy()

三、额外优化建议 #

• 调整中间件优先级：在settings.py中确保代理中间件优先级高于UA中间件，避免UA被覆盖：

  DOWNLOADER_MIDDLEWARES = {
      'your_project.middlewares.RandomProxyMiddleware': 300,
      'your_project.middlewares.ScrapeOpsFakeUserAgentMiddleware': 400,
      'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,  # 禁用默认UA中间件
      'scrapy.downloadermiddlewares.retry.RetryMiddleware': 500,
  }
  

• 添加请求延迟：避免请求过于频繁触发反爬：

  DOWNLOAD_DELAY = 2
  RANDOMIZE_DOWNLOAD_DELAY = True
  

• 启用Cookies中间件：模拟真实用户会话：

  DOWNLOADER_MIDDLEWARES = {
      'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700,
  }
  

• 检查代理质量：如果手动维护的代理列表质量差，可考虑使用专业代理池服务，自动获取有效代理。
• JS渲染适配：如果目标站点用JS渲染校验，可结合scrapy-playwright或splash进行渲染爬取。

内容的提问来源于stack exchange，提问作者Antônio Correia