如何为Scrapy Playwright配置动态代理？遇超时错误求解决

问题背景 #

给Scrapy Playwright配置动态代理时，发现scrapy-proxy-pool和Scrapy Playwright兼容性不佳，于是基于proxyscrape自行实现代理轮换，但始终遇到代理连接超时错误，报错net::ERR_TIMED_OUT。

原代码 #

import scrapy
from scrapy_playwright.page import PageMethod
from proxyscrape import create_collector

collector = create_collector('proxy', 'http')

class ProxySpider(scrapy.Spider):
    name = 'proxy'
    
    PLAYWRIGHT_LAUNCH_OPTIONS = {
        "headless": False,
        "timeout": 100 * 1000,  # 20 seconds
    
    }
    
    def start_requests(self):
        proxy = collector.get_proxy()
        print("Proxy --> http://"+proxy.host+":"+proxy.port)
      
        yield scrapy.Request("http://httpbin.org/get", meta={
            "playwright": True,
            "playwright_context_kwargs": {
                "java_script_enabled": True,
                "ignore_https_errors": True,
                "proxy": {
                    "server": "http://"+proxy.host+":"+proxy.port,
                },
            },
            })       
    
    def parse(self,response):
        print(response.text)

错误日志 #

File "/home/sappy/.virtualenvs/121-server/lib/python3.10/site-packages/scrapy_playwright/handler.py", line 297, in _download_request                                                                                                                                       
result = await self._download_request_with_page(request, page, spider)                                                             
File "/home/sappy/.virtualenvs/121-server/lib/python3.10/site-packages/scrapy_playwright/handler.py", line 331, in _download_request_with_page                                                                                                                              
response = await page.goto(url=request.url, **page_goto_kwargs)                                                                    
File "/home/sappy/.virtualenvs/121-server/lib/python3.10/site-packages/playwright/async_api/_generated.py", line 9162, in goto         
await self._impl_obj.goto(                                                                                                         
File "/home/sappy/.virtualenvs/121-server/lib/python3.10/site-packages/playwright/_impl/_page.py", line 494, in goto                   
return await self._main_frame.goto(**locals_to_params(locals()))                                                                   
File "/home/sappy/.virtualenvs/121-server/lib/python3.10/site-packages/playwright/_impl/_frame.py", line 147, in goto                  
await self._channel.send("goto", locals_to_params(locals()))                                                                       
File "/home/sappy/.virtualenvs/121-server/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 44, in send              
return await self._connection.wrap_api_call(                                                                                       
File "/home/sappy/.virtualenvs/121-server/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 419, in wrap_api_call   
return await cb()                                                                                                                  
File "/home/sappy/.virtualenvs/121-server/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 79, in inner_send       
result = next(iter(done)).result()                                                                                                 
playwright._impl._api_types.Error: net::ERR_TIMED_OUT at http://httpbin.org/get                                                      
=========================== logs ===========================                                                                        
navigating to "http://httpbin.org/get", waiting until "load"                                                                        
============================================================

问题原因 #

1. 免费代理可用性极低：proxyscrape获取的免费代理多为临时资源，大部分已失效或被目标站点封禁，直接使用必然触发超时。
2. 缺少代理预验证：未在使用前测试代理连通性，直接发起请求导致无效代理浪费时间。
3. 超时配置不全：仅设置了Playwright启动超时，未配置页面跳转超时，也没有代理失败后的重试逻辑。

修复方案 #

1. 增加代理预验证逻辑

使用前先测试代理是否能正常访问目标站点，过滤失效代理。

2. 完善超时与重试配置

给Playwright页面跳转、Scrapy请求分别添加超时和重试规则。

3. 添加失败重试机制

代理超时后自动获取新代理并重试请求。

修改后的代码 #

import scrapy
import requests
from scrapy_playwright.page import PageMethod
from proxyscrape import create_collector
from scrapy.utils.retry import get_retry_request

collector = create_collector('proxy', 'http')

def is_proxy_valid(proxy):
    """验证代理是否可用"""
    proxy_url = f"http://{proxy.host}:{proxy.port}"
    try:
        # 5秒超时测试httpbin连通性
        response = requests.get(
            "http://httpbin.org/get",
            proxies={"http": proxy_url, "https": proxy_url},
            timeout=5
        )
        return response.status_code == 200
    except:
        return False

class ProxySpider(scrapy.Spider):
    name = 'proxy'
    # 配置Scrapy重试规则
    custom_settings = {
        "RETRY_TIMES": 3,
        "RETRY_HTTP_CODES": [500, 502, 503, 504, 408, 429],
    }
    
    PLAYWRIGHT_LAUNCH_OPTIONS = {
        "headless": False,
        "timeout": 100 * 1000,
    }
    
    def start_requests(self):
        yield self.get_request_with_valid_proxy()
    
    def get_request_with_valid_proxy(self):
        """循环获取可用代理并生成请求"""
        proxy = None
        # 最多尝试5次获取可用代理
        for _ in range(5):
            proxy = collector.get_proxy()
            if is_proxy_valid(proxy):
                print(f"使用有效代理 --> http://{proxy.host}:{proxy.port}")
                break
        else:
            self.logger.error("未找到可用代理")
            return
        
        return scrapy.Request(
            "http://httpbin.org/get",
            meta={
                "playwright": True,
                "playwright_context_kwargs": {
                    "java_script_enabled": True,
                    "ignore_https_errors": True,
                    "proxy": {
                        "server": f"http://{proxy.host}:{proxy.port}",
                    },
                },
            },
            errback=self.errback,
        )
    
    def parse(self, response):
        print(response.text)
        # 如需持续爬取，可在此生成新请求
        # yield self.get_request_with_valid_proxy()
    
    async def errback(self, failure):
        """请求失败回调，切换新代理重试"""
        self.logger.error(f"请求失败: {failure.value}")
        # 生成重试请求并替换为新代理
        yield self.get_request_with_valid_proxy()

额外建议 #

• 免费代理稳定性差，生产环境建议改用付费代理服务，这类代理池的可用性和稳定性远高于免费资源。
• 可将验证通过的代理缓存，避免重复验证，提升爬取效率。

内容的提问来源于stack exchange，提问作者saprative