如何在同一终端通过代理运行Scrapy与Scrapy-Playwright爬虫

问题背景 #

在公司代理环境下，两个爬虫存在代理配置冲突：

• 普通Scrapy爬虫（quotes）必须依赖HTTP_PROXY和HTTPS_PROXY环境变量才能正常请求
• Scrapy-Playwright爬虫（quotesscroll）通过settings.py的PLAYWRIGHT_LAUNCH_OPTIONS配置代理，但全局设置HTTP_PROXY/HTTPS_PROXY后会直接失效

下面提供三种可行的解决方式：

方式1：给普通爬虫单独配置代理（不依赖系统环境变量） #

通过代码动态为不同爬虫加载代理配置，不需要全局设置环境变量。修改配置后用脚本统一启动两个爬虫：

1. 更新settings.py #

import os

REQUEST_FINGERPRINTER_IMPLEMENTATION = "2.7"
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
FEED_EXPORT_ENCODING = "utf-8"
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

# Playwright专属代理配置
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "proxy": {
        "server": os.environ.get("SERVER"),
        "username": os.environ.get("USERNAME"),
        "password": os.environ.get("PASSWORD"),
    },
}

2. 编写启动脚本（比如run_spiders.py） #

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
import os

def get_quotes_proxy():
    # 从环境变量读取代理信息，转为Scrapy需要的格式
    proxy_server = os.environ.get("SERVER")
    proxy_user = os.environ.get("USERNAME")
    proxy_pwd = os.environ.get("PASSWORD")
    # 代理需要认证时，拼接带用户名密码的URL
    if proxy_user and proxy_pwd:
        proxy_url = f"http://{proxy_user}:{proxy_pwd}@{proxy_server.split('://')[1]}"
    else:
        proxy_url = proxy_server
    return {
        "HTTP_PROXY": proxy_url,
        "HTTPS_PROXY": proxy_url
    }

if __name__ == "__main__":
    settings = get_project_settings()
    
    # 启动quotes爬虫，加载专属代理配置
    quotes_settings = settings.copy()
    quotes_settings.update(get_quotes_proxy())
    process = CrawlerProcess(quotes_settings)
    process.crawl("quotes")
    
    # 启动quotesscroll爬虫，使用默认配置
    process.crawl("quotesscroll")
    
    process.start()

3. 运行脚本 #

# 确保SERVER、USERNAME、PASSWORD环境变量已设置
python run_spiders.py

方式2：启动普通爬虫时通过命令行指定代理 #

无需修改代码，启动两个爬虫时分别传递配置：

# 启动quotes爬虫，临时传递代理参数（需要认证就把用户名密码嵌入URL）
scrapy crawl quotes -s HTTP_PROXY=http://username:password@your-proxy:port -s HTTPS_PROXY=http://username:password@your-proxy:port

# 启动quotesscroll爬虫，直接用默认配置
scrapy crawl quotesscroll

方式3：让Playwright忽略系统代理环境变量 #

如果想全局设置HTTP_PROXY/HTTPS_PROXY给普通爬虫用，同时让Scrapy-Playwright不受影响，只需在settings.py中新增一行配置：

# 原有的Playwright代理配置不变
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "proxy": {
        "server": os.environ.get("SERVER"),
        "username": os.environ.get("USERNAME"),
        "password": os.environ.get("PASSWORD"),
    },
}

# 新增配置：让Playwright无视系统的代理环境变量
PLAYWRIGHT_IGNORE_HTTP_PROXY_ENV_VARS = True

设置完成后，全局配置代理环境变量即可同时运行两个爬虫：

export HTTP_PROXY=http://username:password@your-proxy:port
export HTTPS_PROXY=http://username:password@your-proxy:port

# 依次启动爬虫
scrapy crawl quotes
scrapy crawl quotesscroll

内容的提问来源于stack exchange，提问作者Luftuq