如何在Selenium中为每个URL用随机代理且不重复打开Chrome窗口

问题现状 #

当前代码遍历URL时，每处理一个URL就会启动新的Chrome窗口；若将WebDriver实例移到循环外，又无法为每个URL分配随机代理。

需求 #

• 为每个URL使用随机代理，执行以下逻辑：

  chrome_options.add_argument(f"--proxy-server={proxy}")
  print(f"Using proxy: {proxy} for URL: {url}")
  

• 避免重复启动Chrome窗口

修改后的代码实现 #

from selenium import webdriver
import time
import random

def rand_proxy():
    with open('proxyscrape_premium_http_proxies.txt', 'r') as file:
        proxies = file.read().splitlines()
        return random.choice(proxies)

def set_proxy(driver, proxy):
    # 通过Chrome DevTools Protocol动态设置代理
    driver.execute_cdp_cmd('Network.setProxy', {
        'proxyType': 'manual',
        'httpProxy': proxy,
        'httpsProxy': proxy,
        'ftpProxy': proxy,
        'skipProxyForLocal': False
    })
    driver.execute_cdp_cmd('Network.enable', {})

with open('urls.txt', 'r') as file:
    urls = [url.strip() for url in file.readlines()]

# 仅初始化一次Chrome实例
chrome_options = webdriver.ChromeOptions()
driver = webdriver.Chrome(options=chrome_options)

try:
    for url in urls:
        proxy = rand_proxy()
        set_proxy(driver, proxy)
        print(f"Using proxy: {proxy} for URL: {url}")
        driver.get(url)
        time.sleep(2)  # 等待页面加载，可按需调整
finally:
    time.sleep(10)
    driver.quit()  # 确保浏览器最终关闭

关键说明 #

1. 动态切换代理：借助Chrome DevTools Protocol（CDP）的Network.setProxy命令，无需重启浏览器即可修改代理配置，避免重复创建WebDriver实例。
2. 实例复用：将Chrome实例的初始化移到循环外，仅启动一次浏览器，解决重复窗口问题。
3. 资源安全：用try...finally保证浏览器进程能被正常关闭，避免残留后台进程。

相关文件内容 #

proxyscrape_premium_http_proxies.txt #

38.162.7.130:3128
38.162.15.189:3128
38.162.5.127:3128
38.162.18.151:3128

urls.txt #

https://myexternalip.com/raw
https://2ip.io
http://httpbin.org/ip
https://ipv4.webshare.io

内容的提问来源于stack exchange，提问作者Bhagirath Saxena