使用Python multiprocessing的Pool.map搭配Selenium时程序越来越慢如何解决

问题核心原因 #

• 每爬取1个商品链接就重新创建、销毁1次ChromeDriver实例，频繁的进程启停会产生大量僵尸Chrome/ChromeDriver进程，持续占用系统CPU、内存资源，运行越久资源占用越高，最终拖慢整体速度
• 未给Chrome浏览器设置资源限制参数，单个Chrome实例内存占用随运行持续升高
• 进程池任务无分批处理逻辑，大量任务堆积进一步加剧资源消耗

优化方案 #

1. 单进程复用ChromeDriver #

每个工作进程仅初始化1次Chrome实例，进程生命周期内所有爬取任务复用同一个驱动，避免反复创建销毁的开销。

2. 增加Chrome资源限制参数 #

添加禁用插件、图片、缓存等参数，大幅降低单个Chrome实例的资源占用。

3. 替换固定等待为显式等待 #

移除固定1秒硬等待，改为等待页面核心元素加载完成即终止等待，提升爬取效率。

优化后代码示例 #

# libraries
import os
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from multiprocessing import Pool

# 每个进程单独持有的driver对象
process_driver = None

# 进程初始化函数：每个进程启动时创建一次driver
def init_worker():
    global process_driver
    options = webdriver.ChromeOptions()
    options.headless = True
    options.add_experimental_option("excludeSwitches", ['enable-automation'])
    options.add_argument(
        '--user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"')
    # 新增资源限制参数
    options.add_argument('--disable-gpu')
    options.add_argument('--disable-images')
    options.add_argument('--disable-plugins')
    options.add_argument('--disable-extensions')
    options.add_argument('--no-sandbox')
    options.add_argument('--disable-dev-shm-usage')
    options.add_argument('--disk-cache-size=0')
    
    directory = os.path.dirname(os.path.realpath(__file__))
    chromedriver_path = directory + "\\chromedriver\\chromedriver.exe"
    process_driver = webdriver.Chrome(executable_path=chromedriver_path, options=options)

# 爬取函数复用全局driver
def openChrome_headless(url1, name):
    global process_driver
    try:
        process_driver.get(url=url1)
        # 显式等待页面商品标题加载完成，最多等3秒
        WebDriverWait(process_driver, 3).until(
            EC.presence_of_element_located((By.TAG_NAME, 'h1'))
        )
        with open(name + ".html", "w", encoding="utf-8") as file:
            file.write(process_driver.page_source)
    except Exception as ex:
        print(ex)

def processing_goods_pages(name):
    directory = os.path.dirname(os.path.realpath(__file__))
    url = "https://eldorado.ua/"
    for n in os.listdir(f"brand_pages\\{name}"):
        with open(f"{directory}\\brand_pages\\{name}\\{n}", encoding="utf-8") as file:
            soup = BeautifulSoup(file.read(), "lxml")

        save_dir = f"{directory}\\goods_pages\\{name}\\{n[:-5]}"
        if not os.path.exists(save_dir):
            os.makedirs(save_dir, exist_ok=True)

        links = soup.find_all("header", class_="good-description")
        for li in links:
            ref = url + li.find('a').get('href')
            print(li.text)
            openChrome_headless(ref, f"{save_dir}\\{li.text}")

if __name__ == "__main__":
    dict1 = {"Смартфоны и телефоны": "https://eldorado.ua/node/c1038944/",
         "Телевизоры и аудиотехника": "https://eldorado.ua/node/c1038957/",
         "Ноутбуки, ПК и Планшеты": "https://eldorado.ua/node/c1038958/",
         "Техника для кухни": "https://eldorado.ua/node/c1088594/",
         "Техника для дома": "https://eldorado.ua/node/c1088603/",
         "Игровая зона": "https://eldorado.ua/node/c1285101/",
         "Гаджеты и аксесуары": "https://eldorado.ua/node/c1215257/",
         "Посуда": "https://eldorado.ua/node/c1039055/",
         "Фото и видео": "https://eldorado.ua/node/c1038960/",
         "Красота и здоровье": "https://eldorado.ua/node/c1178596/",
         "Авто и инструменты": "https://eldorado.ua/node/c1284654/",
         "Спорт и туризм": "https://eldorado.ua/node/c1218544/",
         "Товары для дома и сада": "https://eldorado.ua/node/c1285161/",
         "Товары для детей": "https://eldorado.ua/node/c1085100/"}
    ar2 = list(dict1.keys())
    # 进程池指定初始化函数
    with Pool(processes=6, initializer=init_worker) as pool:
        pool.map(processing_goods_pages, ar2)
    # 所有任务完成后自动销毁进程，driver也会随进程退出自动释放

额外优化建议 #

• 可定期手动清理系统残留的chromedriver.exe和chrome.exe僵尸进程
• 若任务量极大，可将任务拆分为多批，每跑完一批就关闭一次进程池释放资源后再跑下一批
• 可适当调低进程数，避免同时运行的Chrome实例过多占满系统资源

内容的提问来源于stack exchange，提问作者Arondy