Selenium UC Chrome驱动进程激增，内存CPU耗尽问题求助

核心原因 #

1. undetected-chromedriver进程残留：这个库为绕过反爬会修改Chrome启动参数，普通的driver.quit()没法彻底终止所有衍生的Chrome进程。
2. 全局与局部Driver变量冲突：代码同时用了global_driver全局变量和main里的局部driver变量，导致部分Driver实例引用丢失，没法正常关闭。
3. Chrome启动参数未优化：Headless模式默认内存占用高，缺少必要的资源限制参数。
4. 异常场景下Driver清理不彻底：部分异常分支没确保Driver被正确销毁。

具体修复步骤 #

1. 统一Driver变量管理，取消全局变量 #

删掉全局global_driver，改用局部变量传递，减少引用混乱：

def close_driver(driver):
    if driver:
        try:
            driver.quit()
        except Exception as e:
            logger.error(f"关闭Driver出错: {e}")
    return None

def create_driver():
    try:
        chrome_options = Options()
        # 加入内存优化和进程清理参数
        chrome_options.add_argument("--headless=new")  # 新版Headless更稳定
        chrome_options.add_argument("--disable-gpu")
        chrome_options.add_argument("--no-sandbox")
        chrome_options.add_argument("--disable-dev-shm-usage")  # 避免/dev/shm内存不足
        chrome_options.add_argument("--single-process")  # 强制单进程，减少进程数量
        chrome_options.add_argument("--disable-extensions")
        chrome_options.add_argument("--disable-plugins")
        chrome_options.add_argument("--disable-background-timer-throttling")
        # 限制V8引擎内存
        chrome_options.add_argument("--js-flags=--max_old_space_size=512")

        driver = uc.Chrome(options=chrome_options)
        driver.set_page_load_timeout(30)  # 设置页面加载超时，避免卡死
        return driver
    except Exception as e:
        logger.error(f"创建Driver实例出错: {e}")
        return None

2. 强制清理残留Chrome进程（Windows环境） #

借助psutil库强制杀掉所有关联Chrome进程，确保彻底清理：

import psutil  # 先安装：pip install psutil

def close_driver(driver):
    if driver:
        try:
            # 获取Driver关联的Chrome进程ID
            pid = driver.service.process.pid
            driver.quit()
            # 终止父进程和所有子进程
            parent = psutil.Process(pid)
            for child in parent.children(recursive=True):
                child.kill()
            parent.kill()
        except Exception as e:
            logger.error(f"关闭Driver或清理进程出错: {e}")
    return None

3. 修正main函数的Driver管理逻辑 #

确保每次重启Driver时旧实例被彻底清理，变量引用正确：

def main():
    logger.info("脚本启动...")
    try:
        source_data = read_source_csv("Data feed-22.01.24.csv")
        if not source_data:
            logger.info("源文件无数据，退出。")
            return

        processed_skus = read_processed_skus("fed_scraped_shipping_rates.csv")
        errors = read_processed_skus("out_of_memory_errors.csv") | read_processed_skus("rate_not_detected.csv") | read_processed_skus("red_errormsg_returned.csv") | read_processed_skus("404.csv")

        process_count = 0
        driver = create_driver()

        for row in source_data:
            for postcode_group in representative_postcodes:
                if (row['SKU'], postcode_group[0]) in processed_skus or (row['SKU'], postcode_group[0]) in errors:
                    continue

                if not driver:
                    logger.error("Driver不存在，重新创建...")
                    driver = create_driver()
                    if not driver:
                        logger.error("无法创建Driver，退出脚本。")
                        return

                result = process_sku(driver, row, postcode_group)
                process_count += 1

                if result == 'Out of Memory' or process_count >= 25:
                    driver = close_driver(driver)
                    driver = create_driver()
                    process_count = 0

    except Exception as e:
        logger.error(f"主函数出错: {e}")
    finally:
        driver = close_driver(driver)
        logger.info("脚本结束。")

4. 优化process_sku的异常处理 #

确保严重异常触发Driver重启：

def process_sku(driver, row, postcode_group):
    try:
        result = scrape_shipping_rates(driver, row['SKU'], row['Product URL'], postcode_group)

        if result == 'Success':
            logger.info(f"成功处理SKU: {row['SKU']}, 邮编组: {postcode_group[0]}")
            save_last_processed(row['SKU'], postcode_group[0])
        elif result == 'Out of Memory':
            logger.error(f"内存不足，SKU: {row['SKU']}, 邮编组: {postcode_group[0]}")
            return 'Out of Memory'
        else:
            logger.error(f"处理出错，SKU: {row['SKU']}, 邮编组: {postcode_group[0]}")

        time.sleep(1)
        return result
    
    except Exception as e:
        logger.error(f"process_sku出错: {e}")
        if "Out of Memory" in str(e) or "Timed out receiving message from renderer" in str(e):
            return 'Out of Memory'
        return 'Error'

额外建议 #

• 调低重启阈值：把process_count >=25改成10或15，更频繁重启Driver，减少单实例资源累积。
• 添加系统状态日志：每次重启前记录Chrome进程数和内存占用，方便排查：

  def log_system_status():
    chrome_processes = [p for p in psutil.process_iter() if 'chrome' in p.name().lower()]
    logger.info(f"当前Chrome进程数: {len(chrome_processes)}")
    logger.info(f"当前内存占用率: {psutil.virtual_memory().percent}%")
  

内容的提问来源于stack exchange，提问作者ptrcao