Ubuntu环境下多线程Selenium崩溃及续爬问题求助

一、内存泄漏与invalid session id问题修复 #

核心原因 #

当前单线程复用单个WebDriver处理大量请求，加上异常场景下资源释放不彻底，导致内存持续累积，最终触发session失效。同时Selenium在多线程环境下存在线程安全风险，进一步加剧了内存泄漏问题。

具体修复方案 #

1. 限制Driver生命周期，定期重建

不让单个Driver处理上千条请求，每处理固定数量的记录后主动销毁并重建，避免内存持续膨胀：

def run_scraper(starting_file_number, end_file_number):
    current_file_number = starting_file_number
    driver = get_new_driver()
    request_count = 0
    MAX_REQUESTS_PER_DRIVER = 50  # 可根据服务器性能调整

    csv_file = 'entity_details2.csv'
    try:
        driver.uc_open_with_reconnect(SITE_URL, 10)
        while current_file_number <= end_file_number:
            try:
                # 原有搜索、验证码处理、数据抓取逻辑保持不变

                current_file_number += 1
                request_count += 1

                # 每处理指定数量请求后重建Driver
                if request_count >= MAX_REQUESTS_PER_DRIVER:
                    driver.quit()
                    driver = get_new_driver()
                    driver.uc_open_with_reconnect(SITE_URL, 10)
                    request_count = 0

            except Exception as e:
                # 原有异常处理逻辑保持不变
                driver.quit()
                driver = get_new_driver()
                driver.uc_open_with_reconnect(SITE_URL, 10)
                request_count = 0  # 重置计数
                continue
    finally:
        driver.quit()

2. 改用多进程替代多线程

Selenium的WebDriver并非线程安全，多进程可隔离每个Driver的内存空间，从根源避免跨线程内存泄漏：

if __name__ == "__main__":
    num_workers = 3
    start_file_number, end_file_number = get_file_numbers()
    range_size = (end_file_number - start_file_number + 1) // num_workers

    file_number_ranges = [
        (start_file_number + i * range_size, start_file_number + (i + 1) * range_size - 1)
        for i in range(num_workers)
    ]
    file_number_ranges[-1] = (file_number_ranges[-1][0], end_file_number)

    print(Fore.CYAN + "File number ranges for each worker:", file_number_ranges)

    # 替换为ProcessPoolExecutor
    from concurrent.futures import ProcessPoolExecutor
    with ProcessPoolExecutor(max_workers=num_workers) as executor:
        futures = [executor.submit(run_scraper, start, end) for start, end in file_number_ranges]
        for future in futures:
            future.result()

    print("All scraping instances are done!")

注意：使用多进程时，确保get_new_driver()及全局变量在子进程中能正确初始化，避免跨进程资源冲突。

3. 优化页面资源清理

每次返回列表页后，主动清理页面缓存，减少内存占用：

# 在driver.back()执行后添加以下代码
driver.execute_script("window.localStorage.clear();")
driver.execute_script("window.sessionStorage.clear();")
driver.execute_script("document.cookie.split(';').forEach(cookie => document.cookie = cookie.replace(/^ +/, '').replace(/=.*/, `=;expires=${new Date(0).toUTCString()};path=/`));")

4. 增强Chrome启动参数，降低内存消耗

修改get_new_driver()，添加内存优化相关启动参数：

from selenium.webdriver.chrome.options import Options

def get_new_driver():
    proxy = proxy_rotator.get_next_proxy()
    options = Options()
    options.add_argument("--disable-extensions")
    options.add_argument("--disable-gpu")
    options.add_argument("--no-sandbox")
    options.add_argument("--disable-dev-shm-usage")  # 解决Ubuntu下/dev/shm空间不足问题
    options.add_argument("--disk-cache-size=10485760")  # 限制缓存为10MB
    options.add_argument("--low-memory")
    return Driver(uc=True, headless=True, multi_proxy=True, proxy=proxy, options=options, block_images=True)

二、续爬功能实现 #

1. 状态保存与加载逻辑 #

通过本地文件记录最后成功处理的文件编号，异常中断后可直接从该位置恢复：

import json
from filelock import FileLock  # 需要先安装：pip install filelock

def save_scraper_state(current_file_number):
    # 用文件锁避免多进程写入冲突
    lock = FileLock("scraper_state.lock")
    with lock:
        with open('scraper_state.json', 'w') as f:
            json.dump({'last_processed': current_file_number}, f)

def load_scraper_state(default_start):
    try:
        lock = FileLock("scraper_state.lock")
        with lock:
            with open('scraper_state.json', 'r') as f:
                state = json.load(f)
                return state.get('last_processed', default_start)
    except FileNotFoundError:
        return default_start

2. 整合到爬虫流程 #

在run_scraper中成功处理每个文件编号后保存状态，主函数启动时加载状态作为起始编号：

def run_scraper(starting_file_number, end_file_number):
    current_file_number = starting_file_number
    driver = get_new_driver()
    request_count = 0
    MAX_REQUESTS_PER_DRIVER = 50

    csv_file = 'entity_details2.csv'
    try:
        driver.uc_open_with_reconnect(SITE_URL, 10)
        while current_file_number <= end_file_number:
            try:
                # 原有搜索、验证码处理、数据抓取逻辑保持不变

                # 成功处理后立即保存状态
                save_scraper_state(current_file_number)
                current_file_number += 1
                request_count += 1

                # 定期重建Driver逻辑保持不变

            except Exception as e:
                # 原有异常处理逻辑保持不变
                continue
    finally:
        driver.quit()

if __name__ == "__main__":
    num_workers = 3
    # 加载上次中断的位置，若无则使用默认起始值
    default_start, end_file_number = get_file_numbers()
    start_file_number = load_scraper_state(default_start)

    range_size = (end_file_number - start_file_number + 1) // num_workers
    # 后续范围划分和执行逻辑保持不变

三、额外建议 #

• 调整num_workers：若Ubuntu内存有限，可尝试将并发数降至2，观察内存占用变化。
• 监控内存使用：使用htop工具实时监控进程内存消耗，根据实际情况调整MAX_REQUESTS_PER_DRIVER参数。

内容的提问来源于stack exchange，提问作者Adil Mughal