Python多线程Selenium爬虫应对未知总页数的Pythonic实现方案咨询
未知总页数的Selenium并发抓取优化方案
核心思路
最符合Pythonic规范的实现从「线程安全终止信号」+「分批提交任务」+「资源复用」三个维度设计,既可以避免无效请求,也能保证代码的可读性和健壮性:
- 用Python原生线程安全的
threading.Event作为终止信号,避免普通全局变量的竞态问题 - 采用分批提交任务的模式,而非一次性提交所有分页任务,触发终止信号后立即停止提交新任务,从根源避免无效请求
- 新增页面有效性判断逻辑,匹配到无效页(404、无数据、参数错误等场景)时触发终止信号
- 线程级复用ChromeDriver,减少驱动初始化的性能开销
实现代码
from concurrent.futures import ThreadPoolExecutor, as_completed from selenium import webdriver from webdriver_manager.chrome import ChromeDriverManager import threading # 线程安全的终止信号 STOP_EVENT = threading.Event() # 线程本地存储,实现每个线程复用一个ChromeDriver THREAD_LOCAL = threading.local() N_THREAD = 10 # 仅作为最大兜底,避免站点异常时无限爬取,不会提前产生无效请求 MAX_PADDING = 100 def get_driver(): """线程本地获取/初始化ChromeDriver""" if not hasattr(THREAD_LOCAL, "driver"): THREAD_LOCAL.driver = webdriver.Chrome(ChromeDriverManager().install()) return THREAD_LOCAL.driver def crawl_example(page: int): # 已触发终止信号的话直接跳过请求,不会产生无效访问 if STOP_EVENT.is_set(): return None driver = get_driver() driver.get(f"https://example.com/{page}") # 此处替换为对应站点的无效页判断逻辑,可按需选择判断规则: # 1. 检测页面是否包含"404" "无相关数据"等关键字 # 2. 检测接口/页面返回状态码 # 3. 检测预期抓取的列表数据是否为空 page_invalid = "没有找到对应内容" in driver.page_source if page_invalid: STOP_EVENT.set() return None # 正常处理页面逻辑 result = f"Fetched page {page} data" return result if __name__ == "__main__": current_page = 1 all_results = [] with ThreadPoolExecutor(N_THREAD) as ex: while not STOP_EVENT.is_set() and current_page <= MAX_PADDING: # 每次提交和线程数等量的分页任务 futures = [ex.submit(crawl_example, page) for page in range(current_page, current_page + N_THREAD)] current_page += N_THREAD # 处理当前批次返回结果 for future in as_completed(futures): res = future.result() if res: all_results.append(res) # 所有任务结束后统一关闭所有线程的driver def close_driver(): if hasattr(THREAD_LOCAL, "driver"): THREAD_LOCAL.driver.quit() with ThreadPoolExecutor(N_THREAD) as ex: [ex.submit(close_driver) for _ in range(N_THREAD)] print(f"共抓取有效页面{len(all_results)}个")
内容的提问来源于stack exchange,提问作者Sean
相关产品推荐
相关产品推荐

