Selenium多链接爬取：处理过期链接并避免爬取流程中断

问题背景 #

原代码在批量爬取链接时，遇到过期/超时链接触发TimeoutException后，调用driver.quit()关闭浏览器，但后续循环仍尝试使用已失效的driver对象，导致出现以下错误：

HTTPConnectionPool(host='localhost', port=xxx): Max retries exceeded with url: /session/xxx/url (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x...>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))

需求：触发超时或driver失效的异常时，关闭当前浏览器并重启，继续爬取下一个链接。

错误原因 #

调用driver.quit()会彻底终止Selenium与Chrome的会话，销毁driver对象。后续循环中再使用该对象调用driver.get()时，无法重新建立连接，因此抛出连接拒绝异常。

优化后的代码 #

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException, TimeoutException
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import json

def init_driver():
    driver_path = "C:\\Users\\Engineer_Stephen\\Downloads\\Compressed\\chromedriver"
    service = Service(driver_path)
    chrome_options = webdriver.ChromeOptions()
    # chrome_options.add_argument("--headless")
    chrome_options.add_argument("--start-maximized")
    return webdriver.Chrome(service=service, options=chrome_options)

data = []
driver = init_driver()
links = ['https://disasteralert.pdc.org/disasteralert/?hazard_id=202739', 'https://disasteralert.pdc.org/disasteralert/?hazard_id=202738']

for link in links:
    try:
        driver.get(link)
        wait = WebDriverWait(driver, 80)
        # 等待目标元素加载
        wait.until(EC.visibility_of_element_located((By.XPATH, '//span[@class="contour-HazardInfo-name montage-Text"]')))
        
        # 提取数据
        disaster = driver.find_element(By.XPATH, '//span[@class="contour-HazardInfo-name montage-Text"]')
        info = driver.find_element(By.XPATH, "//div[@class='contour-TabBarItem-label montage-Text']")
        types = driver.find_element(By.XPATH, "//div[@data-montage-id='type']")
        description = driver.find_element(By.XPATH, "//label/span[contains(text(), 'Description')]")
        descriptionText = driver.find_element(By.XPATH, '//div[@class="contour-HazardInfo-descriptionText montage-Text"]')

        data.append({
            'disaster': disaster.text,
            'info': info.text,
            'type': types.text,
            'description': description.text,
            'descriptionText': descriptionText.text
        })
        
    except NoSuchElementException:
        print(f"Data not available for link: {link}")
    except TimeoutException:
        print(f"Timeout occurred for link: {link}")
        # 关闭当前失效的driver并重启
        driver.quit()
        driver = init_driver()
    except Exception as e:
        print(f"An error occurred for link: {link}")
        print(str(e))
        # 异常导致driver失效时，重启浏览器
        driver.quit()
        driver = init_driver()

# 所有链接处理完成后关闭浏览器
driver.quit()

# 输出结果
resource = json.dumps(data, indent=4)
print(resource)

优化说明 #

1. 封装driver初始化逻辑：新增init_driver()函数，将浏览器配置、启动逻辑封装，方便在需要重启浏览器时快速调用。
2. 异常处理中重启driver：在TimeoutException和通用Exception分支中，先关闭当前失效的driver，再重新初始化新的driver实例，确保下一次循环有可用的浏览器会话。
3. 修正原代码缩进问题：修复原代码中descriptionText变量、except块的缩进错误，避免语法异常。
4. 统一资源释放：仅在所有链接处理完成后调用driver.quit()，避免提前关闭导致后续循环出错。

内容的提问来源于stack exchange，提问作者Stephen Mandivinga