爬取阿根廷护理人员注册网站时遇Selenium StaleElementReferenceException求助

我使用Python结合Selenium和BeautifulSoup爬取阿根廷社会发展部护理人员注册网站，代码能完成导航步骤并访问首个表格，但之后WebDriver会自行关闭，控制台抛出如下错误：

Traceback (most recent call last):
File "/Users/martin/Desktop/Scrap/scrapy1-3.py", line 33, in
select2.select_by_visible_text(option2.text)
...
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

使用Chrome浏览器版本：109.0.5414.87

当前代码如下：

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
import time
from bs4 import BeautifulSoup

driver = webdriver.Chrome(
    '/Users/martin/Downloads/chromedriver_mac64/chromedriver')

# Abrir el sitio web
driver.get("https://registroncd.senaf.gob.ar/ListadoCuidadores.aspx")

# Esperar a que la página cargue antes de hacer scraping
time.sleep(3)

boton1 = driver.find_element(
    By.XPATH, "//*[@id='ContentPlaceHolder1_DropDownListProv']")
select1 = Select(boton1)
options1 = select1.options

for option1 in options1:
    select1.select_by_visible_text(option1.text)
    time.sleep(3)  # wait for the page to load
    boton2 = driver.find_element(
        By.XPATH, "//*[@id='ContentPlaceHolder1_DropDownListLoc']")
    select2 = Select(boton2)
    options2 = select2.options

    for i in range(1, len(options2)):
        option2 = options2[i]
        select2.select_by_visible_text(option2.text)
        time.sleep(3)  # wait for the page to load
        boton3 = driver.find_element(By.ID, "ContentPlaceHolder1_ButtonBuscar")
        boton3.click()
        time.sleep(3)
        wait = WebDriverWait(driver, 10)
        element = wait.until(EC.presence_of_element_located(
            (By.ID, "ContentPlaceHolder1_GridView1")))
        soup = BeautifulSoup(driver.page_source, "html.parser")
        table = soup.find("table", class_="gridview")
        if table:
            rows = table.find_all("tr")
            for row in rows:
                cells = row.find_all("td")
                for cell in cells:
                    print(cell.text)
        else:
            print("La tabla no ha sido encontrada")

异常原因及修复方案 #

问题本质 #

StaleElementReferenceException 是因为页面DOM发生重新渲染（比如选择省份后，城市下拉框会动态加载新选项），之前缓存的元素对象已经和当前页面的元素脱离关联，导致引用失效。

核心修复思路 #

• 不要提前缓存元素列表，每次操作前重新定位元素
• 用显式等待替代固定的time.sleep()，确保元素处于可交互状态后再操作
• 循环内部重新获取下拉框及选项，避免使用失效的元素引用

修改后的代码 #

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from bs4 import BeautifulSoup

# 初始化浏览器
driver = webdriver.Chrome('/Users/martin/Downloads/chromedriver_mac64/chromedriver')
driver.get("https://registroncd.senaf.gob.ar/ListadoCuidadores.aspx")
wait = WebDriverWait(driver, 10)

# 获取所有省份选项的文本（只存文本，不存元素对象）
prov_select = Select(wait.until(EC.element_to_be_clickable((By.ID, "ContentPlaceHolder1_DropDownListProv"))))
prov_options = [opt.text for opt in prov_select.options]

for prov_text in prov_options:
    # 重新定位省份下拉框并选择
    prov_select = Select(wait.until(EC.element_to_be_clickable((By.ID, "ContentPlaceHolder1_DropDownListProv"))))
    prov_select.select_by_visible_text(prov_text)
    
    # 等待城市下拉框加载完成，获取所有城市选项文本
    loc_select = Select(wait.until(EC.element_to_be_clickable((By.ID, "ContentPlaceHolder1_DropDownListLoc"))))
    loc_options = [opt.text for opt in loc_select.options][1:]  # 跳过默认空白选项
    
    for loc_text in loc_options:
        # 重新定位城市下拉框并选择
        loc_select = Select(wait.until(EC.element_to_be_clickable((By.ID, "ContentPlaceHolder1_DropDownListLoc"))))
        loc_select.select_by_visible_text(loc_text)
        
        # 点击搜索按钮
        search_btn = wait.until(EC.element_to_be_clickable((By.ID, "ContentPlaceHolder1_ButtonBuscar")))
        search_btn.click()
        
        # 等待表格加载完成后解析
        wait.until(EC.presence_of_element_located((By.ID, "ContentPlaceHolder1_GridView1")))
        soup = BeautifulSoup(driver.page_source, "html.parser")
        table = soup.find("table", class_="gridview")
        
        if table:
            rows = table.find_all("tr")
            for row in rows:
                cells = row.find_all("td")
                for cell in cells:
                    print(cell.text.strip())
        else:
            print("未找到表格")

# 关闭浏览器
driver.quit()

额外优化建议 #

• 可以添加重试机制：捕获StaleElementReferenceException异常，重试当前操作1-2次，提升代码稳定性
• 若网站有反爬限制，可添加随机等待时间（比如time.sleep(random.uniform(1,3))），避免被检测

内容的提问来源于stack exchange，提问作者Martin