Selenium Python抓取多下拉选项动态表格出现StaleElement异常如何解决

报错核心原因 #

• StaleElementReferenceException属于Selenium动态页面爬取的典型报错：每次选择区域后站点会局部刷新页面内容，上一轮循环中定位到的下拉框、表格元素的引用会直接失效，下一轮循环复用旧引用就会触发该错误。
• 还有两处隐藏问题会导致爬取失败：
  1. 你当前代码向列表中存储的是WebElement对象而非实际文本内容，导出csv只会得到对象内存地址，无法获得有效数据
  2. 你定义的regions列表里Latin America (including USA and Canada缺少右括号，无法匹配站点下拉选项，会额外触发选项匹配错误

修复后可运行代码 #

from bs4 import BeautifulSoup
import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.support.ui import Select

url = 'https://database.ib-net.org/search_utilities?type=2'
browser = webdriver.Chrome()
browser.get(url)
time.sleep(4)
print("Retriving the site...")

# 修正区域名称的括号错误
regions = ['Africa', 'East Asia and Pacific', 'Europe and Central Asia', 'Latin America (including USA and Canada)', 'Middle East and Northern Africa', 'South Asia']

for region in regions:
    print("Starting output for the region: " + region)
    # 每轮循环重新定位下拉框，避免 stale 错误
    selectOption = Select(browser.find_element_by_id('MainContent_ddRegion'))
    print("Now constructing output for: " + region)
    selectOption.select_by_visible_text(region)
    # 等待表格加载完成
    time.sleep(5)
    
    # 重新定位表格，避免旧引用失效
    table = browser.find_element_by_xpath("//*[@id='MainContent_gvUtilities']")
    table_rows = table.find_elements_by_xpath(".//tr")

    utility_name = [] 
    country = [] 
    city = []    
    population = [] 

    for row in table_rows[1:]: # 跳过表头行
        try:
            column_element = row.find_elements_by_xpath(".//td")
            # 提取文本内容而非存储WebElement对象
            utility_name.append(column_element[0].text.strip())
            country.append(column_element[2].text.strip())
            city.append(column_element[3].text.strip())
            population.append(column_element[4].text.strip())
        except:
            # 跳过异常行避免中断爬取
            continue

    dict_output = {
        "Utility Name": utility_name,
        "Country": country,
        "City": city,
        "Population": population,
    }

    df = pd.DataFrame(dict_output)
    # 补充csv后缀
    df.to_csv(f"{region}.csv", index = False)


browser.close()
browser.quit()

额外优化建议 #

如果希望爬取更稳定，可以把硬等待time.sleep替换成Selenium的显式等待逻辑，判断表格元素加载完成再继续执行，能减少不必要的等待时间同时降低加载失败概率。

内容的提问来源于stack exchange，提问作者Ken Ruto