使用while循环爬取网站企业信息时遇StaleElementReferenceException错误求助

从网站抓取企业信息时，代码无while循环可正常运行，加入循环后抛出StaleElementReferenceException错误，提示stale element reference: stale element not found，且确认对应的XPath能定位到元素。

报错行 #

list = stockList.find_element(by="xpath", value = './a[@class="company-announcement-link"]').text

错误追踪 #

Exception has occurred: StaleElementReferenceException Message: stale element reference: stale element not found Backtrace: GetHandleVerifier [0x00A88893+48451] (No symbol) [0x00A1B8A1] (No symbol) [0x00925058]

原代码 #

dropdown = driver.find_elements(by='xpath', value = '//div[@class="main-block flexible-submenu"]/section[@class="container border-bottom border-dark"]/div[@class=" tab-content"]/div/div/div/div[@class="dataTables_length"]/label/select/option')
i=0
while(i<len(dropdown)):
    if(dropdown[i].text=="All"):
        dropdown[i].click()
        
        lists = []
        stockLists = driver.find_elements(by='xpath', value = '//div[@class="main-block flexible-submenu"]/section[@class="container border-bottom border-dark"]/div[@class=" tab-content"]/div/div[@class]/div[@id="DataTables_Table_0_wrapper"]/div[@class="dataTables_scroll"]/div[@class="dataTables_scrollBody"]/table/tbody/tr/td[@class=" text-left position-relative"]')
        for stockList in stockLists:
            list = stockList.find_element(by="xpath", value = './a[@class="company-announcement-link"]').text
            lists.append(list)
        my_dict = {'stock': lists}
        df_headlines=pd.DataFrame(my_dict)
        df_headlines.to_csv('headline.csv')
        
    i=i+1

driver.quit()

错误原因 #

点击"All"选项后，页面DOM会重新渲染（比如表格加载全部数据），之前通过find_elements获取的stockLists元素引用会失效——这些元素已经被新的DOM节点替换，后续操作时自然触发过时元素引用异常。

修复步骤 #

1. 弃用过期元素引用：点击"All"后，必须等待页面完全加载，再重新定位表格元素，不能复用之前缓存的元素列表。
2. 简化循环逻辑：原while循环没必要遍历所有下拉选项，直接定位"All"选项点击即可，减少冗余操作。
3. 添加显式等待：确保元素加载完成后再操作，避免因元素未就绪导致的异常。

修改后的代码 #

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

# 直接定位"All"选项并点击，无需遍历所有option
all_option = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable(('xpath', '//div[@class="main-block flexible-submenu"]/section[@class="container border-bottom border-dark"]/div[@class=" tab-content"]/div/div/div/div[@class="dataTables_length"]/label/select/option[text()="All"]'))
)
all_option.click()

lists = []
# 显式等待表格滚动体加载完成
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located(('xpath', '//div[@class="dataTables_scrollBody"]/table/tbody'))
)
# 重新获取最新的td元素列表
stockLists = driver.find_elements(by='xpath', value = '//div[@class="main-block flexible-submenu"]/section[@class="container border-bottom border-dark"]/div[@class=" tab-content"]/div/div[@class]/div[@id="DataTables_Table_0_wrapper"]/div[@class="dataTables_scroll"]/div[@class="dataTables_scrollBody"]/table/tbody/tr/td[@class=" text-left position-relative"]')

for stockList in stockLists:
    # 显式等待子元素加载完成后获取文本
    company_name = WebDriverWait(stockList, 5).until(
        EC.presence_of_element_located(('xpath', './a[@class="company-announcement-link"]'))
    ).text
    lists.append(company_name)

my_dict = {'stock': lists}
df_headlines = pd.DataFrame(my_dict)
df_headlines.to_csv('headline.csv')

driver.quit()

额外建议 #

• 尽量使用简洁的XPath定位，比如通过id或class直接定位关键节点，减少层级依赖，提升定位稳定性。
• 只要页面发生DOM重渲染，之前获取的所有元素引用都会失效，必须重新定位。

内容的提问来源于stack exchange，提问作者Ong Chan Meng