Python+Selenium循环中StaleElementReferenceException问题求助

尝试用Python+Selenium抓取Le Monde网站新闻，之前运行正常，近期频繁触发StaleElementReferenceException，调试发现错误出在articles["title"] = titles[i].text行，但无法明确原因。

原代码： #

for i in urls[1797:4383]:
    browser.get(i)
    try: 
        pages_section = browser.find_element(by='css selector', value='section[class="river__pagination"]')
        tmp_pages = pages_section.find_elements(by='css selector', value='a')
        nb_pages = len(tmp_pages) #this is the number of page for each date
    except NoSuchElementException:
        nb_pages = 0
    
    #for each page
    for n in range(2, nb_pages+2): #+1 is because range does not include the last element / until excludes
        #collecting everything
        titles = titles = browser.find_elements(by = "css selector", value = "h3[class='teaser__title']")
        subtitles = browser.find_elements(by = "css selector", value = "p[class='teaser__desc']")
        dates = browser.find_elements(by = "css selector", value = "span[class='meta__date']")
     
#these two loops seem to be the problematic ones – i think?    
        #writing them into a df
        for i in range(len(titles)):
                articles = {}
                articles["title"] = titles[i].text
                articles["subtitle"] = subtitles[i].text
                articles["date"] = dates[i].text
                all_articles.append(articles)
                        
        #click on the next page
        sleep(1.3)
        if n <= nb_pages:
                browser.find_element(by='link text', value=str(n)).click()
            
    sleep(1)

all_articles = pd.DataFrame(all_articles)
all_articles.to_excel("/Users/victor/Desktop/test.xlsx", index=False)

错误信息： #

Traceback (most recent call last):
  File "<stdin>", line 18, in <module>
  File "/Users/victor/opt/anaconda3/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 84, in text
    return self._execute(Command.GET_ELEMENT_TEXT)['value']
  File "/Users/victor/opt/anaconda3/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 396, in _execute
    return self._parent.execute(command, params)
  File "/Users/victor/opt/anaconda3/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
    self.error_handler.check_response(response)
  File "/Users/victor/opt/anaconda3/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=107.0.5304.110)

错误原因分析 #

• 元素引用失效：点击分页后页面刷新，之前通过find_elements获取的titles、subtitles、dates属于旧DOM树元素，新页面加载后这些元素不再绑定到当前文档，调用.text时触发异常。
• 循环变量重名：外层循环变量i被内层循环for i in range(len(titles))覆盖，导致后续循环逻辑混乱，可能提前中断或错误引用页面。
• 固定等待不可靠：sleep(1.3)无法适配页面加载速度差异，若页面未完全加载就操作元素，会触发异常。

修复方案 #

1. 避免变量重名：将内层循环变量改为idx，防止覆盖外层循环的URL变量。
2. 分页后重新获取元素：每次分页加载完成后，重新定位当前页面的标题、副标题和日期元素，确保使用当前DOM中的有效引用。
3. 用显式等待替代固定sleep：通过WebDriverWait等待元素加载完成，确保元素可操作后再执行操作，提升稳定性。
4. 添加异常捕获：在单个元素操作时捕获StaleElementReferenceException，避免单次错误中断整个脚本。

修复后的完整代码 #

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException, StaleElementReferenceException
import pandas as pd

all_articles = []

for url in urls[1797:4383]:
    browser.get(url)
    try: 
        # 显式等待分页区域加载
        pages_section = WebDriverWait(browser, 10).until(
            EC.presence_of_element_located(('css selector', 'section[class="river__pagination"]'))
        )
        tmp_pages = pages_section.find_elements(by='css selector', value='a')
        nb_pages = len(tmp_pages)
    except NoSuchElementException:
        nb_pages = 0
    
    # 处理当前第一页数据
    try:
        # 等待标题元素加载完成
        WebDriverWait(browser, 10).until(
            EC.presence_of_all_elements_located(('css selector', 'h3[class="teaser__title"]'))
        )
        titles = browser.find_elements(by="css selector", value="h3[class='teaser__title']")
        subtitles = browser.find_elements(by="css selector", value="p[class='teaser__desc']")
        dates = browser.find_elements(by="css selector", value="span[class='meta__date']")
        
        for idx in range(len(titles)):
            try:
                articles = {}
                articles["title"] = titles[idx].text
                articles["subtitle"] = subtitles[idx].text
                articles["date"] = dates[idx].text
                all_articles.append(articles)
            except StaleElementReferenceException:
                # 单个元素失效时跳过
                continue
    except Exception as e:
        print(f"处理第一页数据出错: {e}")
    
    # 处理后续分页
    for n in range(2, nb_pages+2):
        try:
            # 等待分页按钮可点击
            next_page_btn = WebDriverWait(browser, 10).until(
                EC.element_to_be_clickable(('link text', str(n)))
            )
            next_page_btn.click()
            # 等待新页面元素加载完成
            WebDriverWait(browser, 10).until(
                EC.presence_of_all_elements_located(('css selector', 'h3[class="teaser__title"]'))
            )
            
            # 重新获取当前页面的所有元素
            titles = browser.find_elements(by="css selector", value="h3[class='teaser__title']")
            subtitles = browser.find_elements(by="css selector", value="p[class='teaser__desc']")
            dates = browser.find_elements(by="css selector", value="span[class='meta__date']")
            
            # 写入数据
            for idx in range(len(titles)):
                try:
                    articles = {}
                    articles["title"] = titles[idx].text
                    articles["subtitle"] = subtitles[idx].text
                    articles["date"] = dates[idx].text
                    all_articles.append(articles)
                except StaleElementReferenceException:
                    continue
        except Exception as e:
            print(f"处理第{n}页出错: {e}")
            continue

# 保存数据
all_articles_df = pd.DataFrame(all_articles)
all_articles_df.to_excel("/Users/victor/Desktop/test.xlsx", index=False)

内容的提问来源于stack exchange，提问作者Victor