Selenium中h4标签触发StaleElementReferenceException异常问题

问题场景 #

使用Selenium抓取目标网站的h4标签内容时，触发StaleElementReferenceException异常，现有代码无法正常打印h4标签文本。

原Python代码 #

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException, StaleElementReferenceException

driver = webdriver.Firefox()
url = 'https://magicpin.in/New-Delhi/Paharganj/Restaurant/Eatfit/store/61a193/delivery'

driver.get(url)
delay = 3

try:
    element = WebDriverWait(driver, delay).until(
        EC.presence_of_element_located((By.CLASS_NAME, 'catalogItemsHolder')))
    articles = element.find_elements(By.TAG_NAME, 'article')

    for article in articles:
        try:
            h4_element = article.find_element(By.CLASS_NAME, 'categoryHeading')
            print(h4_element.text)
        except StaleElementReferenceException:
            print("H4 element is stale, re-fetching articles...")
            element = driver.find_element(By.CLASS_NAME, 'catalogItemsHolder')
            articles = element.find_elements(By.TAG_NAME, 'article')

except TimeoutException:
    print("Loading Timeout")
except Exception as e:
    print(e)
finally:
    driver.quit()

目标页面HTML结构 #

<div class="catalogItemsHolder">
   <article id="Kulcha Burger" class="categoryListing ">
      <h4 class="categoryHeading"> Kulcha Burger </h4>
      <div>
         
      </div>
   </article>
</div>

异常原因 #

StaleElementReferenceException的核心原因是：代码提前缓存了articles元素列表，但页面DOM在循环过程中发生刷新或重新渲染，导致列表中部分article元素的引用失效。即使在异常块中重新获取articles，当前循环也不会回溯处理已失效的元素，最终导致内容无法正常打印。

解决方案 #

1. 避免提前缓存元素列表，每次循环重新定位 #

不在循环外提前获取元素列表，而是每次循环都重新定位目标元素，确保操作的是最新的DOM节点。

2. 用显式等待确保元素可见并可交互 #

改用EC.visibility_of_all_elements_located等待元素可见（而非仅存在），避免操作未完全渲染的元素。

3. 优化定位路径 #

直接定位所有categoryHeading类的h4标签，减少多层级定位带来的失效风险。

修改后的代码 #

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException, StaleElementReferenceException

driver = webdriver.Firefox()
url = 'https://magicpin.in/New-Delhi/Paharganj/Restaurant/Eatfit/store/61a193/delivery'

driver.get(url)
delay = 10  # 延长等待时间适配页面加载速度

try:
    # 直接等待所有目标h4元素可见
    h4_elements = WebDriverWait(driver, delay).until(
        EC.visibility_of_all_elements_located((By.CLASS_NAME, 'categoryHeading')))
    
    for h4 in h4_elements:
        try:
            print(h4.text.strip())
        except StaleElementReferenceException:
            # 单个元素失效时重新获取列表，从当前位置继续遍历
            h4_elements = WebDriverWait(driver, delay).until(
                EC.visibility_of_all_elements_located((By.CLASS_NAME, 'categoryHeading')))
            index = h4_elements.index(h4)
            for remaining_h4 in h4_elements[index:]:
                print(remaining_h4.text.strip())
            break

except TimeoutException:
    print("页面加载超时")
except Exception as e:
    print(f"发生异常: {e}")
finally:
    driver.quit()

补充说明 #

如果页面存在滚动加载逻辑，需先触发动态加载再等待元素：

# 滚动到页面底部触发加载
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

内容的提问来源于stack exchange，提问作者user21934236