Selenium爬取层级网页报stale element reference异常求解

你遇到的StaleElementReferenceException本质原因是：每次点击筛选选项后，页面的DOM结构会被局部刷新重渲染，你在循环开始前提前获取并存到变量里的元素引用，会随着DOM刷新直接失效，此时再操作这些旧引用就会抛出元素不存在的异常。
除此之外你的代码还有三个隐性问题：

• 所有层级的筛选项都用了完全相同的XPath//label/div[@class='select-item m-1']，没有按筛选层级做区分，经常会定位到其他层级的选项，逻辑本身就有问题
• 硬编码time.sleep()等待时间不稳定，要么等太久拖慢爬取速度，要么页面没加载完就操作触发报错
• 漏导入了Service类，原代码直接运行会先报导入错误

• 禁止提前缓存跨操作的元素列表：每次点击操作、每次进入下一层/返回上一层时，都重新实时查找当前需要操作的元素，不沿用之前存的旧元素引用
• 按筛选层级区分定位逻辑：每一层只定位当前分组下的选项，不要全局搜所有同class的选项
• 把固定等待换成显式等待：等元素真正可点击了再执行点击操作，稳定性更高
• 修正数据嵌套的append逻辑，避免爬取到的数据层级混乱

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

ITEM = []

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
# 配置显式等待时长，单位秒
WAIT = WebDriverWait(driver, 10)
search_url = "https://northladder.com/en/ae/electronics/laptop/dell/g-series/v1/5e2c888401c848ac4f695c87"
driver.get(search_url)

# 第一层：一级筛选选项
first_level_xpath = "//div[@class='select-box-main align-items-center'][1]/label/div[@class='select-item m-1']"
first_level_count = len(WAIT.until(EC.presence_of_all_elements_located((By.XPATH, first_level_xpath))))
for i in range(first_level_count):
    # 每次循环重新查找一级元素，避免引用失效
    first_ele = WAIT.until(EC.presence_of_all_elements_located((By.XPATH, first_level_xpath)))[i]
    first_name = first_ele.text
    first_ele.click()
    # 等二级选项加载完成
    WAIT.until(EC.presence_of_all_elements_located((By.XPATH, "//div[@class='select-box-main align-items-center'][2]/label/div[@class='select-item m-1']")))

    # 第二层：世代选项
    second_level_xpath = "//div[@class='select-box-main align-items-center'][2]/label/div[@class='select-item m-1']"
    second_level_count = len(driver.find_elements(By.XPATH, second_level_xpath))
    for g in range(second_level_count):
        second_ele = WAIT.until(EC.presence_of_all_elements_located((By.XPATH, second_level_xpath)))[g]
        second_name = second_ele.text
        second_ele.click()
        WAIT.until(EC.presence_of_all_elements_located((By.XPATH, "//div[@class='select-box-main align-items-center'][3]/label/div[@class='select-item m-1']")))

        # 第三层：RAM选项
        third_level_xpath = "//div[@class='select-box-main align-items-center'][3]/label/div[@class='select-item m-1']"
        third_level_count = len(driver.find_elements(By.XPATH, third_level_xpath))
        for r in range(third_level_count):
            third_ele = WAIT.until(EC.presence_of_all_elements_located((By.XPATH, third_level_xpath)))[r]
            third_name = third_ele.text
            third_ele.click()
            WAIT.until(EC.presence_of_all_elements_located((By.XPATH, "//div[@class='select-box-main align-items-center'][4]/label/div[@class='select-item m-1']")))

            # 第四层：型号选项
            fourth_level_xpath = "//div[@class='select-box-main align-items-center'][4]/label/div[@class='select-item m-1']"
            fourth_level_count = len(driver.find_elements(By.XPATH, fourth_level_xpath))
            for m in range(fourth_level_count):
                fourth_ele = WAIT.until(EC.presence_of_all_elements_located((By.XPATH, fourth_level_xpath)))[m]
                fourth_name = fourth_ele.text
                fourth_ele.click()
                WAIT.until(EC.presence_of_all_elements_located((By.XPATH, "//div[@class='select-box-main align-items-center'][5]/label/div[@class='select-item m-1']")))

                # 第五层：屏幕尺寸选项
                fifth_level_xpath = "//div[@class='select-box-main align-items-center'][5]/label/div[@class='select-item m-1']"
                fifth_level_count = len(driver.find_elements(By.XPATH, fifth_level_xpath))
                for s in range(fifth_level_count):
                    fifth_ele = WAIT.until(EC.presence_of_all_elements_located((By.XPATH, fifth_level_xpath)))[s]
                    fifth_name = fifth_ele.text
                    fifth_ele.click()
                    WAIT.until(EC.presence_of_all_elements_located((By.XPATH, "//div[@class='select-box-main align-items-center'][6]/label/div[@class='select-item m-1']")))

                    # 第六层：存储容量选项
                    sixth_level_xpath = "//div[@class='select-box-main align-items-center'][6]/label/div[@class='select-item m-1']"
                    sixth_level_count = len(driver.find_elements(By.XPATH, sixth_level_xpath))
                    for mm in range(sixth_level_count):
                        sixth_ele = WAIT.until(EC.presence_of_all_elements_located((By.XPATH, sixth_level_xpath)))[mm]
                        sixth_name = sixth_ele.text
                        # 按层级存储数据
                        current_item = [first_name, second_name, third_name, fourth_name, fifth_name, sixth_name]
                        ITEM.append(current_item)
                        print(current_item)

补充说明：如果运行时某一层的索引和实际页面层级不对应，按F12查看筛选项的select-box-main分组顺序，调整XPath里的序号即可，核心逻辑不变的前提下不会再出现元素引用失效的问题。

内容的提问来源于stack exchange，提问作者user19399799