Load More按钮触发StaleElementReferenceException问题的解决方法咨询

问题背景 #

编写爬虫爬取产品信息，目标网站带有Load More按钮，逻辑是持续点击直到按钮不可见，但运行时抛出StaleElementReferenceException异常。

原代码 #

def extract_prodirectsports_data():
    print("entered")
    print(url_line)
    url_line = "https://www.prodirectsport.com/soccer/l/adults/departments-boots/activity-football/brand-adidas/silo-predator/"
    spreadsheet_data = []
    pdf_data = []
    sizes = []

    cotacao_libra()  # 假设该函数已定义
    i = 0
    driver.get(url_line)
    
    # 处理Cookies
    WebDriverWait(driver, 100).until(EC.element_to_be_clickable((By.XPATH, "//button[@title='Accept all cookies']"))).click()
    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@id='zonos']")))
    close_button = driver.find_element(By.XPATH, "//a[@class = 'z-close']")
    driver.execute_script("arguments[0].click();", close_button)
    
    close_button_2 = driver.find_element(By.XPATH, "//button[@aria-label='Close']")
    driver.execute_script("arguments[0].click();", close_button_2)
    
    # 初始化表格DataFrame
    df_spreadsheet = pd.DataFrame(columns=SPREADSHEET_COLUMNS)
    pdf = FPDF()
    pdf.add_page()
    pdf.set_font('Arial', 'B', 12)
    
    x = 0
    driver.maximize_window()

    view_more_button = driver.find_element(By.XPATH, "//div[@class = 'product-listing__view-more']/button")
    while True:
        """view_more_button = WebDriverWait(driver, 20).until(
        EC.visibility_of_element_located((By.XPATH, "//div[@class = 'product-listing__view-more']/button")))"""
        
        if view_more_button is not None:
            driver.execute_script("arguments[0].scrollIntoView();", view_more_button)
            driver.execute_script("arguments[0].click();", view_more_button)
            x += 1
            print(x)
            time.sleep(5)
        else:
            pass

    products_list = WebDriverWait(driver, 20).until(
        EC.visibility_of_element_located((By.XPATH, "//div[@class = 'product-listing__grid']"))
    )
    driver.execute_script("arguments[0].scrollIntoView();", products_list)
        
    for index, value in enumerate(products_list):
        products_list = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class = 'product-listing__grid']//div[@class = '_root_129ai_6 product-listing__grid-item']/a")))
        driver.execute_script("arguments[0].scrollIntoView();", products_list[index])
        driver.save_screenshot("window.png")
        image_element = WebDriverWait(driver, 50).until(EC.presence_of_element_located((By.TAG_NAME, "img")))
        image = image_element.get_attribute("src")
        i += 1
        print(i)
        actions.move_to_element(products_list[index]).perform()
                
        driver.execute_script("arguments[0].click();", products_list[index])

抛出的异常 #

selenium.common.exceptions.StaleElementReferenceException: Message: The element with the reference e5fa4a38-375f-488b-b118-2b240e35fcf7 is stale; either its node document is not the active document, or it is no longer connected to the DOM; For documentation on this error, please visit: [相关文档链接]

核心问题分析 #

1. 元素引用失效：点击Load More按钮后，页面加载新内容并更新DOM，循环前获取的view_more_button元素引用会因DOM重构失效，触发异常。
2. 死循环无终止逻辑：原代码while True没有终止条件，按钮消失后仍会尝试操作失效元素。
3. 产品遍历逻辑混乱：初始products_list是单个元素而非列表，遍历过程中重复定义变量，存在逻辑错误。

修复方案 #

1. 重构Load More按钮循环逻辑 #

每次循环重新查找按钮，通过捕获超时异常判断按钮是否消失，避免使用失效引用：

x = 0
driver.maximize_window()

# 循环点击Load More直到按钮不可见
while True:
    try:
        # 每次循环重新等待并获取可点击的按钮
        view_more_button = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.XPATH, "//div[@class = 'product-listing__view-more']/button"))
        )
        driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", view_more_button)
        driver.execute_script("arguments[0].click();", view_more_button)
        x += 1
        print(f"已点击Load More {x}次")
        # 等待新内容加载完成，用显式等待替代固定sleep
        WebDriverWait(driver, 15).until(
            EC.presence_of_all_elements_located((By.XPATH, "//div[@class='_root_129ai_6 product-listing__grid-item']"))
        )
    except TimeoutException:
        # 超时说明按钮已不存在，终止循环
        print("Load More按钮已消失，停止加载")
        break

2. 修复产品列表遍历逻辑 #

避免缓存失效元素，每次操作前重新获取产品元素，确保遍历有效性：

# 获取所有产品链接
products_list = WebDriverWait(driver, 20).until(
    EC.visibility_of_all_elements_located((By.XPATH, "//div[@class = 'product-listing__grid']//div[@class = '_root_129ai_6 product-listing__grid-item']/a"))
)
print(f"共找到{len(products_list)}个产品")

for idx, _ in enumerate(products_list, 1):
    try:
        # 每次操作前重新定位当前产品，防止DOM更新导致引用失效
        current_product = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.XPATH, f"(//div[@class = '_root_129ai_6 product-listing__grid-item']/a)[{idx}]"))
        )
        driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", current_product)
        driver.save_screenshot(f"product_{idx}.png")
        
        # 精准定位产品图片，避免匹配页面其他图片
        image_element = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, f"(//div[@class = '_root_129ai_6 product-listing__grid-item']//img)[{idx}]"))
        )
        image = image_element.get_attribute("src")
        print(f"正在处理第{idx}个产品，图片链接：{image}")
        
        # 进入产品页面
        driver.execute_script("arguments[0].click();", current_product)
        
        # 此处添加产品页面信息抓取逻辑
        
        # 返回列表页并等待加载完成
        driver.back()
        WebDriverWait(driver, 10).until(
            EC.visibility_of_all_elements_located((By.XPATH, "//div[@class = '_root_129ai_6 product-listing__grid-item']/a"))
        )
    except Exception as e:
        print(f"处理第{idx}个产品时出错：{str(e)}")
        continue

3. 通用防StaleElement技巧 #

• 不提前缓存元素引用：每次需要操作元素时，通过定位器重新查找，避免DOM更新后引用失效。
• 用显式等待替代sleep：精准等待元素状态变化，减少无效等待，避免DOM未更新完成就操作。
• 异常捕获与重试：对易出现异常的操作添加try-except块，捕获异常后重新查找元素重试。

内容的提问来源于stack exchange，提问作者André