Selenium点击加载更多报Stale element exception等错误求助

问题根因与修复方案 #

现有代码核心问题 #

1. 滚动逻辑错误：点击load more后滚动回顶部，下次点击按钮时按钮处于视口外，必然出现遮挡无法点击的问题
2. 隐式等待滥用：隐式等待是全局设置，重复声明无意义，且无法满足「元素可点击后再操作」的场景，是Stale元素异常的核心诱因
3. 定位规则错误：find_element_by_class_name仅支持传入单个类名，你传入了dl-result-header dl-map-hover两个空格分隔的类名，必然触发无效定位报错

修复要点 #

• 全局仅设置一次隐式等待，核心交互逻辑换成显式等待，确保元素可点击后再执行操作
• 点击load more前先滚动到按钮所在位置，避免元素被遮挡
• 多类名元素定位换成CSS选择器实现
• 扩展循环异常捕获范围，覆盖Stale元素异常、点击被拦截等偶发异常

修正后可运行代码 #

from selenium import webdriver
from selenium.common.exceptions import ElementClickInterceptedException, ElementNotInteractableException, NoSuchElementException, StaleElementReferenceException
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

path = r"D:\\Chromedriver.exe"
driver = webdriver.Chrome(path)
# 全局隐式等待仅设置一次即可
driver.implicitly_wait(10)
driver.maximize_window()

url = "https://www.carecredit.com/doctor-locator/results/Any-Profession/Any-Specialty//?Sort=D&Radius=75&Page=1"
driver.get(url)

# 搜索地址操作
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="location-display"]'))).click()
search_bar = driver.find_element(By.XPATH, '//*[@id="location-display"]')
search_bar.clear()
search_bar.send_keys("00704")

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="dl-locate-button-new"]'))).click()

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="dl-radius-new"]'))).click()

# 处理cookie弹窗
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="onetrust-accept-btn-handler"]'))).click()

i = 0
loadMoreButtonExists = True
wait = WebDriverWait(driver, 15)

while loadMoreButtonExists:
    try:
        # 等待load more按钮可点击
        load_more = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@id="next-page"]')))
        # 滚动到按钮位置
        driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", load_more)
        # 用JS点击避免偶发遮挡问题
        driver.execute_script("arguments[0].click();", load_more)
        print(f"done {i}")
        i += 1
        # 等待新内容加载完成
        wait.until(lambda d: d.execute_script('return document.readyState') == 'complete')
    except (ElementNotInteractableException, NoSuchElementException, StaleElementReferenceException, ElementClickInterceptedException):
        loadMoreButtonExists = False

print("done loading button")

# 抓取数据
practise_items = driver.find_elements(By.CLASS_NAME, 'page-results')
try:
    for element in practise_items:
        # 多类名用CSS选择器定位
        practise_name = element.find_element(By.CSS_SELECTOR, '.dl-result-header.dl-map-hover').text
        address = element.find_element(By.CLASS_NAME, 'dl-result-address').text
        phone_no = element.find_element(By.CLASS_NAME, 'no-mobile').text
        print(practise_name)
        print(address)
        print(phone_no)
except Exception as e:
    print(e)
finally:
    driver.quit()

内容的提问来源于stack exchange，提问作者Samyak jain