使用Selenium和Python循环点击按钮直至消失时遇TimeoutException问题

问题描述 #

爬取指定页面时，输入邮编后无法通过while循环持续点击“Load More”按钮，且获取位置结果的代码抛出TimeoutException，错误信息如下：

raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

用户原代码：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select

options = webdriver.ChromeOptions()
driver = webdriver.Chrome(options = options)
action = ActionChains(driver)

driver.get("https://www.vertexconnects.com/find-atc")
driver.maximize_window()
wait = WebDriverWait(driver,5)
# Use below line only if you are getting the Accept/Reject cookies pop-up
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[contains(.,'Accept All')]"))).click()

location_textbox = wait.until(EC.presence_of_element_located((By.ID,"location-search-input")))
action.move_to_element(location_textbox).click().send_keys("10001").perform()
wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "atc-finder-button"))).click()

while True:
    try:
        wait.until(EC_element_to_be_clickable((By.ID, "loadMore"))).click()
    except:
        break

print("done")

locations = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='location-result']")))

for location in locations:
    name = location.find_element(By.TAG_NAME, "h4").text()
    address = location.find_element(By.CLASS_NAME, "address atc-finder-hospital-address").text()
    phone_num = location.find_element(By.TAG_Name, "a").text
    print(name, address, phone_num)

问题排查与修复方案 #

1. 修正拼写错误 #

• EC_element_to_be_clickable 应为 EC.element_to_be_clickable，缺少点运算符导致无法调用预期条件方法
• By.TAG_Name 应为 By.TAG_NAME，大小写拼写错误

2. 延长等待时间 #

原等待时间仅5秒，页面加载更多内容可能超时，将WebDriverWait(driver,5)改为WebDriverWait(driver,15)

3. 修复元素定位与文本获取逻辑 #

• text是元素属性而非方法，需将.text()改为.text
• CLASS_NAME不支持多类名定位，将By.CLASS_NAME, "address atc-finder-hospital-address"改为By.CSS_SELECTOR, ".address.atc-finder-hospital-address"

4. 优化“Load More”点击逻辑 #

每次点击后需等待新内容加载，避免重复点击或提前终止循环，可通过等待按钮状态变化判断加载完成

修正后的代码 #

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
driver = webdriver.Chrome(options=options)
action = ActionChains(driver)

driver.get("https://www.vertexconnects.com/find-atc")
driver.maximize_window()
# 延长等待时间至15秒，避免加载超时
wait = WebDriverWait(driver, 15)

# 处理Cookie弹窗（若出现）
try:
    wait.until(EC.element_to_be_clickable((By.XPATH, "//button[contains(.,'Accept All')]"))).click()
except:
    pass

# 输入邮编并搜索
location_textbox = wait.until(EC.presence_of_element_located((By.ID, "location-search-input")))
action.move_to_element(location_textbox).click().send_keys("10001").perform()
wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "atc-finder-button"))).click()

# 循环点击Load More按钮
while True:
    try:
        # 修正EC方法调用的拼写错误
        load_more_btn = wait.until(EC.element_to_be_clickable((By.ID, "loadMore")))
        load_more_btn.click()
        # 等待新内容加载完成，通过按钮状态变化判断
        wait.until(EC.staleness_of(load_more_btn))
    except:
        break

print("所有内容加载完成")

# 获取所有位置结果
locations = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='location-result']")))

# 提取位置信息
for location in locations:
    # 修正text()为text属性
    name = location.find_element(By.TAG_NAME, "h4").text
    # 用CSS_SELECTOR定位多类名元素
    address = location.find_element(By.CSS_SELECTOR, ".address.atc-finder-hospital-address").text
    # 修正TAG_Name为TAG_NAME
    phone_num = location.find_element(By.TAG_NAME, "a").text
    print(f"名称: {name}\n地址: {address}\n电话: {phone_num}\n---")

driver.quit()

内容的提问来源于stack exchange，提问作者user3628240