Selenium循环第9次迭代触发TimeoutException：无法定位enrollment_num的XPath

运行爬取https://beta.clinicaltrials.gov/新加坡地区招募中临床试验信息的Selenium代码时，循环遍历搜索页10条试验链接，第9次迭代触发TimeoutException，无法定位enrollment_num对应的XPath；仅遍历前8条时代码运行正常。

原始代码 #

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time

wait = WebDriverWait(driver, 20) # 等待20秒让页面加载

driver.get('https://beta.clinicaltrials.gov/') # 访问目标网站

driver.maximize_window()
time.sleep(1)

country = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@id="content"]/div/ctg-home/div/div[2]/ctg-advanced-search-home/div[2]/div[1]/fieldset/div[2]/div[3]/ctg-location-search-input/form/div[2]/div/label')))
country.click()

searchBar = driver.find_element("id",'location-input')
searchBar.send_keys("Singapore") # 输入国家名称

search_dropdown = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@id="mat-option-14"]/span'))) # 等待下拉选项可点击
search_dropdown.click()


search_button = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@id="content"]/div/ctg-home/div/div[2]/ctg-advanced-search-home/div[2]/div[2]/div/div[2]/button')))
search_button.click()


# 找到招募状态筛选按钮
filter_button = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@id="filter-button-statusGroup"]')))
filter_button.click()

# 选择"招募中"状态
recruiting = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@id="adv-check-status"]/div[2]/div[2]/div/label')))
recruiting.click()

# 存储临床试验详情的变量
clinical_trial = {}
name_list = []
phone_list = []
email_list = []
enrollment = []
condition_list = []

# 遍历搜索页的10条试验链接
for i in range(1,11):
    time.sleep(2) # 等待页面加载
    xpath = '//*[@id="content"]/div/ctg-search-results/div[2]/div/div[2]/div/div[2]/div[1]/ctg-search-hit-card[{}]/div/header/a'.format(i)
    trials = driver.find_element("xpath", xpath)
    trials.click()
    
    # 获取联系人姓名
    name = wait.until(EC.visibility_of_element_located((By.XPATH, '//*[@id="studyDetailsInfo"]/ctg-study-info/div/ctg-study-info-view/div/div[2]/ctg-study-contacts-and-locations/div/div/div/ctg-study-contact-info/p[1]/span')))
    name_list.append(name.text)
    
    # 获取联系电话
    phone = wait.until(EC.visibility_of_element_located((By.XPATH, '//*[@id="studyDetailsInfo"]/ctg-study-info/div/ctg-study-info-view/div/div[2]/ctg-study-contacts-and-locations/div/div/div/ctg-study-contact-info/p[2]/span')))
    phone_list.append(phone.text)
    
    # 获取联系邮箱
    email = wait.until(EC.visibility_of_element_located((By.XPATH, '//*[@id="studyDetailsInfo"]/ctg-study-info/div/ctg-study-info-view/div/div[2]/ctg-study-contacts-and-locations/div/div/div/ctg-study-contact-info/p[3]/ctg-study-contact-email/span/a')))
    email_list.append(email.text)
    
    # 获取招募人数
    enrollment_num = wait.until(EC.visibility_of_element_located((By.XPATH, '//*[@id="studyDetailsInfo"]/ctg-study-info/div/ctg-study-info-view/div/div[1]/ctg-study-overview/div[3]/div[2]/div[3]/div[2]')))
    enrollment.append(enrollment_num.text)
    
    # 获取试验病症
    conditions = wait.until(EC.visibility_of_element_located((By.XPATH, '//*[@id="studyDetailsInfo"]/ctg-study-info/div/ctg-study-info-view/div/div[1]/ctg-study-overview/div[3]/div[2]/div[1]/div[2]')))
    condition_list.append(conditions.text)
    
    driver.back() # 返回搜索页


# 将所有列表数据存入字典
clinical_trial["name"] = name_list
clinical_trial["phone_num"] = phone_list
clinical_trial["email_address"] = email_list
clinical_trial["Enrollment"] = enrollment
clinical_trial["Conditions"] = condition_list

原因分析 #

• 目标试验未公开招募人数：第9条临床试验可能未披露Enrollment数据，导致你编写的XPath对应的元素根本不存在，显式等待超时。
• XPath定位过于脆弱：依赖固定的DOM层级（如div[3]），如果该试验的Overview模块内元素顺序或结构与其他试验不同，XPath就会失效。
• 页面加载差异：第9条试验的页面动态渲染速度更慢，固定等待时间不足以让元素加载完成。

解决方案 #

1. 捕获异常处理缺失数据 #

给enrollment_num的定位逻辑添加try-except块，捕获超时异常时填充默认值：

# 修改后的招募人数获取部分
try:
    # 优化XPath，通过文本定位更稳定
    enrollment_num = wait.until(EC.visibility_of_element_located((By.XPATH, '//div[contains(text(),"Enrollment")]/following-sibling::div')))
    enrollment.append(enrollment_num.text)
except TimeoutException:
    enrollment.append("未公开")

2. 优化XPath定位 #

避免依赖固定层级，改用元素的文本内容关联定位，比如找包含"Enrollment"文本的元素的兄弟节点：

//div[contains(text(),"Enrollment")]/following-sibling::div

这种方式不受DOM层级变化影响，稳定性更高。

3. 调整等待策略 #

可以将EC.visibility_of_element_located改为EC.presence_of_element_located（只要元素存在就返回，无需等待可见），或适当延长等待时间：

wait = WebDriverWait(driver, 30) # 延长至30秒

4. 手动验证目标页面 #

手动打开第9条试验的详情页，确认是否存在Enrollment数据，判断是数据缺失还是定位问题。

内容的提问来源于stack exchange，提问作者Tan Jun Rong