Selenium Python：如何跳过不存在元素及处理多h4标签

问题描述 #

我正在使用Selenium Python抓取多个相似网页，同一信息在部分页面对应不同的XPATH。现有两个需交替使用的XPATH：

city_e = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, "//div/h4")))

alternative_name_e = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//*[@id='SFbizctc53fcd34ec260b1442c7bd7b4']/div/div[1]")))

完整代码如下：

results = []

for i in range(6):
    links = WebDriverWait(driver, 60).until(
            EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".SFpne a"))
        )
    links[i].click()
    time.sleep(4)


    name_e = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, "//div/h3"))
        )
    alternative_name_e = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, "//*[@id='SFbizctc53fcd34ec260b1442c7bd7b4']/div/div[1]"))
        )
    city_e = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.XPATH, "//div/h4"))
        )       
    
    jobTitle_e = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, "//div[@itemprop='contactPoint']/div"))
        ) 
    address_e = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, "//div[@itemprop='contactPoint']/address"))
        )
    cell_e = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, "//div/a[@class='SFbizctcphn']"))
        )
    email_e = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, "//*[@id='SFbizpne0']/div[3]/div/a"))
        )


    full_member_content = {
    'Member_Name': [alternative_name_e.text],
    'member_Name': [name_e.text],
    'member_City': [city_e.text],
    'member_JobTitle': [jobTitle_e.text],
    'member_Address': [address_e.text],
    'member_Cell': [cell_e.text],
    'member_Email': [email_e.text]
        }
    
    
    results.append(full_member_content)
    time.sleep(4)
    driver.back()
    
print(results)

请问是否可以通过try-catch或if语句实现元素不存在时跳过？另外，当页面存在多个h4标签时，能否重新执行city_e的定位代码？

解决方案 #

一、用try-except实现元素不存在时跳过 #

Python中使用try-except捕获Selenium的TimeoutException（WebDriverWait超时会抛出该异常），即可实现元素不存在时跳过抓取，避免整个循环中断。同时可以给字段设置默认值，保证数据结构完整。

修改后的核心代码示例：

from selenium.common.exceptions import TimeoutException

results = []

for i in range(6):
    links = WebDriverWait(driver, 60).until(
            EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".SFpne a"))
        )
    links[i].click()
    time.sleep(4)

    # 初始化所有字段默认值
    member_data = {
        'Member_Name': None,
        'member_Name': None,
        'member_City': None,
        'member_JobTitle': None,
        'member_Address': None,
        'member_Cell': None,
        'member_Email': None
    }

    # 逐个元素抓取，用try-except包裹
    try:
        name_e = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, "//div/h3"))
        )
        member_data['member_Name'] = name_e.text
    except TimeoutException:
        pass  # 元素不存在则跳过，保留默认值

    try:
        alternative_name_e = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, "//*[@id='SFbizctc53fcd34ec260b1442c7bd7b4']/div/div[1]"))
        )
        member_data['Member_Name'] = alternative_name_e.text
    except TimeoutException:
        pass

    # 支持city的双XPATH切换
    try:
        city_e = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, "//div/h4"))
        )
        member_data['member_City'] = city_e.text
    except TimeoutException:
        # 第一个XPATH失败，尝试第二个（替换为实际的city替代XPATH）
        try:
            city_e = WebDriverWait(driver, 10).until(
                EC.presence_of_element_located((By.XPATH, "你的city替代XPATH"))
            )
            member_data['member_City'] = city_e.text
        except TimeoutException:
            pass

    # 其他元素同理处理
    try:
        jobTitle_e = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, "//div[@itemprop='contactPoint']/div"))
        )
        member_data['member_JobTitle'] = jobTitle_e.text
    except TimeoutException:
        pass

    try:
        address_e = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, "//div[@itemprop='contactPoint']/address"))
        )
        member_data['member_Address'] = address_e.text
    except TimeoutException:
        pass

    try:
        cell_e = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, "//div/a[@class='SFbizctcphn']"))
        )
        member_data['member_Cell'] = cell_e.text
    except TimeoutException:
        pass

    try:
        email_e = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, "//*[@id='SFbizpne0']/div[3]/div/a"))
        )
        member_data['member_Email'] = email_e.text
    except TimeoutException:
        pass

    # 可选：仅当核心字段存在时，才将数据加入结果
    if member_data['member_Name'] is not None:
        results.append(member_data)
    
    time.sleep(4)
    driver.back()

print(results)

二、处理多个h4标签的情况 #

当页面存在多个h4标签时，有两种处理方式：

1. 优化XPATH直接定位目标元素

优先调整XPATH，通过父容器、属性、文本特征等缩小范围，直接定位到目标h4，避免无效的重复定位。例如：

# 假设目标h4在class为city-container的div下
city_e = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, "//div[@class='city-container']/h4"))
)

2. 封装定位函数实现重新定位

如果必须重新执行定位逻辑，可以将定位逻辑封装成函数，当第一次获取的元素不符合要求时，自动尝试其他定位方式：

def get_target_city(driver):
    # 先获取所有h4元素，筛选有效结果
    try:
        h4_elements = WebDriverWait(driver, 10).until(
            EC.presence_of_all_elements_located((By.XPATH, "//div/h4"))
        )
        # 筛选非空文本的h4，或者根据文本内容匹配
        for elem in h4_elements:
            if elem.text.strip() and "City" in elem.text:
                return elem
        # 没有符合条件的，尝试替代XPATH
        return WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, "你的city替代XPATH"))
        )
    except TimeoutException:
        return None

# 在循环中调用函数
city_e = get_target_city(driver)
if city_e:
    member_data['member_City'] = city_e.text

内容的提问来源于stack exchange，提问作者Chandler O'Neal