Python Selenium遇NoSuchElementException，如何筛选订阅量<500的创作者

我有一个包含3个YouTube视频链接的list_of_links，其中2个视频的创作者订阅量不足500，1个创作者订阅量极高。我的目标是仅提取订阅量小于500的创作者链接，但运行代码时抛出NoSuchElementException错误，回溯信息如下：

NoSuchElementException                    Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_18264\3954802536.py in <module>
      5     driver.get(link)
      6     x_path_for_followers = '//*[@id="owner-sub-count"]'
----> 7     followers = driver.find_element(By.XPATH, value=x_path_for_followers)
      8     nFollowers = getNumberOfFolowers(followers) # a function to parse from the text here to number
      9     if nFollowers > 500:

尝试的代码 #

list_of_links = ['https://www.youtube.com/watch?v=rXUW-BfPNx8', 'https://www.youtube.com/watch?v=b9s_xrycvGA', 'https://www.youtube.com/watch?v=k9TUPpGqYTo']

creators = set()
for link in list_of_links:
    driver.get(link)
    x_path_for_followers = '//*[@id="owner-sub-count"]'
    followers = driver.find_element(By.XPATH, value=x_path_for_followers)
    nFollowers = getNumberOfFolowers(followers) # a function to parse from the text here to number
    if nFollowers > 500:
        continue
    x_path_for_creator = '//*[@id="text"]/a'
    creator = driver.find_element(By.XPATH, value=x_path_for_creator)
    href = creator.get_attribute("href")
    creators.append(href)

需求：获取2个订阅量不足500的创作者链接：@translationalbiology5464、@imjustslightlybetter。

错误原因 #

NoSuchElementException由以下情况导致：

• 页面未完全加载就执行元素查找，目标元素还未渲染
• YouTube页面结构变更，原XPATH定位失效
• 部分视频创作者隐藏了订阅数元素（隐私设置）
• 代码中错误地对set()类型使用append()方法

修复步骤 #

1. 添加显式等待，确保元素加载完成后再操作
2. 优化XPATH定位逻辑，使用更稳定的标签+属性组合
3. 增加异常捕获，单个链接处理失败不中断整个循环
4. 修正集合操作，用add()替代append()

修复后的代码 #

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException, NoSuchElementException

list_of_links = ['https://www.youtube.com/watch?v=rXUW-BfPNx8', 'https://www.youtube.com/watch?v=b9s_xrycvGA', 'https://www.youtube.com/watch?v=k9TUPpGqYTo']

creators = set()
wait = WebDriverWait(driver, 10)  # 设置10秒超时阈值

def getNumberOfFolowers(follower_element):
    # 解析订阅数字符串，处理"123 subscribers"或"4.5K subscribers"格式
    text = follower_element.text.strip().split()[0]
    if 'K' in text:
        return int(float(text.replace('K', '')) * 1000)
    elif 'M' in text:
        return int(float(text.replace('M', '')) * 1000000)
    else:
        return int(text) if text.isdigit() else 0

for link in list_of_links:
    driver.get(link)
    try:
        # 等待订阅数元素加载完成
        followers = wait.until(EC.presence_of_element_located(("xpath", '//yt-formatted-string[@id="owner-sub-count"]')))
        nFollowers = getNumberOfFolowers(followers)
        
        if nFollowers >= 500:
            continue
        
        # 等待创作者主页链接加载完成
        creator = wait.until(EC.presence_of_element_located(("xpath", '//ytd-channel-name//a')))
        href = creator.get_attribute("href")
        creators.add(href)
    except (TimeoutException, NoSuchElementException):
        # 跳过加载失败或元素缺失的链接
        print(f"无法处理链接: {link}")
        continue

# 输出结果
print("订阅量不足500的创作者链接:")
for creator_link in creators:
    print(creator_link)

关键说明 #

• 显式等待避免了页面未加载完成导致的元素查找失败
• 优化后的XPATH通过标签+ID组合定位，适配页面结构小幅度变更
• 异常捕获保证循环稳定执行，不会因单个链接报错终止
• 补充了getNumberOfFolowers函数的实现，支持解析带K/M的订阅数格式

内容的提问来源于stack exchange，提问作者Aleksandra25