Selenium CSS选择器无法定位元素：LinkedIn爬取故障排查

可能的故障原因 #

• 元素匹配优先级变化：你的CSS选择器span[class='t-14 t-normal'] span会匹配父容器下的两个<span>标签（带aria-hidden="true"和visually-hidden类的）。find_element默认取第一个匹配元素，若LinkedIn调整了元素渲染顺序，或某个元素先加载但文本未填充，就会导致获取失败。
• 反爬隐性干扰：LinkedIn可能针对爬虫动态调整了学位文本的加载时机，即使HTML结构无变化，学位文本的填充也可能晚于院校信息，隐式等待的时长不足以覆盖这个延迟。
• 部分条目无学位信息：如果遍历的教育经历中存在无学位的条目，直接调用find_element会抛出异常，导致整个循环中断。

修复方案 #

方案1：优化选择器，精准定位目标元素 #

优先选择可见的学位文本元素，排除隐藏类元素：

from selenium.webdriver.common.by import By

for education in edu:
    ed['College'] += [education.find_element(By.CSS_SELECTOR, 'span.mr1.hoverable-link-text.t-bold span').text]
    # 定位非隐藏的学位span
    degree_elem = education.find_element(By.CSS_SELECTOR, 'span.t-14.t-normal > span:not(.visually-hidden)')
    ed['Degree'] += [degree_elem.text.strip()]

也可以用XPath筛选有实际文本的元素：

degree_elem = education.find_element(By.XPATH, ".//span[@class='t-14 t-normal']/span[normalize-space(text())!='']")

方案2：显式等待文本加载完成 #

针对学位元素的文本内容设置显式等待，确保文本填充后再获取：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

for education in edu:
    ed['College'] += [education.find_element(By.CSS_SELECTOR, 'span.mr1.hoverable-link-text.t-bold span').text]
    # 等待学位元素文本非空
    degree_text = WebDriverWait(education, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, 'span.t-14.t-normal span'))
    ).text.strip()
    ed['Degree'] += [degree_text]

方案3：添加异常处理，避免循环中断 #

捕获元素定位失败的异常，对无学位的条目填充默认值：

from selenium.common.exceptions import NoSuchElementException

for education in edu:
    ed['College'] += [education.find_element(By.CSS_SELECTOR, 'span.mr1.hoverable-link-text.t-bold span').text]
    try:
        degree_elem = education.find_element(By.CSS_SELECTOR, 'span.t-14.t-normal span')
        ed['Degree'] += [degree_elem.text.strip()]
    except NoSuchElementException:
        ed['Degree'] += [None]

内容的提问来源于stack exchange，提问作者Jesper Ezra