Glassdoor分类评分爬取失败问题求助

爬取Glassdoor评论时，基础数据（综合评分、优缺点、日期、职位名称、员工类型）均可正常获取，但分类评分始终无法爬取，执行时触发NoSuchElementException元素定位失败错误。

现有代码 #

分类评分提取函数 #

def extract_star_rating(review, category_name):
    xpath = f'//span[text()="{category_name}"]/ancestor::div[@class="common__EIReviewsRatingsStyles__RatingItemWrapper-sc-1dl5e6p-3 gdGrid"]//div[@class]'
    category_div = review.find_element(By.XPATH, xpath)
    class_name = category_div.get_attribute('class')
    if 'css-1mfncox' in class_name:
        return 1
    elif 'css-1lp3h8x' in class_name:
        return 2
    elif 'css-k58126' in class_name:
        return 3
    elif 'css-94nhxw' in class_name:
        return 4
    else:
        return 5

调用逻辑 #

# 遍历所有页面
for i in range(1, 3697):
    # 访问目标页面
    page_url = f"{url[:-4]}_P{i}.htm"
    driver.get(page_url)
    # 获取页面所有评论元素
    review_elements = driver.find_elements(by=By.XPATH, value="//div[@class='gdReview']")
    # 逐个提取评论数据
    for element in review_elements:
        review = {}
        review['Work/Life Balance'] = extract_star_rating(element, 'Work/Life Balance')
        review['Culture & Values'] = extract_star_rating(element, 'Culture & Values')
        review['Diversity & Inclusion'] = extract_star_rating(element, 'Diversity & Inclusion')
        review['Career Opportunities'] = extract_star_rating(element, 'Career Opportunities')
        review['Compensation and Benefits'] = extract_star_rating(element, 'Compensation and Benefits')
        review['Senior Management'] = extract_star_rating(element, 'Senior Management')
        reviews.append(review)

报错信息 #

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//span[text()="Work/Life Balance"]/ancestor::div[@class="common__EIReviewsRatingsStyles__RatingItemWrapper-sc-1dl5e6p-3 gdGrid"]//div[@

1. 修复XPATH核心问题 #

你的XPATH存在两个致命错误：

• 错误使用"转义引号：Python的f-string中无需转义，直接用双引号包裹内部字符串即可，否则会把"当成字符串的一部分，导致匹配不到元素。
• 未使用相对路径：XPATH开头加.表示在当前review子元素下查找，否则会从整个页面根节点搜索，这是定位失败的核心原因。

修正后的XPATH：

xpath = f'.//span[text()="{category_name}"]/ancestor::div[@class="common__EIReviewsRatingsStyles__RatingItemWrapper-sc-1dl5e6p-3 gdGrid"]//div[@class]'

2. 捕获元素不存在的异常 #

部分评论可能存在用户未填写的分类评分，直接查找会触发异常，需要捕获NoSuchElementException并返回默认值：

from selenium.common.exceptions import NoSuchElementException

def extract_star_rating(review, category_name):
    try:
        xpath = f'.//span[text()="{category_name}"]/ancestor::div[@class="common__EIReviewsRatingsStyles__RatingItemWrapper-sc-1dl5e6p-3 gdGrid"]//div[@class]'
        category_div = review.find_element(By.XPATH, xpath)
        class_name = category_div.get_attribute('class')
        if 'css-1mfncox' in class_name:
            return 1
        elif 'css-1lp3h8x' in class_name:
            return 2
        elif 'css-k58126' in class_name:
            return 3
        elif 'css-94nhxw' in class_name:
            return 4
        else:
            return 5
    except NoSuchElementException:
        return None  # 可根据需求改为0或其他默认值

3. 替换易变的CSS类名定位（可选） #

Glassdoor的动态CSS类名（如css-1mfncox）随时可能变更，建议改用语义化的aria-label属性提取星级，稳定性更高：

def extract_star_rating(review, category_name):
    try:
        star_element = review.find_element(By.XPATH, f'.//span[text()="{category_name}"]/following-sibling::div//span[@aria-label]')
        aria_label = star_element.get_attribute('aria-label')
        return int(aria_label.split()[0])  # 从"1 out of 5 stars"提取数字1
    except NoSuchElementException:
        return None

4. 增加页面加载等待 #

页面渲染延迟也可能导致元素未加载完成就被查找，添加显式等待确保元素就绪：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

for i in range(1, 3697):
    page_url = f"{url[:-4]}_P{i}.htm"
    driver.get(page_url)
    # 等待评论元素加载完成，最长等待10秒
    WebDriverWait(driver, 10).until(
        EC.presence_of_all_elements_located((By.XPATH, "//div[@class='gdReview']"))
    )
    review_elements = driver.find_elements(By.XPATH, "//div[@class='gdReview']")
    # 后续提取逻辑不变...

内容的提问来源于stack exchange，提问作者userrr