Glassdoor企业评论爬虫问题：子评分不全时全部显示N.A.

我需要爬取Glassdoor平台上某企业的全部评论，为此修改了@Driftr95的代码：

def extract(pg): 
    headers = {'user-agent' : 'Mozilla/5.0'}
    url = f'https://www.glassdoor.com/Reviews/3M-Reviews-E446_P{pg}.htm?filter.iso3Language=eng'
    # f'https://www.glassdoor.com/Reviews/Google-Engineering-Reviews-EI_IE9079.0,6_DEPT1007_IP{pg}.htm?sort.sortType=RD&sort.ascending=false&filter.iso3Language=eng'

    r = requests.get(url, headers)
    soup = BeautifulSoup(r.content, 'html.parser')# this a soup function that retuen the whole html
    return soup

for j in range(1,21,10):
    for i in range(j+1,j+11,1): #3M: 4251 reviews
        soup = extract( f'https://www.glassdoor.com/Reviews/3M-Reviews-E446_P{i}.htm?filter.iso3Language=eng')
        print(f' page {i}')
        for r in soup.select('li[id^="empReview_"]'):
            rDet = {'reviewId': r.get('id')}
            for sr in r.select(subRatSel):
                k = sr.select_one('div:first-of-type').get_text(' ').strip()
                sval = getDECstars(sr.select_one('div:nth-of-type(2)'), soup)
                rDet[f'[rating] {k}'] = sval

            for k, sel in refDict.items():
                sval = r.select_one(sel)
                if sval: sval = sval.get_text(' ').strip()
                rDet[k] = sval

            empRevs.append(rDet) 

问题 #

当评论中并非所有子评分都存在时，四个子评分的结果都会显示为N.A.，需要解决这个问题。

解决方法 #

问题根源在于：现有代码仅对存在的子评分赋值，未初始化缺失的子评分键；同时getDECstars函数可能在处理空元素时导致异常或统一返回N.A.，进而影响所有子评分的显示。以下是修正步骤：

1. 初始化所有子评分的默认值

先定义所有可能的子评分名称，在创建rDet时先将这些子评分的键设为N.A.，后续仅覆盖存在的子评分值：

# 定义Glassdoor常见的四个子评分名称
SUB_RATING_NAMES = [
    'Work/Life Balance',
    'Culture & Values',
    'Career Opportunities',
    'Compensation & Benefits'
]

# 在处理单条评论时，先初始化子评分默认值
rDet = {'reviewId': r.get('id')}
for name in SUB_RATING_NAMES:
    rDet[f'[rating] {name}'] = 'N.A.'

2. 修正子评分遍历逻辑

遍历页面中实际存在的子评分，仅对存在的子评分更新对应的值，避免缺失的子评分影响其他项：

for sr in r.select(subRatSel):
    k = sr.select_one('div:first-of-type').get_text(' ').strip()
    star_elem = sr.select_one('div:nth-of-type(2)')
    sval = getDECstars(star_elem, soup)
    # 仅当获取到有效星级时才覆盖默认值
    if sval and sval != 'N.A.':
        rDet[f'[rating] {k}'] = sval

3. 完善getDECstars函数

确保函数在传入空元素时返回N.A.，而非抛出异常或导致全局错误：

def getDECstars(elem, soup):
    if not elem:
        return 'N.A.'
    # 示例：从aria-label属性提取星级（根据实际页面结构调整）
    star_label = elem.get('aria-label')
    if star_label:
        # 假设aria-label格式为"4.0 stars"，提取数字部分
        return star_label.split()[0]
    # 备用：从页面其他元素提取星级（如span的文本）
    star_text = elem.get_text(strip=True)
    if star_text.replace('.', '').isdigit():
        return star_text
    return 'N.A.'

4. 修正extract函数的冗余调用

原代码中extract函数已经接收pg参数并生成URL，但循环中又直接传入完整URL，导致函数内部的URL生成逻辑失效，需修正：

# 循环中调用extract时，直接传入页码i即可
soup = extract(i)

修正后的完整代码 #

import requests
from bs4 import BeautifulSoup

def getDECstars(elem, soup):
    if not elem:
        return 'N.A.'
    star_label = elem.get('aria-label')
    if star_label:
        return star_label.split()[0]
    star_text = elem.get_text(strip=True)
    if star_text.replace('.', '').isdigit():
        return star_text
    return 'N.A.'

def extract(pg): 
    headers = {'user-agent' : 'Mozilla/5.0'}
    url = f'https://www.glassdoor.com/Reviews/3M-Reviews-E446_P{pg}.htm?filter.iso3Language=eng'
    r = requests.get(url, headers)
    soup = BeautifulSoup(r.content, 'html.parser')
    return soup

# 假设subRatSel和refDict的定义（需根据实际页面结构调整）
subRatSel = '.subRatings .subRating'
refDict = {
    'title': '.reviewTitle',
    'pros': '.pros',
    'cons': '.cons',
    'date': '.reviewDate'
}

empRevs = []
SUB_RATING_NAMES = [
    'Work/Life Balance',
    'Culture & Values',
    'Career Opportunities',
    'Compensation & Benefits'
]

for j in range(1,21,10):
    for i in range(j+1,j+11,1): #3M: 4251 reviews
        soup = extract(i)
        print(f' page {i}')
        for r in soup.select('li[id^="empReview_"]'):
            rDet = {'reviewId': r.get('id')}
            # 初始化子评分默认值
            for name in SUB_RATING_NAMES:
                rDet[f'[rating] {name}'] = 'N.A.'
            # 更新存在的子评分
            for sr in r.select(subRatSel):
                k = sr.select_one('div:first-of-type').get_text(' ').strip()
                star_elem = sr.select_one('div:nth-of-type(2)')
                sval = getDECstars(star_elem, soup)
                if sval and sval != 'N.A.':
                    rDet[f'[rating] {k}'] = sval
            # 提取其他评论信息
            for k, sel in refDict.items():
                sval = r.select_one(sel)
                if sval:
                    sval = sval.get_text(' ').strip()
                rDet[k] = sval if sval else 'N.A.'
            empRevs.append(rDet)

内容的提问来源于stack exchange，提问作者Jaevapple