使用BeautifulSoup提取data-score值时DOM与输出不匹配问题求助

我需要提取一批评分的data-score值，该值位于class为index__stars__nfK6S的<span>标签中，且此<span>包含在class为index__block__7hodp的<div>内。我编写了嵌套for循环进行提取，但输出结果与DOM中的实际值不匹配——例如DOM中Work-Life-Balance的data-score为5，输出却为4。

内层循环代码：

for rating in rating_parents:
    label = rating.find('h4', class_='index__title__Rq0Po')
    test = rating.find_next('div', class_='index__block__7hodp')
    print(label.text)
    if test is not None:
        #data_score = test.get('data-score')
        data_score = test.find('span', class_='index__stars__nfK6S')
        print(data_score.get('data-score'))
    else:
        print('none')

完整循环代码：

for rating_article in rating_articles_1:
rating_parent_container = rating_article.find('div', class_='index__content__HTaY9')
if rating_parent_container is not None:
    rating_container = rating_parent_container.find('div', class_='index__reviewBlock__I8pdb')
    if rating_container is not None:
        rating_title = rating_container.find('h3')
        print(rating_title)
        rating_parents = rating_container.find_all('div', class_='index__factor__Mo6xW')
        for rating in rating_parents:
            label = rating.find('h4', class_='index__title__Rq0Po')
            test = rating.find_next('div', class_='index__block__7hodp')
            print(label.text)
            if test is not None:
                #data_score = test.get('data-score')
                data_score = test.find('span', class_='index__stars__nfK6S')
                print(data_score.get('data-score'))
            else:
                print('none')

问题根源 #

你使用的find_next('div', class_='index__block__7hodp')方法，是从当前rating节点之后的所有节点里查找第一个匹配元素，而非在当前rating节点（class为index__factor__Mo6xW的div）内部查找。一旦某个rating节点内部没有目标div，就会抓取到下一个rating节点里的评分，最终导致标签与评分错位、数值不匹配。

修正方案 #

把find_next替换成find，直接在当前rating节点内部查找目标div，确保每个标签对应自身节点内的评分值。

修正后的内层循环代码 #

for rating in rating_parents:
    label = rating.find('h4', class_='index__title__Rq0Po')
    # 用find替代find_next，限定在当前节点内部查找
    test = rating.find('div', class_='index__block__7hodp')
    print(label.text)
    if test is not None:
        data_score = test.find('span', class_='index__stars__nfK6S')
        print(data_score.get('data-score'))
    else:
        print('none')

修正后的完整循环代码 #

for rating_article in rating_articles_1:
    rating_parent_container = rating_article.find('div', class_='index__content__HTaY9')
    if rating_parent_container is not None:
        rating_container = rating_parent_container.find('div', class_='index__reviewBlock__I8pdb')
        if rating_container is not None:
            rating_title = rating_container.find('h3')
            print(rating_title)
            rating_parents = rating_container.find_all('div', class_='index__factor__Mo6xW')
            for rating in rating_parents:
                label = rating.find('h4', class_='index__title__Rq0Po')
                # 替换find_next为find，限定在当前节点内查找
                test = rating.find('div', class_='index__block__7hodp')
                print(label.text)
                if test is not None:
                    data_score = test.find('span', class_='index__stars__nfK6S')
                    print(data_score.get('data-score'))
                else:
                    print('none')

内容的提问来源于stack exchange，提问作者S.H