解析独立区块相同元素时数组输出重复条目问题排查

我写了一段解析器代码，想从网页中每个独立的<ul class="PhotoListSmall">区块里，提取区块内<li>标签下<a>元素的href属性。但代码运行结果不对，以下是相关信息：

解析器代码 #

import requests
from bs4 import BeautifulSoup as bs
import pandas as pd

URL_TEMPLATE = "http://127.0.0.1:5500/rr.html"
FILE_NAME = "img.csv"


def parse(url=URL_TEMPLATE):
    result_list = {'id': []}
    result_l = {'id': []}
    r = requests.get(url)
    soup = bs(r.text, "html.parser")
    vacancies_names = soup.find_all('ul', class_='PhotoListSmall')
    vacancies_li = soup.find_all('li')
    for name in vacancies_names:
        for i in vacancies_li:
            result_list['id'].append(i.a['href'])
        result_l['id'].append(result_list['id'])
        result_list['id'] = []

    return result_l


df = pd.DataFrame(data=parse())
df.to_csv(FILE_NAME)

待处理网页结构 #

<ul class="PhotoListSmall">
        <li class="one">
            <a href="one_1"></a>
        </li>
        <li class="two">
            <a href="one_2"></a>
        </li>
    </ul>
    <ul class="PhotoListSmall">
        <li class="one">
            <a href="two_1"></a>
        </li>
        <li class="two">
            <a href="two_2"></a>
        </li>
    </ul>

当前输出结果 #

,id
0,"['one_1', 'one_2', 'two_1', 'two_2']"
1,"['one_1', 'one_2', 'two_1', 'two_2']"

期望输出结果 #

,id
0,"['one_1', 'one_2']"
1,"['two_1', 'two_2']"

错误原因 #

核心错误是全局查找所有<li>元素，而非在当前遍历的<ul>区块内查找：

• 代码中vacancies_li = soup.find_all('li')会从整个HTML文档提取所有<li>，导致每次处理一个<ul>时，都会把所有<li>的href全收集一遍，最终每个<ul>对应的结果都是全量的href列表。

修复后的代码 #

import requests
from bs4 import BeautifulSoup as bs
import pandas as pd

URL_TEMPLATE = "http://127.0.0.1:5500/rr.html"
FILE_NAME = "img.csv"


def parse(url=URL_TEMPLATE):
    result_l = {'id': []}
    r = requests.get(url)
    soup = bs(r.text, "html.parser")
    # 找到所有目标ul区块
    vacancies_names = soup.find_all('ul', class_='PhotoListSmall')
    
    for ul_block in vacancies_names:
        current_hrefs = []
        # 仅在当前ul区块内查找li元素
        for li in ul_block.find_all('li'):
            current_hrefs.append(li.a['href'])
        result_l['id'].append(current_hrefs)

    return result_l


df = pd.DataFrame(data=parse())
df.to_csv(FILE_NAME)

关键修改说明 #

1. 移除全局的vacancies_li变量，改为在遍历每个<ul>时，用ul_block.find_all('li')仅从当前<ul>下提取<li>，确保每个区块只收集自己内部的href。
2. 简化临时变量逻辑，直接用current_hrefs存储当前区块的href列表，避免重复清空字典列表的操作，代码更易读。

内容的提问来源于stack exchange，提问作者Alex