使用BeautifulSoup提取类下深层嵌套无类名span标签值的方法

场景说明 #

遍历页面房源条目时，需提取class为bill_of_sale和mortgage的div节点下，未设置class属性的span标签存储的文本，目标值分别为契据状态Kupça var、抵押状态İpoteka var。
目标HTML结构片段：

<div class="preview">
    <div class="item_slider" data-swiper-wrap="" style="touch-action: pan-y; user-select: none; -webkit-user-drag: none; -webkit-tap-highlight-color: rgba(0, 0, 0, 0);">
    <div class="bill_of_sale"><a class="item_link" target="_blank" href="/items/2810476"></a><span>Kupça var</span></div>
    <div class="mortgage"><a class="item_link" target="_blank" href="/items/2810476"></a><span>İpoteka var</span></div>
    <div class="products-paid"><span class="featured-icon"></span><span class="vipped-icon"></span></div>
    <div class="products-label">Agentlik</div>
</div>

现有代码可正常提取文本直接挂载在目标节点上的内容（比如class为location的div下的地址文本），代码如下：

page = 1
locations=[] # 存储地址数据

while page != 1200:
    url = f"https://bina.az/baki/alqi-satqi/menziller?page={page}"
    page_main = requests.get(url)
    soup = BeautifulSoup(page_main.content, "html.parser")

    results = soup.find(id="js-items-search")
    job_elements = results.find_all("div", class_="card_params")

    for job_element in job_elements:
        location = job_element.find(class_="location")
        locations.append(location.text)
    
    page = page + 1

现有逻辑无法提取嵌套在目标div下层的span文本，需调整提取规则。

实现代码 #

定位到目标class的div节点后，二次筛选节点内未设置class属性的span标签，即可自动过滤掉带class的无关span（如示例里的featured-icon、vipped-icon），同时增加非空判断避免字段缺失导致程序中断。修改后完整代码：

import requests
from bs4 import BeautifulSoup

page = 1
locations = []
bill_of_sale_res = [] # 存储契据状态
mortgage_res = [] # 存储抵押状态

while page != 1200:
    url = f"https://bina.az/baki/alqi-satqi/menziller?page={page}"
    page_main = requests.get(url)
    soup = BeautifulSoup(page_main.content, "html.parser")

    results = soup.find(id="js-items-search")
    job_elements = results.find_all("div", class_="card_params")

    for job_element in job_elements:
        # 原有地址提取逻辑，增加非空判断
        location = job_element.find(class_="location")
        if location:
            locations.append(location.text.strip())
        
        # 提取bill_of_sale字段
        bill_node = job_element.find(class_="bill_of_sale")
        if bill_node:
            target_span = bill_node.find("span", class_=False)
            if target_span:
                bill_of_sale_res.append(target_span.text.strip())
        
        # 提取mortgage字段
        mortgage_node = job_element.find(class_="mortgage")
        if mortgage_node:
            target_span = mortgage_node.find("span", class_=False)
            if target_span:
                mortgage_res.append(target_span.text.strip())
    
    page = page + 1

关键逻辑说明 #

• find("span", class_=False)：BeautifulSoup原生筛选语法，匹配所有未设置class属性的span标签，精准命中目标节点，不会误选同层级带class的其他span
• .text.strip()：清除提取文本前后的换行、空格等无效字符，保证数据干净
• 多层非空判断：适配部分房源缺失对应字段的场景，避免出现NoneType属性调用报错导致爬虫中断

内容的提问来源于stack exchange，提问作者Farid Mammadaliyev