使用BeautifulSoup提取面包屑文本时如何去除冗余的breadcrumb内容

看你的HTML结构，冗余的"breadcrumb"文本藏在visually-hidden类的div里，分隔符"/"在带aria-hidden="true"的span里。直接用get_text()会把这些子元素的内容也带出来，得针对性清理。

方法1：移除冗余元素后提取文本 #

先找到那些冗余的子元素删掉，再取文本：

from bs4 import BeautifulSoup
import requests 
page = requests.get("https://groceries.asda.com/product/grapes/asda-refreshing-sweet-seedless-red-grapes/1000076345132")
soup = BeautifulSoup(page.content, 'html.parser')
# 获取当前面包屑项（最后一个）
current_breadcrumb = soup.find('span', attrs={'class':'breadcrumb__current breadcrumb__current--bold','data-auto-id':'selectedBreadcrumb'}).get_text().strip()
# 获取所有面包屑链接
breadcrumb_links = soup.find_all('a', attrs={'class':'asda-link asda-link--primary breadcrumb__link','data-auto-id':'linkBreadcrumbName'})

for link in breadcrumb_links:
    # 删除带"breadcrumb"的隐藏div
    hidden_div = link.find('div', class_='visually-hidden')
    if hidden_div:
        hidden_div.decompose()
    # 删除分隔符span
    separator = link.find('span', attrs={'aria-hidden':'true'})
    if separator:
        separator.decompose()
    # 提取清理后的文本，去掉多余空格
    print(link.get_text().strip())
# 打印最后一个面包屑项
print(current_breadcrumb)

方法2：直接提取a标签内的纯文本节点 #

不用删除元素，直接筛选a标签下的原生文本内容，跳过子元素的文本：

from bs4 import BeautifulSoup
import requests 
page = requests.get("https://groceries.asda.com/product/grapes/asda-refreshing-sweet-seedless-red-grapes/1000076345132")
soup = BeautifulSoup(page.content, 'html.parser')
current_breadcrumb = soup.find('span', attrs={'class':'breadcrumb__current breadcrumb__current--bold','data-auto-id':'selectedBreadcrumb'}).get_text().strip()
breadcrumb_links = soup.find_all('a', attrs={'class':'asda-link asda-link--primary breadcrumb__link','data-auto-id':'linkBreadcrumbName'})

for link in breadcrumb_links:
    # 只取a标签下的字符串内容，过滤掉子元素的文本
    clean_text = ''.join([text.strip() for text in link.contents if isinstance(text, str) and text.strip()])
    print(clean_text)
print(current_breadcrumb)

两种方法都能得到干净的输出：

Fruit, Veg & Flowers
Fruit
Grapes
Grapes

内容的提问来源于stack exchange，提问作者ShacoPoggers