如何用BeautifulSoup4提取div块内所有超链接的文本内容？

问题 #

使用以下代码获取并解析页面：

from bs4 import BeautifulSoup
import requests

url = 'https://www.example.com/something'

result = requests.get(url).content
#print(result)
soup = BeautifulSoup(result,"lxml")
result = soup.find_all("div", class_="header-subtitle")
print(result)

返回结果：

[<div class="header-subtitle"><a href="https://example/cat1" title="cat1">Cat1</a> <span>/</span> <a href="https://www.example.com/cat2" title="cat2">Cat2</a> <span>/</span> <a href="https://www.example.com/cat3" title="cat3">Cat3</a>
</div>]

需要提取其中的Cat1、Cat2、Cat3及后续更多超链接文本，但使用以下代码仅能获取Cat1：

for div in result:
    print(div.find('a').contents)

尝试正则提取也未成功，求解决方法。

解决方法 #

方法一：遍历所有<a>标签 #

find('a')仅会返回第一个匹配的链接标签，要获取全部链接，改用find_all('a')遍历：

from bs4 import BeautifulSoup
import requests

url = 'https://www.example.com/something'

result = requests.get(url).content
soup = BeautifulSoup(result,"lxml")
divs = soup.find_all("div", class_="header-subtitle")

for div in divs:
    links = div.find_all('a')
    for link in links:
        # 获取并清理链接文本
        print(link.get_text(strip=True))

方法二：用CSS选择器直接定位目标链接 #

通过CSS选择器直接选中.header-subtitle下的所有<a>标签，代码更简洁：

from bs4 import BeautifulSoup
import requests

url = 'https://www.example.com/something'

result = requests.get(url).content
soup = BeautifulSoup(result,"lxml")
links = soup.select('.header-subtitle a')

for link in links:
    print(link.get_text(strip=True))

方法三：文本分割提取（仅作补充） #

如果分隔符/稳定，也可以先提取div的全部文本再分割，但这种方式依赖页面结构不变，不如标签定位可靠：

from bs4 import BeautifulSoup
import requests

url = 'https://www.example.com/something'

result = requests.get(url).content
soup = BeautifulSoup(result,"lxml")
divs = soup.find_all("div", class_="header-subtitle")

for div in divs:
    cats = [item.strip() for item in div.get_text().split('/')]
    print(cats)

内容的提问来源于stack exchange，提问作者liuru