如何通过文本'Service request'用BeautifulSoup提取<div>内<a>的href

需求：通过文本“Service request”定位对应的<div>，提取其中<a>标签的href属性值，目标URL为：

https://-2B06XEJ0r8DVmc_kuX2cI8baDAaOcj-2Fp3iIjU6R7PXKa3dAYAr0B7iMyKwz-2FaV0nnIuCVP1pcf8DEy1UidQbR2IywCV5ueXy1TowXMzFcIPYG2hp7HjP1WzHYI-2FJNMGLZMtC4LXybcCZ4cUOV4DnC6s-2FCIJ-2FrumGmdnE2leBJgM3rWJaEyXOwi4JiHjBHr4rtNh-2BPeP3JFBHpGNp5KWrZkxg-2F9zrih4tp7-2BUUrBo0g8hlG5It1yEVfz9Im2iRVAvdjqHvAqxn63TsV2OFNp8M8DMjuS6aRL3Vki8HfkXx0kD8fGJ6GAKUAOZv-2BCSAgxtcdnIpR8sQU6Jkcm9vxdjG2zmDYFEmVykg0-2BY3uD1ZbGl79dsB68mqJdbbQlgb8ERSRAW3t8cYTXsegrGd5-2Fox-2B9Yo-2FPmUC9cnqEaA-3D-3D

原尝试代码 #

from bs4 import BeautifulSoup
import re
text = """<div class="content">
   <div class="body">
    <img src="https:/document-images/leasing/mynd-logo.png" alt="j0b" width="96px" height="auto" class="kjhjhv">
    <div class="title">
      You've received a new message.
     <div class="subtitle">
      Service request <a href="https://-2B06XEJ0r8DVmc_kuX2cI8baDAaOcj-2Fp3iIjU6R7PXKa3dAYAr0B7iMyKwz-2FaV0nnIuCVP1pcf8DEy1UidQbR2IywCV5ueXy1TowXMzFcIPYG2hp7HjP1WzHYI-2FJNMGLZMtC4LXybcCZ4cUOV4DnC6s-2FCIJ-2FrumGmdnE2leBJgM3rWJaEyXOwi4JiHjBHr4rtNh-2BPeP3JFBHpGNp5KWrZkxg-2F9zrih4tp7-2BUUrBo0g8hlG5It1yEVfz9Im2iRVAvdjqHvAqxn63TsV2OFNp8M8DMjuS6aRL3Vki8HfkXx0kD8fGJ6GAKUAOZv-2BCSAgxtcdnIpR8sQU6Jkcm9vxdjG2zmDYFEmVykg0-2BY3uD1ZbGl79dsB68mqJdbbQlgb8ERSRAW3t8cYTXsegrGd5-2Fox-2B9Yo-2FPmUC9cnqEaA-3D-3D" style="color:#5A88AA; text-decoration:underline;" target="_blank">#2819138</a>
     </div>
    </div> <!-- CONTENT OF EMAIL -->
    <p> May 17, 2023 05:33 PDT <b> Charisse </b> wrote: </p>
    <p style="white-space: pre-wrap;">Hi team. Is this completed? Please advise. Thank you, Charisse</p>
   </div>"""
soup = BeautifulSoup(text,"html.parser")
scores = soup.find_all(text=re.compile('Service request'))
print(scores)
score2 = soup.find('div',text=re.compile('Service request'))
print(score2)

问题分析 #

原代码存在两个核心问题：

• find_all(text=re.compile('Service request'))仅能匹配到孤立的文本节点，无法直接关联到包含该文本的div元素
• find('div', text=re.compile('Service request'))无法匹配目标div，因为该div内除了“Service request”文本，还包含<a>标签，text参数要求完全匹配div的全部文本内容，因此匹配失败

解决方案 #

先定位到“Service request”文本节点，再获取其父div，最后提取该div内a标签的href属性：

from bs4 import BeautifulSoup
import re

text = """<div class="content">
   <div class="body">
    <img src="https:/document-images/leasing/mynd-logo.png" alt="j0b" width="96px" height="auto" class="kjhjhv">
    <div class="title">
      You've received a new message.
     <div class="subtitle">
      Service request <a href="https://-2B06XEJ0r8DVmc_kuX2cI8baDAaOcj-2Fp3iIjU6R7PXKa3dAYAr0B7iMyKwz-2FaV0nnIuCVP1pcf8DEy1UidQbR2IywCV5ueXy1TowXMzFcIPYG2hp7HjP1WzHYI-2FJNMGLZMtC4LXybcCZ4cUOV4DnC6s-2FCIJ-2FrumGmdnE2leBJgM3rWJaEyXOwi4JiHjBHr4rtNh-2BPeP3JFBHpGNp5KWrZkxg-2F9zrih4tp7-2BUUrBo0g8hlG5It1yEVfz9Im2iRVAvdjqHvAqxn63TsV2OFNp8M8DMjuS6aRL3Vki8HfkXx0kD8fGJ6GAKUAOZv-2BCSAgxtcdnIpR8sQU6Jkcm9vxdjG2zmDYFEmVykg0-2BY3uD1ZbGl79dsB68mqJdbbQlgb8ERSRAW3t8cYTXsegrGd5-2Fox-2B9Yo-2FPmUC9cnqEaA-3D-3D" style="color:#5A88AA; text-decoration:underline;" target="_blank">#2819138</a>
     </div>
    </div> <!-- CONTENT OF EMAIL -->
    <p> May 17, 2023 05:33 PDT <b> Charisse </b> wrote: </p>
    <p style="white-space: pre-wrap;">Hi team. Is this completed? Please advise. Thank you, Charisse</p>
   </div>"""

soup = BeautifulSoup(text, "html.parser")
# 找到包含目标文本的节点
target_text = soup.find(string=re.compile('Service request'))
# 获取该文本节点的父div
target_div = target_text.parent
# 提取div内a标签的href属性
target_url = target_div.find('a')['href']
print(target_url)

执行上述代码后，将输出目标URL：

https://-2B06XEJ0r8DVmc_kuX2cI8baDAaOcj-2Fp3iIjU6R7PXKa3dAYAr0B7iMyKwz-2FaV0nnIuCVP1pcf8DEy1UidQbR2IywCV5ueXy1TowXMzFcIPYG2hp7HjP1WzHYI-2FJNMGLZMtC4LXybcCZ4cUOV4DnC6s-2FCIJ-2FrumGmdnE2leBJgM3rWJaEyXOwi4JiHjBHr4rtNh-2BPeP3JFBHpGNp5KWrZkxg-2F9zrih4tp7-2BUUrBo0g8hlG5It1yEVfz9Im2iRVAvdjqHvAqxn63TsV2OFNp8M8DMjuS6aRL3Vki8HfkXx0kD8fGJ6GAKUAOZv-2BCSAgxtcdnIpR8sQU6Jkcm9vxdjG2zmDYFEmVykg0-2BY3uD1ZbGl79dsB68mqJdbbQlgb8ERSRAW3t8cYTXsegrGd5-2Fox-2B9Yo-2FPmUC9cnqEaA-3D-3D

内容的提问来源于stack exchange，提问作者Ben David