如何使用BeautifulSoup从同class的div中提取指定数据

我需要从Cagematch的选手页面（示例地址：https://www.cagematch.net/?id=2&nr=448&gimmick=Adam+Pearce）提取Current Gimmick（当前角色）、**Age（年龄）和Brand（品牌）**这三个信息。页面中所有数据都存放在class相同的div标签里，我尝试了以下代码，但会获取到页面所有数据，有没有简便方法只提取需要的特定信息？

url = "http://www.cagematch.net/?id=8&nr=1&page=15"
headers = {"Accept-Encoding": "deflate"}
soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")

links = [
    "https://www.cagematch.net/" + a["href"] for a in soup.select(".TCol a")
]
list = []
for u in links:
    soup = BeautifulSoup(
        requests.get(u, headers=headers).content, "html.parser"
    )
    # print(soup.h1.text)
    with open("wwe/maw.csv", 'a', encoding="utf-8", newline="") as f:
        wrt = writer(f)
        for info in soup.select_all(".InformationBoxRow"):
            # header1 = info.select_one(".InformationBoxTitle").text
            # header2 = info.select_one(".InformationBoxContents").text
            # table = [header1, header2, "/"]

            header1 = info.select("div", class_="InformationBoxContents").text
            print(header1)

解决方案 #

核心思路是定位到包含目标字段的行，只提取对应内容，无需遍历所有信息行。以下是优化后的代码：

import requests
from bs4 import BeautifulSoup
from csv import writer

url = "http://www.cagematch.net/?id=8&nr=1&page=15"
headers = {"Accept-Encoding": "deflate"}
soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")

# 修正链接拼接逻辑，避免重复添加域名
links = []
for a in soup.select(".TCol a"):
    href = a["href"]
    links.append(href if href.startswith("http") else f"https://www.cagematch.net/{href}")

# 先写入CSV表头，让数据结构更清晰
with open("wwe/maw.csv", 'w', encoding="utf-8", newline="") as f:
    writer(f).writerow(["选手姓名", "当前角色", "年龄", "所属品牌"])

# 遍历每个选手页面提取信息
for u in links:
    page_soup = BeautifulSoup(requests.get(u, headers=headers).content, "html.parser")
    wrestler_name = page_soup.h1.text.strip() if page_soup.h1 else "未知姓名"
    
    # 初始化目标字段默认值
    current_gimmick = ""
    age = ""
    brand = ""
    
    # 只提取需要的字段
    for row in page_soup.select_all(".InformationBoxRow"):
        title = row.select_one(".InformationBoxTitle").text.strip()
        content = row.select_one(".InformationBoxContents").text.strip()
        
        if title == "Current Gimmick":
            current_gimmick = content
        elif title == "Age":
            age = content
        elif title == "Brand":
            brand = content
    
    # 将数据追加到CSV
    with open("wwe/maw.csv", 'a', encoding="utf-8", newline="") as f:
        writer(f).writerow([wrestler_name, current_gimmick, age, brand])

优化说明 #

1. 链接拼接修正：避免重复添加域名，兼容完整路径和相对路径的a标签href
2. CSV表头初始化：提前写入表头，让导出的数据结构更清晰
3. 精准字段匹配：遍历信息行时，只匹配目标标题对应的内容，跳过无关数据
4. 默认值处理：为目标字段设置空字符串默认值，避免缺失数据时出现报错
5. 代码可读性优化：变量命名更清晰，拆分逻辑步骤

更简洁的定位方式（可选） #

如果你的BeautifulSoup版本支持:-soup-contains选择器，可以直接通过标题文本定位，省去遍历循环：

# 直接提取当前角色
current_gimmick_elem = page_soup.select_one('.InformationBoxRow:has(.InformationBoxTitle:-soup-contains("Current Gimmick")) .InformationBoxContents')
current_gimmick = current_gimmick_elem.text.strip() if current_gimmick_elem else ""

# 同理提取年龄和品牌
age_elem = page_soup.select_one('.InformationBoxRow:has(.InformationBoxTitle:-soup-contains("Age")) .InformationBoxContents')
age = age_elem.text.strip() if age_elem else ""

brand_elem = page_soup.select_one('.InformationBoxRow:has(.InformationBoxTitle:-soup-contains("Brand")) .InformationBoxContents')
brand = brand_elem.text.strip() if brand_elem else ""

内容的提问来源于stack exchange，提问作者user19353374