如何用requests+BeautifulSoup抓取详情页数据并追加到原有列表

原有代码核心问题 #

• 未对详情页发起独立请求，直接复用列表页的解析对象无法获取详情页数据
• 语法错误：for ls in lslinks 后缺失冒号
• 选择器规则错误：BeautifulSoup中仅class属性需要加下划线写为class_，id属性直接用id参数即可；且目标内容存放在tr标签下的第二个td节点内，直接取tr的文本会包含冗余字段名
• 变量逻辑错误：单个详情链接字符串被直接赋值给lslinks进行遍历，会被拆分为单个字符循环，完全不符合业务逻辑
• 缺失异常捕获：部分企业详情页可能缺少电话、CNAE等字段，直接取值会触发程序中断

完整可运行代码 #

import requests
import csv
from bs4 import BeautifulSoup

# 配置请求头避免被反爬拦截
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}

# 生成所有列表页链接
baseurl = ["https://www.expansion.com/empresas-de/ganaderia/granjas-en-general/index.html"]
urls = [f'https://www.expansion.com/empresas-de/ganaderia/granjas-en-general/{i}.html' for i in range(2,65)]
allurls = baseurl + urls

# 初始化csv文件，直接生成可导入Excel的格式
csv_header = ["企业名称", "所在地", "省份", "详情链接", "地址", "CNAE编码", "电话号码"]
with open("企业数据.csv", "w", newline="", encoding="utf-8-sig") as f:
    writer = csv.writer(f)
    writer.writerow(csv_header)

for url in allurls:
    page = requests.get(url, headers=headers, timeout=10)
    soup = BeautifulSoup(page.content, "html.parser")
    lists = soup.select("div#simulacion_tabla ul")

    # 抓取列表页基础信息
    for lis in lists:
        title = lis.find('li', class_="col1").text.strip()
        location = lis.find('li', class_="col2").text.strip()
        province = lis.find('li', class_="col3").text.strip()
        link = lis.select_one("li.col1 a")['href'].strip()
        info = [title, location, province, link]

        # 发起详情页请求提取额外字段
        try:
            detail_page = requests.get(link, headers=headers, timeout=10)
            detail_soup = BeautifulSoup(detail_page.content, "html.parser")

            # 字段不存在则默认填空，避免程序中断
            address = ""
            addr_tr = detail_soup.find('tr', id="tamano_empresa")
            if addr_tr and len(addr_tr.find_all('td'))>=2:
                address = addr_tr.find_all('td')[1].text.strip()

            cnae = ""
            cnae_tr = detail_soup.find('tr', id="cnae_codigo_empresa")
            if cnae_tr and len(cnae_tr.find_all('td'))>=2:
                cnae = cnae_tr.find_all('td')[1].text.strip()

            phone = ""
            phone_tr = detail_soup.find('tr', id="telefono_empresa")
            if phone_tr and len(phone_tr.find_all('td'))>=2:
                phone = phone_tr.find_all('td')[1].text.strip()

            # 追加额外字段并输出
            info.extend([address, cnae, phone])
            print(info)

            # 逐行写入csv
            with open("企业数据.csv", "a", newline="", encoding="utf-8-sig") as f:
                writer = csv.writer(f)
                writer.writerow(info)
        except:
            # 抓取失败的记录保留基础信息，额外字段留空
            info.extend(["", "", ""])
            continue

输出说明 #

• 运行代码后控制台会直接打印你要求格式的列表，示例：['AGRICOLA CALLEJA SL', 'CARPIO', 'VALLADOLID', 'https://www.expansion.com/directorio-empresas/agricola-calleja-sl_1480101_A02_47.html', 'C/ LA TORRE, 2.', '150', '983863247']
• 生成的企业数据.csv采用utf-8-sig编码，可直接用Excel打开无需转码，完全适配导入需求。

内容的提问来源于stack exchange，提问作者Nick Gordon