Python/BeautifulSoup批量爬取Yahoo Finance公司所有者姓名求助

问题描述 #

目标：使用Python/BeautifulSoup爬取Yahoo Finance上公开公司所有者的姓名。

以下单URL爬取代码可正常运行：

from bs4 import BeautifulSoup
import requests

url = 'https://finance.yahoo.com/quote/GTVI/profile?p=GTVI'
page = requests.get(url, headers={
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
})
soup = BeautifulSoup(page.text, 'html.parser')
price = soup.find_all("tr", {"class": "C($primaryColor) BdB Bdc($seperatorColor) H(36px)"})
print(soup.select_one("td > span").text)

但无法实现批量循环爬取多个URL，同时保持浏览器User-Agent伪装。本人为Python新手，以下是尝试的批量代码，请求帮助：

from bs4 import BeautifulSoup
import requests

url = ['https://finance.yahoo.com/quote/GTVI/profile?p=GTVI',
'https://finance.yahoo.com/quote/RAFA/profile?p=RAFA',
'https://finance.yahoo.com/quote/CYDX/profile?p=CYDX',
'https://finance.yahoo.com/quote/TTHG/profile?p=TTHG']
names = []
for link in url:
    w=1
    reqs2 = requests.get(link)
page = requests.get(url, headers={
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
})
soup = BeautifulSoup(page.text, 'html.parser')
for x in soup.find_all("tr", {"class": "C($primaryColor) BdB Bdc($seperatorColor) H(36px)"})
names.append(x.text)
print(names)(soup.select_one("td > span").text)

问题修正与可运行代码 #

原批量代码存在多个错误：循环内请求未带User-Agent、错误将URL列表传入requests.get()、循环语法缺少冒号、最后打印语句无效。

下面是修正后的完整可运行代码：

from bs4 import BeautifulSoup
import requests

# 待爬取的URL列表
url_list = [
    'https://finance.yahoo.com/quote/GTVI/profile?p=GTVI',
    'https://finance.yahoo.com/quote/RAFA/profile?p=RAFA',
    'https://finance.yahoo.com/quote/CYDX/profile?p=CYDX',
    'https://finance.yahoo.com/quote/TTHG/profile?p=TTHG'
]
owner_names = []
# 统一设置User-Agent，避免被反爬拦截
headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
}

# 逐个处理每个URL
for url in url_list:
    try:
        # 发送带User-Agent的请求
        page = requests.get(url, headers=headers)
        # 检查请求是否成功
        page.raise_for_status()
        # 解析页面
        soup = BeautifulSoup(page.text, 'html.parser')
        # 提取所有者姓名，和单URL代码选择器一致
        name = soup.select_one("td > span").text
        owner_names.append(name)
        print(f"{url} 爬取成功: {name}")
    except Exception as e:
        print(f"{url} 爬取失败: {str(e)}")

# 输出所有结果
print("\n所有公司所有者姓名:")
for idx, name in enumerate(owner_names, 1):
    print(f"{idx}. {name}")

核心要点说明 #

• 单独定义User-Agent变量，确保每个请求都带上，防止被网站识别为爬虫拦截
• 用try-except捕获异常，单个URL爬取失败不会导致整个程序终止
• 遍历URL列表时，对每个URL单独发起请求、解析、提取数据，再存入结果列表
• page.raise_for_status()会在请求返回错误状态码（如404、500）时抛出异常，方便排查问题

内容的提问来源于stack exchange，提问作者MattHagens.com