如何用Python从公司官网动态获取领导团队姓名与职位？

核心思路 #

因为公司名称运行时才确定，且网站结构差异大，整体流程分为定位官网→识别领导板块→提取目标信息三个核心步骤，以下是具体实现方法：

1. 动态获取公司官网URL #

基于输入的公司名称，从搜索引擎结果中定位官方网站（需注意反爬策略）：

import requests
from bs4 import BeautifulSoup

def get_company_website(company_name):
    search_url = f"https://www.google.com/search?q={company_name}+official+website"
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
    }
    response = requests.get(search_url, headers=headers)
    soup = BeautifulSoup(response.text, "html.parser")
    
    # 提取第一个匹配的官网链接
    for result in soup.find_all("div", class_="yuRUbf"):
        link = result.find("a")["href"]
        if company_name.lower() in link.lower() or "www." in link:
            return link
    return None

2. 自动识别领导团队板块 #

遍历官网导航栏或页面内的链接，匹配关键词定位目标页面：

def find_leadership_page(base_url):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
    }
    response = requests.get(base_url, headers=headers)
    soup = BeautifulSoup(response.text, "html.parser")
    
    # 定义匹配关键词（大小写不敏感）
    target_keywords = ["leadership", "board", "executive", "team", "management", "governance"]
    leadership_links = []
    
    # 遍历所有链接，拼接完整URL并匹配关键词
    for a_tag in soup.find_all("a", href=True):
        link_text = a_tag.get_text(strip=True).lower()
        link_href = a_tag["href"]
        
        if not link_href.startswith("http"):
            link_href = f"{base_url.rstrip('/')}/{link_href.lstrip('/')}"
        
        if any(keyword in link_text for keyword in target_keywords):
            leadership_links.append(link_href)
    
    return leadership_links[0] if leadership_links else None

3. 提取领导姓名与职位信息 #

针对不同页面结构，采用通用标签匹配逻辑提取信息（动态渲染页面需改用Selenium/Playwright）：

def extract_leadership_info(leadership_url):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
    }
    response = requests.get(leadership_url, headers=headers)
    soup = BeautifulSoup(response.text, "html.parser")
    
    leadership_team = []
    # 优先匹配常见的领导信息容器
    containers = soup.find_all(["div", "article"], class_=["executive", "team-member", "board-member", "leadership-item"])
    
    if containers:
        for container in containers:
            name = container.find(["h3", "h4"]).get_text(strip=True) if container.find(["h3", "h4"]) else "N/A"
            position = container.find("p").get_text(strip=True) if container.find("p") else "N/A"
            leadership_team.append({"name": name, "position": position})
    else:
        # 无特定容器时，直接基于标题+相邻段落提取
        for name_tag in soup.find_all(["h3", "h4"]):
            name = name_tag.get_text(strip=True)
            position_tag = name_tag.find_next_sibling("p")
            if position_tag:
                position = position_tag.get_text(strip=True)
                leadership_team.append({"name": name, "position": position})
    
    # 去重避免重复条目
    return list({item["name"]: item for item in leadership_team}.values())

4. 整合流程调用 #

def main(company_name):
    website = get_company_website(company_name)
    if not website:
        print("无法找到公司官网")
        return
    
    leadership_page = find_leadership_page(website)
    if not leadership_page:
        print("无法找到领导团队板块")
        return
    
    team_info = extract_leadership_info(leadership_page)
    print("领导团队信息：")
    for member in team_info:
        print(f"- {member['name']}: {member['position']}")

# 示例调用
main("Apple Inc.")

注意事项 #

• 反爬处理：务必添加合理的User-Agent，可加入随机延迟（time.sleep(random.uniform(1,3))），避免频繁请求被封禁。
• 动态页面兼容：若目标页面是JS动态渲染，需替换requests为selenium或playwright，模拟浏览器行为获取完整内容。
• 结果校验：由于网站结构差异大，可根据实际情况调整关键词和标签匹配规则，提升提取准确率。

内容的提问来源于stack exchange，提问作者sugupta0