如何通过Python自动化执行curl命令实现多层级组织架构数据批量抓取

你可以直接用Python把全流程串起来实现自动化，具体操作如下：

步骤1：调整现有代码输出规范 #

先修正你之前提取CEO下属ID的代码，确保输出的CSV每行是一个经理ID，方便后续读取：

from bs4 import BeautifulSoup
import json
import pandas as pd

# 读取CEO页面HTML
with open("output/ceo.html", "r", encoding="utf-8") as f:
    contents = f.read()
soup = BeautifulSoup(contents, features="html.parser")

script = soup.find_all('script')[8].text.strip()[28:-1]
data = json.loads(script)
# 提取所有高级经理ID
manager_ids = [child['id'] for child in data['children']]
# 保存为每行一个ID的CSV，方便后续读取
pd.Series(manager_ids, name="manager_id").to_csv('manager_ids.csv', index=False, header=False)

步骤2：封装批量执行逻辑 #

直接在Python中调用curl、循环处理每个经理ID即可，不需要手动执行20次命令：

import os
import subprocess
import time
import pandas as pd
from bs4 import BeautifulSoup
import json

# 配置项
OUTPUT_HTML_DIR = "output/manager_html"
FINAL_RESULT_PATH = "all_subordinates.csv"
# 创建存储经理HTML的文件夹
os.makedirs(OUTPUT_HTML_DIR, exist_ok=True)

# 1. 读取所有高级经理ID
with open("manager_ids.csv", "r", encoding="utf-8") as f:
    manager_ids = [line.strip() for line in f if line.strip()]

all_subordinate_list = []

# 2. 循环处理每个经理ID
for manager_id in manager_ids:
    # 2.1 构造curl命令，替换URL中的CEO为当前经理ID
    html_save_path = os.path.join(OUTPUT_HTML_DIR, f"{manager_id}.html")
    curl_cmd = f'curl -ksS --negotiate --location-trusted -u: -c ~/cookie -b ~/cookie "https://webserver/users/{manager_id}/org" -o {html_save_path}'
    # 执行curl命令
    subprocess.run(curl_cmd, shell=True, check=True)
    
    # 2.2 解析当前经理的HTML提取下属信息
    with open(html_save_path, "r", encoding="utf-8") as f:
        contents = f.read()
    soup = BeautifulSoup(contents, features="html.parser")
    script = soup.find_all('script')[8].text.strip()[28:-1]
    data = json.loads(script)
    
    # 2.3 提取下属信息，可根据需要调整字段
    for child in data['children']:
        subordinate_info = {
            "upper_manager_id": manager_id,
            "subordinate_id": child["id"],
            "subordinate_name": child["data"].get("name", "")
            # 其他需要的字段自行补充
        }
        all_subordinate_list.append(subordinate_info)
    # 调整请求间隔避免被拦截
    time.sleep(1)

# 3. 汇总所有下属信息保存到CSV
pd.DataFrame(all_subordinate_list).to_csv(FINAL_RESULT_PATH, index=False, encoding="utf-8-sig")

注意事项 #

• 代码默认你已经有可用的cookie鉴权，不需要额外处理登录逻辑
• 如果要递归拉取区域经理的下属，只需要在上述逻辑基础上增加递归判断，只要data['children']不为空就继续往下拉即可

内容的提问来源于stack exchange，提问作者HipYThON0