如何用Python调用clinicaltrials.gov API获取完整研究数据？

1. 获取完整记录（分页处理） #

clinicaltrials.gov API单页最多返回1000条记录，若总记录数超过1000，需通过分页循环获取：

• 先调用一次API拿到总记录数TotalStudies
• 按每页1000条，循环设置min_rnk和max_rnk参数，直到获取全部数据

2. 替换为新API（v2版本） #

新API端点为https://clinicaltrials.gov/api/v2/studies，参数规则与旧API不同：

• 用filter指定搜索条件，格式如filter=conditions:"Diabetes"
• 用fields指定返回字段，字段名采用驼峰式（如nctNumber而非NCT Number）
• 分页用pageSize（最大1000）和pageToken实现

3. 修正代码生成简洁DataFrame #

你的现有代码存在字符串格式化错误、未提取有效数据节点、字段名不匹配API要求等问题，以下是修正版本：

旧API兼容版代码（匹配你的字段需求） #

import requests
import pandas as pd

# 动态搜索条件（后续可替换为输入项）
search_condition = "Diabetes"
base_url = "https://clinicaltrials.gov/api/query/study_fields"

# 旧API要求的驼峰式字段名
extract_fields = [
    'NCTNumber', 'Status', 'Conditions', 'Interventions', 'Sponsor',
    'StudyType', 'Collaborators', 'Acronym', 'OutcomeMeasures', 'Sex',
    'Age', 'Phase', 'Enrollment', 'FunderType', 'StudyDesign',
    'OtherIDs', 'StudyStartDate', 'PrimaryCompletionDate', 'StudyCompletionDate',
    'FirstPostDate', 'ResultsFirstPostDate', 'LastUpdatePostDate', 'Locations', 'StudyDocuments'
]

# 先获取总记录数
init_params = {
    "expr": search_condition,
    "fields": ",".join(extract_fields),
    "max_rnk": 1,
    "fmt": "json"
}
init_response = requests.get(base_url, params=init_params)
init_data = init_response.json()
total_studies = init_data["StudyFieldsResponse"]["NStudiesFound"]

all_studies = []
# 分页循环拉取数据
for min_rnk in range(1, total_studies + 1, 1000):
    max_rnk = min(min_rnk + 999, total_studies)
    params = {
        "expr": search_condition,
        "fields": ",".join(extract_fields),
        "min_rnk": min_rnk,
        "max_rnk": max_rnk,
        "fmt": "json"
    }
    response = requests.get(base_url, params=params)
    data = response.json()
    # 提取有效研究数据
    studies = data["StudyFieldsResponse"]["StudyFields"]
    # 格式化多值字段为字符串，避免嵌套结构
    formatted_studies = []
    for study in studies:
        study_dict = {}
        for field in study:
            study_dict[field] = ", ".join(study[field]) if isinstance(study[field], list) else study[field]
        formatted_studies.append(study_dict)
    all_studies.extend(formatted_studies)

# 生成简洁DataFrame
df = pd.DataFrame(all_studies)
print(df.head())

新API（v2版本）示例代码 #

import requests
import pandas as pd

search_condition = "Diabetes"
base_url = "https://clinicaltrials.gov/api/v2/studies"

# 新API驼峰式字段名，含嵌套字段
extract_fields = [
    "nctNumber", "overallStatus", "conditions", "interventions", "sponsor",
    "studyType", "collaborators", "acronym", "outcomeMeasures", "eligibility.gender",
    "eligibility.age", "phase", "enrollmentCount", "fundingType", "studyDesign",
    "otherIdentifiers", "startDate", "primaryCompletionDate", "completionDate",
    "firstPostDate", "resultsFirstPostDate", "lastUpdatePostDate", "locations", "documents"
]

all_studies = []
page_token = None

while True:
    params = {
        "filter": f"conditions:\"{search_condition}\"",
        "fields": ",".join(extract_fields),
        "pageSize": 1000,
        "pageToken": page_token
    }
    response = requests.get(base_url, params=params)
    data = response.json()
    
    # 扁平化嵌套字段并提取目标数据
    for study in data.get("studies", []):
        def flatten_dict(d, parent_key=''):
            items = []
            for k, v in d.items():
                new_key = f"{parent_key}.{k}" if parent_key else k
                if isinstance(v, dict):
                    items.extend(flatten_dict(v, new_key).items())
                elif isinstance(v, list):
                    items.append((new_key, ", ".join(str(i) for i in v)))
                else:
                    items.append((new_key, v))
            return dict(items)
        flattened = flatten_dict(study)
        filtered = {k: flattened[k] for k in extract_fields if k in flattened}
        all_studies.append(filtered)
    
    # 终止条件：无下一页token
    page_token = data.get("nextPageToken")
    if not page_token:
        break

df = pd.DataFrame(all_studies)
print(df.head())

关键修正说明 #

• 旧API字段名需用驼峰式，原代码中的空格命名会导致API返回空值
• 必须从StudyFieldsResponse.StudyFields节点提取有效数据，而非直接转换整个JSON响应
• 多值字段（如Locations、Interventions）需转为字符串，避免DataFrame出现嵌套结构
• 新API需处理嵌套字段的扁平化，分页逻辑依赖pageToken而非旧API的min_rnk/max_rnk

内容的提问来源于stack exchange，提问作者drymolasses