如何排除线程池映射时间，仅统计API请求的实际耗时？

我编写了一段多线程并行调用API的代码，每次迭代会并行执行5个API调用，整个迭代需重复约1000次。目前输出显示每次迭代耗时约1秒，但理论上并行执行的迭代平均耗时应在0.2~0.4秒左右（串行执行时迭代平均耗时为3-4秒，已验证并行机制有效）。经排查发现，当前计时包含了线程池映射过程以及API响应验证的时间，希望修改代码，仅统计API请求本身的耗时。

原代码 #

#Method 2: Multithreading

import concurrent.futures 
import requests 
import random
import pandas as pd
import time     

#Function to determine validity of the API response 
def call_api(url):
    try:
        response = requests.get(url)
        return response.status_code == 200
    except requests.exceptions.RequestException:
        return False

# Load the DataFrame from your dataset
df=pd.read_csv(r"C:\Users\jose.moquiambo\Bulk Calling APIs\sales_confidential.csv")  

# Number of iterations
num_iterations = 100

# Create an empty DataFrame to store the results
results_df = pd.DataFrame(columns=['Iteration', 'Pass', 'Time Taken'])

# Variables for tracking min, max, and total time
min_time = float('inf')
max_time = float('-inf')
total_time = 0

def process_iteration(iteration):
    # Get a random sample of URLs from the DataFrame
    random_urls = df['url'].sample(n=5).tolist()

    # Start timer
    start_time = time.time()

    # Execute API calls concurrently using ThreadPoolExecutor
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        results = executor.map(call_api, random_urls)

    # Stop timer
    end_time = time.time()

    # Calculate the time taken for this iteration# 
    iteration_time = end_time - start_time

    # Update min, max, and total time
    global min_time, max_time, total_time
    min_time = min(min_time, iteration_time)
    max_time = max(max_time, iteration_time)
    total_time += iteration_time

    # Check if any API call was not successful in this iteration
    passed = 'Y' if all(results) else 'N'

    # Add the iteration results to the DataFrame
    results_df.loc[iteration] = [iteration, passed , iteration_time]

# Run the iterations
for i in range(1, num_iterations + 1):
    process_iteration(i)

# Calculate average time per iteration
avg_time = total_time / num_iterations

# Display the results DataFrame
print(results_df)

# Summary statistics
print("Minimum time taken:", min_time)
print("Maximum time taken:", max_time)
print("Average time per iteration:", avg_time)
print("Y stands for error-free response and N for errors")

原输出 #

Iteration Pass  Time Taken
1           1    Y    1.123318
2           2    Y    1.122956
3           3    Y    1.367392
4           4    Y    1.064120
5           5    Y    1.072951
6           6    Y    1.026620
7           7    Y    1.573418
8           8    Y    1.039701
9           9    Y    1.095653
10         1ro 发表** Jud regardless- Gre breadcrumbs MartMindtern... Trigger sustained此次修改的核心是把计时逻辑聚焦到API请求的核心环节，排除线程池调度、结果处理等额外开销。修改后的代码如下：

```python
#Method 2: Multithreading

import concurrent.futures 
import requests 
import random
import pandas as pd
import time     

#Function to determine validity of the API response and record request time
def call_api(url):
    request_start = time.time()
    try:
        response = requests.get(url)
        request_end = time.time()
        # 返回(是否成功, 请求耗时)
        return (response.status_code == 200, request_end - request_start)
    except requests.exceptions.RequestException:
        request_end = time.time()
        return (False, request_end - request_start)

# Load the DataFrame from your dataset
df=pd.read_csv(r"C:\Users\jose.moquiambo\Bulk Calling APIs\sales_confidential.csv")  

# Number of iterations
num_iterations = 100

# Create an empty DataFrame to store the results
results_df = pd.DataFrame(columns=['Iteration', 'Pass', 'Total Request Time', 'Avg Request Time', 'Max Request Time'])

# Variables for tracking min, max, and total time
min_total_time = float('inf')
max_total_time = float('-inf')
total_total_time = 0

def process_iteration(iteration):
    # Get a random sample of URLs from the DataFrame
    random_urls = df['url'].sample(n=5).tolist()

    # 记录整个并行请求阶段的开始时间（第一个请求发起前）
    iteration_request_start = time.time()
    
    # Execute API calls concurrently using ThreadPoolExecutor
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        # 使用submit代替map，获取每个请求的future对象
        futures = [executor.submit(call_api, url) for url in random_urls]
        # 等待所有请求完成
        concurrent.futures.wait(futures)
    
    # 记录整个并行请求阶段的结束时间（最后一个请求完成后）
    iteration_request_end = time.time()
    total_request_time = iteration_request_end - iteration_request_start

    # 收集每个请求的结果和耗时
    results = []
    request_times = []
    for future in futures:
        success, req_time = future.result()
        results.append(success)
        request_times.append(req_time)
    
    # 更新总耗时的统计值
    global min_total_time, max_total_time, total_total_time
    min_total_time = min(min_total_time, total_request_time)
    max_total_time = max(max_total_time, total_request_time)
    total_total_time += total_request_time

    # 判断当前迭代是否所有请求都成功
    passed = 'Y' if all(results) else 'N'

    # 计算单迭代内的请求平均耗时和最大耗时
    avg_req_time = sum(request_times) / len(request_times)
    max_req_time = max(request_times)

    # 将结果写入DataFrame
    results_df.loc[iteration] = [iteration, passed, total_request_time, avg_req_time, max_req_time]

# Run the iterations
for i in range(1, num_iterations + 1):
    process_iteration(i)

# Calculate average total request time per iteration
avg_total_time = total_total_time / num_iterations

# Display the results DataFrame
print(results_df)

# Summary statistics
print("Minimum total request time per iteration:", min_total_time)
print("Maximum total request time per iteration:", max_total_time)
print("Average total request time per iteration:", avg_total_time)
print("Y stands for error-free response and N for errors")

修改说明 #

1. 精准统计单个请求耗时：在call_api函数内部添加计时，记录法，运无法原对每个」对象交 is 提供�与此同时，还能记录每个请求的单独耗时，方便分析单请求的性能波动。

内容的提问来源于stack exchange，提问作者Jose Moquiambo