使用asyncio嵌套循环发起HTTP请求时结果重复问题排查

首次接触asyncio，通过嵌套for循环发起HTTP请求获取Bybit的K线数据，发现部分查询结果重复，无法定位原因。

代码实现 #

import sys
sys.path.append(".")
import inspect

import json
import time
import asyncio
import aiohttp
    
async def get_kline_async(symbol:str, interval:str):
    try:
        async with aiohttp.ClientSession() as session:
            url = f"https://api.bybit.com/v5/market/kline?category=linear&symbol={symbol}&interval={interval}&limit=5"
           
            response = await session.get(url)
            
            if response.status == 200:
                json_result = await response.json()
                return json_result["result"]
            else: print("Response Status: ", response.status)
    except: pass


async def get_klines_async(symbols:[], intervals:[]):
    tasks = []
    
    for symbol in symbols:
        for interval in intervals:
            task = asyncio.create_task(get_kline_async(symbol, interval))
            tasks.append(task)
            
    results = await asyncio.gather(*tasks)
    return results

async def main():
    symbols = ["BTCUSDT", "ETHUSDT", "ADAUSDT", "SOLUSDT", "ARBUSDT"]
    intervals = ["D", "240", "120", "60", "30"]
    
    klines = await get_klines_async(symbols, intervals)
    return klines


if __name__ == '__main__':
    start = time.perf_counter()
    klines = asyncio.run(main())
    for kline in klines:
        print(f"{kline["symbol"]} - {kline["list"][0][0]} - {kline["list"][0][1]} - {kline["list"][0][2]} - {kline["list"][0][3]} - {kline["list"][0][4]} - {kline["list"][0][5]} - {kline["list"][0][6]}")
    
    print()
    stop  = time.perf_counter()
    print("Performance Time: ", stop-start)

输出结果 #

BTCUSDT - 1712793600000 - 70655.8 - 71341.7 - 69580 - 69799.9 - 87509.783 - 6165403723.4089
BTCUSDT - 1712836800000 - 70002.6 - 71101 - 69580 - 69799.9 - 44153.837 - 3100720734.2907
BTCUSDT - 1712844000000 - 70301 - 70364.8 - 69580 - 69799.9 - 12260.45 - 857290562.9599
BTCUSDT - 1712847600000 - 69612.5 - 69891.8 - 69580 - 69799.9 - 1812.849 - 126396435.9985
BTCUSDT - 1712847600000 - 69612.5 - 69891.8 - 69580 - 69799.9 - 1812.849 - 126396435.9985
ETHUSDT - 1712793600000 - 3547.32 - 3619.99 - 3487.02 - 3502.99 - 430873.25 - 1528854092.3474
ETHUSDT - 1712836800000 - 3524.17 - 3577.79 - 3487.02 - 3502.99 - 212993.55 - 751487653.6895
ETHUSDT - 1712844000000 - 3527.86 - 3529.7 - 3487.02 - 3502.99 - 55825.45 - 195629671.4847
ETHUSDT - 1712847600000 - 3494.73 - 3512.82 - 3493.13 - 3502.99 - 10998.54 - 38522669.2256
ETHUSDT - 1712847600000 - 3494.73 - 3512.82 - 3493.13 - 3502.99 - 10998.54 - 38522669.2256
ADAUSDT - 1712793600000 - 0.5869 - 0.5951 - 0.5763 - 0.5816 - 73943246 - 43347716.0114
ADAUSDT - 1712836800000 - 0.5821 - 0.5903 - 0.5763 - 0.5816 - 28563457 - 16645209.5703
ADAUSDT - 1712844000000 - 0.5831 - 0.5841 - 0.5763 - 0.5816 - 9015239 - 5229336.8144
ADAUSDT - 1712847600000 - 0.5802 - 0.5829 - 0.5799 - 0.5816 - 702409 - 408504.0364
ADAUSDT - 1712847600000 - 0.5802 - 0.5829 - 0.5799 - 0.5816 - 702409 - 408504.0364
SOLUSDT - 1712793600000 - 173.48 - 176.1 - 170.01 - 174.38 - 4864095.2 - 841277216.041
SOLUSDT - 1712836800000 - 170.51 - 175.41 - 170.01 - 174.38 - 1899449.5 - 327919113.046
SOLUSDT - 1712844000000 - 172.89 - 175 - 170.34 - 174.38 - 705035.2 - 121775729.778
SOLUSDT - 1712847600000 - 173.46 - 175 - 173.39 - 174.38 - 185217.6 - 32284592.776
SOLUSDT - 1712847600000 - 173.46 - 175 - 173.39 - 174.38 - 185217.6 - 32284592.776
ARBUSDT - 1712793600000 - 1.4678 - 1.4837 - 1.4158 - 1.4295 - 23526436.1 - 34101710.94794
ARBUSDT - 1712836800000 - 1.4334 - 1.4569 - 1.4158 - 1.4295 - 9867305.3 - 14161115.12616
ARBUSDT - 1712844000000 - 1.4381 - 1.4403 - 1.4158 - 1.4295 - 3706167.7 - 5284219.49419
ARBUSDT - 1712847600000 - 1.4227 - 1.4333 - 1.4221 - 1.4295 - 696756 - 995386.52836
ARBUSDT - 1712847600000 - 1.4227 - 1.4333 - 1.4221 - 1.4295 - 696756 - 995386.52836

Performance Time:  1.5946870999905514

原因分析 #

1. API返回特性导致的“重复” #

观察输出可以发现，每个交易对的最后两条结果（对应interval=60和interval=30）重复，这是因为：

• interval=60代表1小时K线，interval=30代表30分钟K线
• 当请求时，当前时间处于1小时周期的前30分钟内，此时1小时K线的最新一根和30分钟K线的最新一根属于同一个未完成的时间段，API返回的实时数据完全一致，并非代码错误。

2. 代码潜在优化点（非重复原因，但影响性能） #

• 每次请求都创建新的ClientSession，会产生额外的连接开销，降低效率
• 异常处理直接pass，无法排查请求失败的情况

解决方案 #

1. 区分未完成K线与历史K线 #

可以通过时间戳判断K线是否完成：计算K线周期对应的结束时间，若当前时间未超过结束时间，则该K线为未完成状态，可根据需求过滤或标记。

2. 复用ClientSession优化性能 #

修改代码，在外部创建单个ClientSession并传递给请求函数，避免重复创建连接：

async def get_kline_async(session: aiohttp.ClientSession, symbol:str, interval:str):
    try:
        url = f"https://api.bybit.com/v5/market/kline?category=linear&symbol={symbol}&interval={interval}&limit=5"
        response = await session.get(url)
        
        if response.status == 200:
            json_result = await response.json()
            return json_result["result"]
        else: 
            print(f"Response Status {response.status} for {symbol}-{interval}")
    except Exception as e:
        print(f"Error fetching {symbol}-{interval}: {str(e)}")

async def get_klines_async(symbols:[], intervals:[]):
    async with aiohttp.ClientSession() as session:
        tasks = [asyncio.create_task(get_kline_async(session, symbol, interval)) 
                 for symbol in symbols 
                 for interval in intervals]
        results = await asyncio.gather(*tasks)
        return results

3. 完善异常处理 #

移除except: pass，改为捕获具体异常并打印日志，方便排查请求失败问题。

内容的提问来源于stack exchange，提问作者Emre