请求间隔不同时YOLOv5推理时长异常变化问题排查

我编写了一段运行YOLOv5s模型进行图像推理的脚本，发现在循环中插入time.sleep(2)后，平均推理时长明显增加。已尝试在计时前添加torch.cuda.synchronize()，但结果无变化。相关代码及两种场景下的推理时长数据如下：

import os
import torch
from PIL import Image
import time

ROOT_PATH = os.environ.get("ROOT_PATH", "..")
MODEL_SOURCE = ROOT_PATH + "/dependencies/yolov5"
MODEL_PATH = ROOT_PATH + "/models//yolo/yolov5s"
DEVICE = torch.device('cuda:0')

def load_model():
    model = torch.hub.load(MODEL_SOURCE, 'custom', path=MODEL_PATH, source='local')
    model.to(DEVICE)
    return model

def main():
    model = load_model()
    avg_time = 0
    count = 20
    
    for _ in range(count):
        image = Image.open(ROOT_PATH + "/images/kitchen.webp")
        t1 = time.time()
        results = model(image)
        t2 = time.time()
        avg_time += t2 - t1
        print(f"Time elapsed: {t2 - t1}")
        # this makes difference
        # time.sleep(2)

    print("Average time: ", avg_time / count)

if __name__ == "__main__":
    main()

带time.sleep(2)的推理时长输出 #

# 加sleep后，推理时长呈现固定规律：从第9次循环开始，推理时间大幅上升
Time elapsed: 0.054245710372924805
Time elapsed: 0.024655580520629883
Time elapsed: 0.017188310623168945
Time elapsed: 0.0163266658782959
Time elapsed: 0.01728343963623047
Time elapsed: 0.014595270156860352
Time elapsed: 0.016335487365722656
Time elapsed: 0.015712738037109375
Time elapsed: 0.03265953063964844
Time elapsed: 0.02926349639892578
Time elapsed: 0.03238844871520996
Time elapsed: 0.031163930892944336
Time elapsed: 0.028283119201660156
Time elapsed: 0.03512239456176758
Time elapsed: 0.028989076614379883
Time elapsed: 0.027245521545410156
Time elapsed: 0.01793360710144043
Time elapsed: 0.024689197540283203
Time elapsed: 0.016361713409423828
Time elapsed: 0.026474952697753906
Average time:  0.0253459095954895

不带time.sleep(2)的推理时长输出 #

# 不加sleep的情况
Time elapsed: 0.05458378791809082
Time elapsed: 0.025354385375976562
Time elapsed: 0.017561674118041992
Time elapsed: 0.016532421112060547
Time elapsed: 0.01777052879333496
Time elapsed: 0.01485443115234375
Time elapsed: 0.016752958297729492
Time elapsed: 0.016268014907836914
Time elapsed: 0.019514083862304688
Time elapsed: 0.017979145050048828
Time elapsed: 0.019189119338989258
Time elapsed: 0.018964529037475586
Time elapsed: 0.016851186752319336
Time elapsed: 0.022213220596313477
Time elapsed: 0.017797231674194336
Time elapsed: 0.016551494598388672
Time elapsed: 0.01830887794494629
Time elapsed: 0.01486515998840332
Time elapsed: 0.016211271286010742
Time elapsed: 0.01635599136352539
Average time:  0.01972397565841675

问题原因与解决方案 #

核心原因 #

这种现象是GPU自动休眠/降频导致的：当两次推理间隔2秒时，GPU在空闲期会自动降低功耗（降频）甚至进入休眠状态，等到下一次推理时需要重新唤醒并恢复频率，这个过程会额外消耗时间，从而拉高单次推理时长。

验证方法 #

可以通过nvidia-smi dmon命令持续监控GPU实时频率：加sleep时，空闲期GPU频率会下降，推理时频率回升但需要时间；不加sleep时GPU一直保持高频率运行。

解决方案 #

1. 禁用GPU自动降频：
   NVIDIA GPU可通过nvidia-smi命令锁定频率：

   nvidia-smi -lgc <最低频率>,<最高频率>
   

   例如nvidia-smi -lgc 1500,1500（具体数值需根据GPU型号调整），使用完成后可执行nvidia-smi -rgc恢复默认设置。

2. 保持GPU活跃：
   在sleep期间执行一个极轻量的GPU操作，避免GPU进入休眠：

   # 替换原time.sleep(2)
   torch.cuda.empty_cache()
   torch.tensor([1]).to(DEVICE)
   time.sleep(2)
   

3. 优化计时方式：
   你之前添加的torch.cuda.synchronize()位置有误，正确的计时应在推理前后都同步，确保准确反映GPU实际执行时间：

   t1 = time.time()
   torch.cuda.synchronize()
   results = model(image)
   torch.cuda.synchronize()
   t2 = time.time()
   

内容的提问来源于stack exchange，提问作者Leo Chen