多GPU训练YOLOv4出现cuDNN_STATUS_BAD_PARAM错误求助

问题背景 #

使用4块Tesla T4 GPU训练YOLOv4目标检测模型，迭代1000至2000次左右出现以下错误：

(next mAP calculation at 1214 iterations)
1216: 1.498149, 1.476265 avg loss, 0.010440 rate, 2.871675 seconds, 311296 images, 4.278861 hours left
4Darknet error location: ./src/convolutional_kernels.cu, forward_convolutional_layer_gpu(), line #543
cuDNN Error: CUDNN_STATUS_BAD_PARAM: No such file or directory
backtrace (13 entries)
1/13: ./darknet(log_backtrace+0x38) [0x55e9c42cfc18]
2/13: ./darknet(error+0x3d) [0x55e9c42cfcfd]
3/13: ./darknet(+0x834b0) [0x55e9c42d24b0]
4/13: ./darknet(cudnn_check_error_extended+0x7c) [0x55e9c42d2a9c]
5/13: ./darknet(forward_convolutional_layer_gpu+0x2c2) [0x55e9c43b0d12]
6/13: ./darknet(forward_network_gpu+0x101) [0x55e9c43c4d41]
7/13: ./darknet(network_predict_gpu+0x131) [0x55e9c43c7711]
8/13: ./darknet(validate_detector_map+0xa2e) [0x55e9c435afce]
9/13: ./darknet(train_detector+0x17f8) [0x55e9c435db48]
10/13: ./darknet(run_detector+0xa04) [0x55e9c4361eb4]
11/13: ./darknet(main+0x341) [0x55e9c428c311]
12/13: /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f5165d4d083]
13/13: ./darknet(_start+0x2e) [0x55e9c428e58e]
Resizing to initial size: 608 x 608  try to allocate additional workspace_size = 70.08 MB
 CUDA allocate done!
 try to allocate additional workspace_size = 70.08 MB
 CUDA allocate done!
 try to allocate additional workspace_size = 70.08 MB
 CUDA allocate done!
 try to allocate additional workspace_size = 70.08 MB
 CUDA allocate done!

 calculation mAP (mean average precision)...
 Detection layer: 30 - type = 28
 Detection layer: 37 - type = 28
 Detection layer: 44 - type = 28

 cuDNN status Error in: file: ./src/convolutional_kernels.cu function: forward_convolutional_layer_gpu() line: 543

 cuDNN Error: CUDNN_STATUS_BAD_PARAM

使用同一配置文件单GPU训练时无异常，配置文件如下：

[net]
# Testing
#batch=1
#subdivisions=1
# Training
batch=64
subdivisions=8
width=608
height=608
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1

learning_rate=0.00261
burn_in=1000
max_batches = 10000
policy=steps
steps=8000,9000
scales=.1,.1

[convolutional]
batch_normalize=1
filters=32
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

[route]
layers=-1
groups=2
group_id=1

[convolutional]
batch_normalize=1
filters=32
size=3
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=32
size=3
stride=1
pad=1
activation=leaky

[route]
layers = -1,-2

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=leaky

[route]
layers = -6,-1

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[route]
layers=-1
groups=2
group_id=1

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

[route]
layers = -1,-2

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[route]
layers = -6,-1

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[route]
layers=-1
groups=2
group_id=1

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[route]
layers = -1,-2

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[route]
layers = -6,-1

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

##################################

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=21
activation=linear



[yolo]
mask = 6,7,8
anchors =  6, 14,  14, 33,  34, 63,  55,134, 108, 77,  98,162, 127,277, 280,179, 274,405
classes=2
num=9
jitter=.3
scale_x_y = 1.05
cls_normalizer=1.0
iou_normalizer=0.07
iou_loss=ciou
ignore_thresh = .7
truth_thresh = 1
random=1
resize=1.5
nms_kind=greedynms
beta_nms=0.6

[route]
layers = -4

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[upsample]
stride=2

[route]
layers = -1, 23

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=21
activation=linear

[yolo]
mask = 3,4,5
anchors =  6, 14,  14, 33,  34, 63,  55,134, 108, 77,  98,162, 127,277, 280,179, 274,405
classes=2
num=9
jitter=.3
scale_x_y = 1.05
cls_normalizer=1.0
iou_normalizer=0.07
iou_loss=ciou
ignore_thresh = .7
truth_thresh = 1
random=1
resize=1.5
nms_kind=greedynms
beta_nms=0.6


[route]
layers = -3

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=leaky

[upsample]
stride=2

[route]
layers = -1, 15

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=21
activation=linear

[yolo]
mask = 0,1,2
anchors = 6, 14,  14, 33,  34, 63,  55,134, 108, 77,  98,162, 127,277, 280,179, 274,405
classes=2
num=9
jitter=.3
scale_x_y = 1.05
cls_normalizer=1.0
iou_normalizer=0.07
iou_loss=ciou
ignore_thresh = .7
truth_thresh = 1
random=1
resize=1.5
nms_kind=greedynms
beta_nms=0.6

环境信息：

CUDA-version: 11040 (12020), cuDNN: 8.2.4, GPU count: 4, OpenCV version: 4.2.0, 0 : compute_capability = 750, cudnn_half = 0, GPU: Tesla T4

疑问与求助 #

1. 为什么YOLOv4官方建议先使用单GPU训练1000次再进行多GPU迁移学习？
2. 如何解决当前出现的CUDNN_STATUS_BAD_PARAM错误？

解答 #

一、官方建议单GPU预训练1000次的原因 #

• 稳定参数初始化：多GPU分布式训练初期，参数同步机制（如AllReduce）可能因参数随机波动出现延迟或不一致，单GPU训练能让模型完成基础初始化，参数趋于稳定后再迁移，避免同步冲突。
• 适配数据集特征：前1000次属于burn_in学习率预热阶段，单GPU下模型可快速适配数据集的特征分布，后续多GPU训练时能基于稳定参数更快收敛。
• 降低异常风险：随机初始化的参数在多GPU同步时，可能因维度或数值差异触发底层计算库（如cuDNN）的参数错误，预训练后参数波动减小，能降低这类概率。

二、CUDNN_STATUS_BAD_PARAM错误的解决办法 #

1. 调整batch与subdivisions参数
   多GPU下总batch是单GPU的N倍（N为GPU数），当前batch=64，4GPU下每个GPU分摊16个样本，再除以subdivisions=8，单GPU每次处理2个样本。可尝试调大subdivisions=16或降低总batch=32，避免显存分配或参数维度不匹配触发cuDNN错误。

2. 修复CUDA与cuDNN版本冲突
   环境中同时存在CUDA 11.4和12.2，版本冲突会导致cuDNN调用异常。需清理多余CUDA版本，确保环境变量仅指向CUDA 11.4，同时验证cuDNN 8.2.4与CUDA 11.4的兼容性（官方标注两者兼容）。

3. 调整训练时的mAP计算触发逻辑
   错误出现在mAP计算阶段，可修改Darknet源码，暂时关闭训练过程中的自动mAP计算，或调整触发间隔到迭代后期。具体操作是在train_detector.c中找到validate_detector_map的调用逻辑，注释或修改触发的迭代次数。

4. 重新编译支持多GPU的Darknet
   修改Makefile确保开启多GPU支持：

   GPU=1
   CUDNN=1
   CUDNN_HALF=0
   OPENCV=1
   MULTI_GPU=1
   

   执行make clean && make重新编译，确保编译过程无报错，生成的二进制文件支持多GPU训练。

5. 遵循官方迁移流程
   先单GPU训练1000次得到yolov4_1000.weights，再启动多GPU训练时加载该权重：

   ./darknet detector train data/obj.data cfg/yolov4.cfg yolov4_1000.weights -gpus 0,1,2,3
   

   基于稳定参数启动多GPU训练，能大幅降低初始化阶段的参数错误概率。

内容的提问来源于stack exchange，提问作者Malla Raraju