RTX3090Ti运行PyTorch代码出现CUDNN_STATUS_EXECUTION_FAILED错误求助
问题描述
我在搭载RTX 3090Ti的电脑上运行HMN项目代码,代码在CPU上能正常运行,但执行第一个前向传播层时触发错误:RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED。已经按照项目仓库的说明配置了环境,现在不确定是否需要额外安装cuDNN包,特此求助。
完整堆栈跟踪信息:
Traceback (most recent call last): File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/tekre/.vscode/extensions/ms-python.python-2022.18.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module> cli.main() File "/home/tekre/.vscode/extensions/ms-python.python-2022.18.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main run() File "/home/tekre/.vscode/extensions/ms-python.python-2022.18.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file runpy.run_path(target, run_name="__main__") File "/home/tekre/.vscode/extensions/ms-python.python-2022.18.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 322, in run_path pkg_name=pkg_name, script_name=fname) File "/home/tekre/.vscode/extensions/ms-python.python-2022.18.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 136, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home/tekre/.vscode/extensions/ms-python.python-2022.18.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code exec(code, run_globals) File "/home/tekre/Desktop/video_captioning_studies/HMN/main.py", line 37, in <module> model = train_fn(cfgs, cfgs.model_name, model, hungary_matcher, train_loader, valid_loader, device) File "/home/tekre/Desktop/video_captioning_studies/HMN/train.py", line 66, in train_fn preds, objects_pending, action_pending, video_pending = model(objects, object_masks, feature2ds, feature3ds, numeric_caps) File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/tekre/Desktop/video_captioning_studies/HMN/models/caption_models/hierarchical_model.py", line 95, in forward objects_feats, action_feats, video_feats, objects_semantics, action_semantics, video_semantics = self.forward_encoder(objects_feats, objects_mask, feature2ds, feature3ds) File "/home/tekre/Desktop/video_captioning_studies/HMN/models/caption_models/hierarchical_model.py", line 57, in forward_encoder objects_feats, objects_semantics = self.entity_level(feature2ds, feature3ds, objects, objects_mask) File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/tekre/Desktop/video_captioning_studies/HMN/models/encoders/entity_level.py", line 53, in forward features_2d = self.feature2d_proj(features_2d.view(-1, features_2d.shape[-1])) File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward input = module(input) File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 107, in forward exponential_average_factor, self.eps) File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/site-packages/torch/nn/functional.py", line 1670, in batch_norm training, momentum, eps, torch.backends.cudnn.enabled RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
解决方案建议
- 确认版本兼容性:通过conda安装的PyTorch通常自带适配的cuDNN,但手动安装CUDA时需单独匹配版本。用
torch.backends.cudnn.version()查看当前cuDNN版本,确保它与CUDA、PyTorch版本兼容(比如CUDA 11.7对应cuDNN 8.5+,PyTorch 1.13+)。 - 临时禁用cuDNN测试:在代码开头添加
torch.backends.cudnn.enabled = False,如果能正常运行,说明是cuDNN兼容性问题,需调整版本。 - 检查BatchNorm输入:从堆栈看错误出在BatchNorm层,确认输入张量的维度、数据类型是否正确,有没有NaN/Inf值。可以在
feature2d_proj前打印features_2d的形状和数值范围排查问题。 - 更新显卡驱动:RTX 3090Ti需要较新的驱动版本,确保NVIDIA驱动是适配当前CUDA版本的最新稳定版。
- 重建环境:如果以上方法无效,删除现有conda环境,重新按照项目文档创建环境,优先用conda安装PyTorch(自动处理CUDA和cuDNN依赖),避免手动混合安装导致冲突。
内容的提问来源于stack exchange,提问作者tealy
相关产品推荐
相关产品推荐

