You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

RTX3090Ti运行PyTorch代码出现CUDNN_STATUS_EXECUTION_FAILED错误求助

问题描述

我在搭载RTX 3090Ti的电脑上运行HMN项目代码,代码在CPU上能正常运行,但执行第一个前向传播层时触发错误:RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED。已经按照项目仓库的说明配置了环境,现在不确定是否需要额外安装cuDNN包,特此求助。

完整堆栈跟踪信息:

Traceback (most recent call last):
  File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/tekre/.vscode/extensions/ms-python.python-2022.18.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/home/tekre/.vscode/extensions/ms-python.python-2022.18.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/home/tekre/.vscode/extensions/ms-python.python-2022.18.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/home/tekre/.vscode/extensions/ms-python.python-2022.18.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 322, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/home/tekre/.vscode/extensions/ms-python.python-2022.18.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 136, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/home/tekre/.vscode/extensions/ms-python.python-2022.18.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/home/tekre/Desktop/video_captioning_studies/HMN/main.py", line 37, in <module>
    model = train_fn(cfgs, cfgs.model_name, model, hungary_matcher, train_loader, valid_loader, device)
  File "/home/tekre/Desktop/video_captioning_studies/HMN/train.py", line 66, in train_fn
    preds, objects_pending, action_pending, video_pending = model(objects, object_masks, feature2ds, feature3ds, numeric_caps)
  File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tekre/Desktop/video_captioning_studies/HMN/models/caption_models/hierarchical_model.py", line 95, in forward
    objects_feats, action_feats, video_feats, objects_semantics, action_semantics, video_semantics = self.forward_encoder(objects_feats, objects_mask, feature2ds, feature3ds)
  File "/home/tekre/Desktop/video_captioning_studies/HMN/models/caption_models/hierarchical_model.py", line 57, in forward_encoder
    objects_feats, objects_semantics = self.entity_level(feature2ds, feature3ds, objects, objects_mask)
  File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tekre/Desktop/video_captioning_studies/HMN/models/encoders/entity_level.py", line 53, in forward
    features_2d = self.feature2d_proj(features_2d.view(-1, features_2d.shape[-1]))
  File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 107, in forward
    exponential_average_factor, self.eps)
  File "/home/tekre/miniconda3/envs/hmn_env/lib/python3.7/site-packages/torch/nn/functional.py", line 1670, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
解决方案建议
  • 确认版本兼容性:通过conda安装的PyTorch通常自带适配的cuDNN,但手动安装CUDA时需单独匹配版本。用torch.backends.cudnn.version()查看当前cuDNN版本,确保它与CUDA、PyTorch版本兼容(比如CUDA 11.7对应cuDNN 8.5+,PyTorch 1.13+)。
  • 临时禁用cuDNN测试:在代码开头添加torch.backends.cudnn.enabled = False,如果能正常运行,说明是cuDNN兼容性问题,需调整版本。
  • 检查BatchNorm输入:从堆栈看错误出在BatchNorm层,确认输入张量的维度、数据类型是否正确,有没有NaN/Inf值。可以在feature2d_proj前打印features_2d的形状和数值范围排查问题。
  • 更新显卡驱动:RTX 3090Ti需要较新的驱动版本,确保NVIDIA驱动是适配当前CUDA版本的最新稳定版。
  • 重建环境:如果以上方法无效,删除现有conda环境,重新按照项目文档创建环境,优先用conda安装PyTorch(自动处理CUDA和cuDNN依赖),避免手动混合安装导致冲突。

内容的提问来源于stack exchange,提问作者tealy

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.08.14 10:40:58