Windows10下detectron2报Not compiled with GPU support错误求助

• 运行系统：Windows 10
• 目标运行项目：ContactHands，基于detectron2与支持CUDA的PyTorch开发
• 已部署的环境组件：
  • CUDA 11.6
  • PyTorch 1.12.0，安装命令为conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge
  • detectron2 0.1.1
  • Python 3.7.13

采集环境信息得到的核心参数如下：

sys.platform              win32
Python                    3.7.13 (default, Mar 28 2022, 08:03:21) [MSC v.1916 64 bit (AMD64)]
numpy                     1.21.6
detectron2                0.1.1 @c:\detectron2_v0.1.1\detectron2
detectron2 compiler       MSVC 191627048
detectron2 CUDA compiler  not available
detectron2 arch flags     c:\detectron2_v0.1.1\detectron2\_C.cp37-win_amd64.pyd; cannot find cuobjdump
DETECTRON2_ENV_MODULE     <not set>
PyTorch                   1.12.0 @C:\Users\berge\.conda\envs\detectron2_v0.1.1\lib\site-packages\torch
PyTorch debug build       False
CUDA available            True
GPU 0                     NVIDIA GeForce MX250
CUDA_HOME                 C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6
NVCC                      Not Available
Pillow                    9.2.0
torchvision               0.13.0 @C:\Users\berge\.conda\envs\detectron2_v0.1.1\lib\site-packages\torchvision
torchvision arch flags    C:\Users\berge\.conda\envs\detectron2_v0.1.1\lib\site-packages\torchvision\_C.pyd; cannot find cuobjdump
cv2                       4.5.5
------------------------  --------------------------------------------------------------------------------------------------------
PyTorch built with:
  - C++ Version: 199711
  - MSVC 192829337
  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
  - OpenMP 2019
  - LAPACK is enabled (usually provided by MKL)
  - CPU capability usage: AVX2
  - CUDA Runtime 11.6
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.3.2  (built against CUDA 11.5)
  - Magma 2.5.4
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -IC:/cb/pytorch_1000000000000/work/mkl/include -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF,

执行CUDA可用性校验命令python -c "import torch; from torch.utils.cpp_extension import CUDA_HOME; print(torch.cuda.is_available(), CUDA_HOME)"，返回结果为：

True C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6

校验结论：PyTorch可正常识别CUDA环境

执行训练命令python train_net.py --first-config-file configs/ContactHands.yaml时，抛出错误RuntimeError: Not compiled with GPU support，完整报错栈如下：

Traceback (most recent call last):
  File "train_net.py", line 281, in <module>
    args=(args,),
  File "c:\detectron2_v0.1.1\detectron2\engine\launch.py", line 52, in launch
    main_func(*args)
  File "train_net.py", line 269, in main
    do_train(cfg1, model1, model2, resume=args.resume)
  File "train_net.py", line 171, in do_train
    second_stream_outputs = inference_second_stream(model2, data, height, width)
  File "train_net.py", line 60, in inference_second_stream
    outputs = model(im)
  File "C:\detectron2_v0.1.1\ContactHands\contact_hands_two_stream\engine\custom_predictor.py", line 43, in __call__
    predictions = self.model([inputs])
  File "C:\Users\berge\.conda\envs\detectron2_v0.1.1\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\detectron2_v0.1.1\ContactHands\contact_hands_two_stream\modeling\meta_arch\second_stream_rcnn.py", line 52, in forward
    return self.inference(batched_inputs)
  File "C:\detectron2_v0.1.1\ContactHands\contact_hands_two_stream\modeling\meta_arch\second_stream_rcnn.py", line 112, in inference
    second_stream_outputs = self.roi_heads(images, features, proposals, None)
  File "C:\Users\berge\.conda\envs\detectron2_v0.1.1\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\detectron2_v0.1.1\ContactHands\contact_hands_two_stream\modeling\roi_heads\second_stream_roi_head.py", line 123, in forward
    pred_instances = self._forward_box(features, proposals)
  File "C:\detectron2_v0.1.1\ContactHands\contact_hands_two_stream\modeling\roi_heads\second_stream_roi_head.py", line 176, in _forward_box
    box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
  File "C:\Users\berge\.conda\envs\detectron2_v0.1.1\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "c:\detectron2_v0.1.1\detectron2\modeling\poolers.py", line 233, in forward
    output[inds] = pooler(x_level, pooler_fmt_boxes_level)
  File "C:\Users\berge\.conda\envs\detectron2_v0.1.1\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "c:\detectron2_v0.1.1\detectron2\layers\roi_align.py", line 95, in forward
    input, rois, self.output_size, self.spatial_scale, self.sampling_ratio, self.aligned
  File "c:\detectron2_v0.1.1\detectron2\layers\roi_align.py", line 20, in forward
    input, roi, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned
RuntimeError: Not compiled with GPU support

核心问题出在detectron2的安装版本上：

• 环境信息明确显示detectron2 CUDA compiler not available、NVCC Not Available，说明当前安装的detectron2是CPU版本，编译过程中没有检测到CUDA编译工具链，最终生成的C++扩展没有包含CUDA算子实现。当训练流程调用ROIAlign这类必须走CUDA的自定义算子时，就会抛出Not compiled with GPU support错误。
• 不要混淆PyTorch的CUDA支持和detectron2的CUDA支持：PyTorch自带预编译的CUDA runtime，所以能直接识别CUDA设备，但detectron2的自定义算子需要本地编译，编译阶段找不到NVCC等CUDA开发组件就只会生成CPU版本的扩展。

• 补全CUDA开发组件：当前环境只安装了CUDA运行时，缺少编译必须的NVCC等开发工具。重新运行CUDA 11.6安装程序，勾选完整的CUDA Development组件安装，装完新开终端执行nvcc -V能正常输出版本号即说明组件装全。
• 清理旧的detectron2残留：执行pip uninstall detectron2 -y，手动删掉c:\detectron2_v0.1.1下的build目录、所有*.pyd后缀的编译产物，避免旧的CPU版扩展被加载。
• 对齐编译器版本：当前安装的PyTorch 1.12是用MSVC 1928（对应VS2019）编译的，之前编译detectron2用的是MSVC 1916（对应VS2017），版本不匹配会导致编译/运行异常。安装VS2019并勾选「使用C++的桌面开发」组件，后续编译操作都从VS2019的x64原生工具命令提示符进入conda环境执行。
• 重新编译GPU版detectron2：进入detectron2源码目录，执行python setup.py build develop，编译过程中留意日志，出现CUDA detected: 11.6、Building CUDA operators相关提示就说明正在编译GPU版本。
• 安装完成后重新校验环境，确认detectron2 CUDA compiler项显示11.6、NVCC状态可用后，再执行训练命令即可。

内容的提问来源于stack exchange，提问作者Volcan Tajumulco