CuPy配置无CUDA设备检测问题求助（Singularity部署HPC+GPU容器）

构建Docker容器并通过Singularity部署在GPU HPC集群上，运行cp.show_config()时出现CUDA设备检测失败，报错：

CUDARuntimeError('cudaErrorNoDevice: no CUDA-capable device is detected')

CuPy配置信息如下：

OS: Linux-5.4.0-135-generic-x86_64-with-glibc2.31
Python Version: 3.9.12
CuPy Version: 9.6.0
CuPy Platform: NVIDIA CUDA
NumPy Version: 1.21.5
SciPy Version: 1.6.0
Cython Build Version: 0.29.24
Cython Runtime Version: 0.29.28
CUDA Root: /opt/conda/envs/rapids
nvcc PATH: None
CUDA Build Version: 11020
CUDA Driver Version: 11060
CUDA Runtime Version: CUDARuntimeError('cudaErrorNoDevice: no CUDA-capable device is detected')
cuBLAS Version: (available)
cuFFT Version: 10400
cuRAND Version: 10203
cuSOLVER Version: (11, 3, 4)
cuSPARSE Version: (available)
NVRTC Version: (11, 2)
Thrust Version: 101000
CUB Build Version: 101000
Jitify Build Version: 65946d2
cuDNN Build Version: None
cuDNN Version: None
NCCL Build Version: 21104
NCCL Runtime Version: 21210
cuTENSOR Version: None
cuSPARSELt Build Version: None

使用的Dockerfile内容：

# Pulls the basic Image from NVIDIA repository
FROM rapidsai/rapidsai:22.04-cuda11.2-runtime-ubuntu20.04-py3.9

# OS install cuda toolkit
RUN apt-get update
RUN apt-get install -y cuda-toolkit-11.2

# Pulls the basic Image from NVIDIA repository
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
    --fix-missing git python3-setuptools python3-pip build-essential libcurl4-gnutls-dev \
    zlib1g-dev rsync vim cmake tabix && \
    apt-get clean

# Conda install on base env cudnn
RUN conda install --yes -c conda-forge cudnn=8.0.5.39

# Adding env directory to path and activate rapids env
ENV PATH /opt/conda/envs/rapids/bin:$PATH
RUN /bin/bash -c "source activate rapids"

# Install libraries needed in the examples
RUN pip install \
    scanpy==1.9.1 wget pytabix dash-daq \
    dash-html-components dash-bootstrap-components dash-core-components \
    pytest utils tensorflow

RUN pip install --upgrade tensorflow-gpu

WORKDIR /workspace
ENV HOME /workspace

RUN mkdir -p /.singularity.d/env
RUN echo "#!/usr/bin/env bash" >  /.singularity.d/env/99-custom_prompt.sh
RUN echo 'PS1="[${SINGULARITY_NAME%.*}]\u@\h:\w\$ "' >>  /.singularity.d/env/99-custom_prompt.sh
RUN conda install batchspawner

1. 确保Singularity启动时正确挂载GPU #

HPC环境下Singularity必须通过--nv参数启用GPU支持，否则容器无法访问宿主机GPU设备。启动命令应为：

singularity run --nv your_image.sif

同时在宿主机验证GPU可用性：运行nvidia-smi确认设备正常识别，无硬件驱动问题。

2. 修复容器内CUDA环境变量配置 #

从CuPy配置看nvcc PATH: None，且缺少CUDA库路径的环境变量，需在Dockerfile中添加LD_LIBRARY_PATH配置，确保CUDA和cudnn的库被正确加载：

# 添加到Dockerfile的环境变量设置部分
ENV LD_LIBRARY_PATH /opt/conda/envs/rapids/lib:/usr/local/cuda/lib64:$LD_LIBRARY_PATH

重新构建镜像并转换为Singularity格式后，验证容器内echo $LD_LIBRARY_PATH输出包含上述路径。

3. 清理CUDA环境冲突 #

容器内同时通过apt安装了cuda-toolkit-11.2和conda安装了cudnn，容易导致环境冲突：基础镜像rapidsai/rapidsai已包含CUDA Runtime，无需额外安装toolkit。删除Dockerfile中apt-get install -y cuda-toolkit-11.2这一步，避免路径和组件冲突。

4. 验证容器内基础CUDA可用性 #

进入容器后先执行nvidia-smi，如果无法输出GPU信息，说明Singularity GPU挂载失败，需联系集群管理员确认节点GPU权限、nvidia-container-runtime是否正确安装；如果nvidia-smi正常，再执行以下命令测试CuPy：

import cupy
print(cupy.cuda.is_available())
print(cupy.cuda.Device(0))

若仍报错，尝试重新安装对应CUDA版本的CuPy：

pip uninstall -y cupy
pip install cupy-cuda112==9.6.0

5. 确认CUDA_VISIBLE_DEVICES环境变量 #

HPC集群中可能通过CUDA_VISIBLE_DEVICES限制GPU访问，需确保该变量在容器内正确传递。启动Singularity时可手动指定：

singularity run --nv --env CUDA_VISIBLE_DEVICES=0 your_image.sif

或在容器内检查变量值：echo $CUDA_VISIBLE_DEVICES，若为空则设置为对应设备编号（如0,1表示使用前两块GPU）。

内容的提问来源于stack exchange，提问作者fabio.geraci