使用gVisor+containerd+crictl调用NVIDIA GPU时沙箱启动失败

1. 验证runsc的GPU支持 #

• 确认安装的runsc版本支持NVProxy，运行命令：

  runsc --version
  

  输出需包含nvproxy相关标识，若缺失则重新安装带GPU支持的runsc版本。

2. 修正runscgpu脚本 #

• 确保脚本权限和路径正确：

  chmod +x /usr/local/bin/runscgpu
  which runsc
  

  脚本内的runsc路径需与which runsc输出一致，修正后脚本示例：

  #!/bin/bash
  exec $(which runsc) --nvproxy "$@"
  

3. 调整nvidia-container-runtime配置 #

编辑/etc/nvidia-container-runtime/config.toml：

• 将运行模式从legacy改为cdi（更适配gVisor GPU支持）：

  [nvidia-container-runtime]
  debug = "/var/log/nvidia-container-runtime.log"
  log-level = "info"
  mode = "cdi"
  runtimes = ["runscgpu", "runc"]
  

• 确保CDI配置路径正确：

  [nvidia-container-runtime.modes.cdi]
  annotation-prefixes = ["cdi.k8s.io/"]
  default-kind = "nvidia.com/gpu"
  spec-dirs = ["/etc/cdi", "/var/run/cdi"]
  

4. 配置containerd Runtime Handler #

编辑/etc/containerd/config.toml，添加或修正runscgpu和nvidia runtime配置：

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runscgpu]
  runtime_type = "io.containerd.runsc.v1"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
  runtime_type = "io.containerd.runc.v2"
  runtime_engine = "/usr/bin/nvidia-container-runtime"

重启containerd服务：

systemctl restart containerd

5. 直接验证runsc GPU功能 #

跳过containerd，直接测试runsc的GPU可用性：

runsc --nvproxy run --rm nvidia/cuda:12.0.1-base-ubuntu22.04 nvidia-smi

若能正常输出GPU信息，说明runsc本身GPU功能正常，问题出在containerd集成环节。

6. 排查日志细节 #

• 开启runsc调试日志，启动时添加注解：

  SANDBOX_ID=$(sudo crictl runp --runtime nvidia --annotations "io.kubernetes.cri.runtime-config=runsc.debug-log=/tmp/runsc.log" pod.json)
  

• 查看nvidia-container-runtime日志：

  cat /var/log/nvidia-container-runtime.log
  

• 实时查看containerd日志：

  journalctl -u containerd -f
  

内容的提问来源于stack exchange，提问作者PSKP