启动Pod前等待NVIDIA GPU Operator完成初始化的问题排查

初始化操作及正常状态验证 #

• 执行命令：microk8s enable gpu
• 等待所有GPU Operator相关Pod运行正常（状态为绿色）
• 执行命令：microk8s helm install -n morpheus morpheus-ai-engine morpheus-ai-engine
• 等待morpheus相关Pod运行正常（状态为绿色）

此时验证AI Engine Pod的GPU访问权限：

kubectl exec ai-engine-897d65cff-b2trz -- nvidia-smi
Wed Feb 22 16:35:32 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.78.01    Driver Version: 525.78.01    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P400         Off  | 00000000:04:00.0 Off |                  N/A |
|  0%   38C    P8    N/A /  30W |     98MiB /  2048MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

运行vector-add测试Pod，返回Test PASSED。

重启后的异常现象 #

重启MicroK8s后出现以下问题：

• nvidia-device-plugin-validator Pod启动失败，报错UnexpectedAdmissionError，提示无可用GPU
• 在AI Engine Pod中执行nvidia-smi返回“command not found”
• vector-add测试Pod因GPU资源不足无法启动

临时恢复方法 #

卸载ai-engine Chart，重启MicroK8s并等待所有GPU Operator Pod运行正常后，重新安装ai-engine Chart，所有功能及vector-add测试可恢复正常。

内容的提问来源于stack exchange，提问作者directedition