从Kubeflow Dashboard启动Notebook遇CrashLoopBackOff，kubectl查Pod无异常

本地Ubuntu 20.04桌面电脑（48GB内存、16核CPU），以minikube v1.23.2（Kubernetes Server版本v1.22.2）作为Kubernetes服务器，部署了kubeflow/manifests v1.6分支版本。尝试从Kubeflow Dashboard启动Notebook时，实例陷入CrashLoopBackOff状态无法正常启动，但执行kubectl get po -n kubeflow命令查询kubeflow命名空间下的Pod时，所有Pod均显示正常运行状态。

已在Kubeflow社区提交相关issue，也查找过类似问题，但均与当前情况不符。

环境与配置信息 #

• 资源分配：内存10GB以上、4核CPU
• 选择的Docker镜像：kubeflownotebookswg/jupyter-scipy:v1.6.0-rc.1
• kubectl版本：v1.21.14

相关命令输出 #

$ kubectl get ns
NAME                        STATUS   AGE
auth                        Active   28m
cert-manager                Active   28m
default                     Active   31m
istio-system                Active   28m
knative-eventing            Active   28m
knative-serving             Active   28m
kube-node-lease             Active   31m
kube-public                 Active   31m
kube-system                 Active   31m
kubeflow                    Active   28m
kubeflow-user-example-com   Active   15m

$ kubectl get po -n kubeflow
NAME                                                     READY   STATUS    RESTARTS   AGE
admission-webhook-deployment-c77d48bbb-qqw7q             1/1     Running   0          33m
cache-server-56d94f5d78-8l5mx                            2/2     Running   0          33m
centraldashboard-5864f74d99-t2jv7                        2/2     Running   0          33m
jupyter-web-app-deployment-5bc998bcb5-fwx4x              1/1     Running   0          33m
katib-controller-6848d4dd9f-sgqbz                        1/1     Running   0          33m
katib-db-manager-665954948-qstn8                         1/1     Running   1          33m
katib-mysql-5bf95ddfcc-l4h78                             1/1     Running   0          33m
katib-ui-56ccff658f-jthrl                                1/1     Running   0          33m
kserve-controller-manager-0                              2/2     Running   0          33m
kserve-models-web-app-5878544ffd-59rf4                   2/2     Running   0          33m
kubeflow-pipelines-profile-controller-5d98fd7b4f-dh6n6   1/1     Running   0          33m
metacontroller-0                                         1/1     Running   0          33m
metadata-envoy-deployment-5b685dfb7f-6vj5x               1/1     Running   0          33m
metadata-grpc-deployment-f8d68f687-nd9jq                 2/2     Running   2          33m
metadata-writer-d6498d6b4-8ckxk                          2/2     Running   0          33m
minio-5b65df66c9-4mkn5                                   2/2     Running   0          33m
ml-pipeline-844c786c48-gw7kj                             2/2     Running   7          33m
ml-pipeline-persistenceagent-5854f86f8b-brz62            2/2     Running   0          33m
ml-pipeline-scheduledworkflow-5dddbf664f-786pm           2/2     Running   0          33m
ml-pipeline-ui-6bdfc6dbcd-psf52                          2/2     Running   0          33m
ml-pipeline-viewer-crd-85f6fd557b-22h7w                  2/2     Running   1          33m
ml-pipeline-visualizationserver-7c4885999-pk6ml          2/2     Running   0          33m
mysql-5c7f79f986-z48n5                                   2/2     Running   0          33m
notebook-controller-deployment-6478d4858c-bwzbf          2/2     Running   1          33m
profiles-deployment-7bc47446fb-hc472                     3/3     Running   1          33m
tensorboard-controller-deployment-f4f555b95-cjngm        3/3     Running   2          33m
tensorboards-web-app-deployment-7578c885f7-dm7dl         1/1     Running   0          33m
training-operator-6c9f6fd894-bk5wc                       1/1     Running   0          33m
volumes-web-app-deployment-7bc5754bd4-bhq9n              1/1     Running   0          33m
workflow-controller-6b9b6c5b46-998r2                     2/2     Running   2          33m

相关截图 #

排查过的类似问题 #

• kubernetes-dashboard CrashLoopBackOff
• kubernetes dashboard CrashLoopBackOff

内容的提问来源于stack exchange，提问作者Ryo Matsuzaka