MicroK8s环境下Kubernetes容器创建耗时过长求助

Pod详情 #

light@siddhalok:~/camera_serv$ k describe pod lh-graphql-6db75544cf-j65wp
Name:             lh-graphql-6db75544cf-j65wp
Namespace:        default
Priority:         0
Service Account:  default
Node:             siddhalok/10.0.2.15
Start Time:       Wed, 01 Feb 2023 05:45:13 +0000
Labels:           app=lh-graphql
                  pod-template-hash=6db75544cf
Annotations:      cni.projectcalico.org/containerID: 1a6c41da95677512e3206b51ca15e7cdbe5784abe4599cba7e6fc1ec4b3188f7
                  cni.projectcalico.org/podIP: 10.1.186.238/32
                  cni.projectcalico.org/podIPs: 10.1.186.238/32
Status:           Pending
IP:                
IPs:              <none>
Controlled By:    ReplicaSet/lh-graphql-6db75544cf
Containers:
  lh-graphql:
    Container ID:   
    Image:          hasura/graphql-engine:v2.13.2.cli-migrations-v3
    Image ID:       
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      HASURA_GRAPHQL_DATABASE_URL:  postgresql://postgres:postgres@$(ORCH_POSTGRES_IP):5432/lh
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nd8c2 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-nd8c2:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

Pod状态列表 #

注：lh-graphql Pod已处于ContainerCreating状态46分钟，另有lh-detector Pod同样处于该状态超过50分钟

light@siddhalok:~/camera_serv$ k get pods
NAME                           READY   STATUS              RESTARTS   AGE
lh-detector-6c4c696cbb-zrqcs   0/1     ContainerCreating   0          53m
lh-postgres-7697c87948-grbvd   1/1     Running             0          47m
lh-pgadmin-58db679494-kmcq2    1/1     Running             0          71m
lh-graphql-6db75544cf-j65wp    0/1     ContainerCreating   0          46m

日志查看报错 #

light@siddhalok:~/camera_serv$ k logs lh-graphql-6db75544cf-j65wp
Error from server (BadRequest): container "lh-graphql" in pod "lh-graphql-6db75544cf-j65wp" is waiting to start: ContainerCreating

系统内存资源 #

light@siddhalok:~$ cat /proc/meminfo
MemTotal:        2988196 kB
MemFree:          175132 kB
MemAvailable:    1697564 kB

排查与解决步骤 #

1. 检查容器运行时日志
   在节点上查看containerd服务日志，排查容器创建阶段的异常：

   journalctl -u containerd -f
   

2. 验证环境变量解析
   Pod中使用了$(ORCH_POSTGRES_IP)作为环境变量，需确认该变量是否已在Pod所在命名空间或节点中正确定义。如果变量未解析，会导致容器启动流程卡住。

3. 检查Calico网络插件状态
   虽然Pod已分配Calico IP，但网络组件异常可能导致容器无法就绪：

   microk8s kubectl get pods -n kube-system -l k8s-app=calico-node
   

   确保所有Calico节点Pod处于Running状态，查看具体节点日志排查网络配置问题。

4. 确认镜像实际拉取状态
   验证节点上是否已存在目标镜像：

   crictl images | grep hasura/graphql-engine
   

   如果镜像不存在，说明之前的拉取未完成，需检查节点网络连接或镜像仓库访问权限。

5. 排查磁盘IO与空间
   检查节点磁盘使用率和IO负载，磁盘资源不足或高IO会导致容器创建延迟：

   df -h
   iostat -x 1
   

内容的提问来源于stack exchange，提问作者noname7619