AKS安装Helm Chart遇CreateContainerConfigError故障排查

问题背景 #

在Azure Kubernetes Service（AKS）上安装Helm Chart时，Pod始终处于CreateContainerConfigError状态。

初始错误事件日志 #

Events:
Type Reason Age From Message

---

Normal Scheduled 44s default-scheduler Successfully assigned xxx/xxx-cdn-864bb-sz8p to aks-2810-vmss00001y
Normal Pulled 43s kubelet Container image "df3.azurecr.io/cist/cist-istio-proxyv2:1.16.3" already present on machine
Normal Created 43s kubelet Created container istio-init
Normal Started 43s kubelet Started container istio-init
Normal Pulled 42s kubelet Container image "ncdaksbsztankosdwdbdf3.azurecr.io/cist/cist-istio-proxyv2:1.16.3" already present on machine
Normal Created 42s kubelet Created container istio-proxy
Normal Started 42s kubelet Started container istio-proxy
Normal Pulling 42s kubelet Pulling image "df3.azurecr.io/xxx-engine:23.6.0-123.gf12345"
Normal Pulled 30s kubelet Successfully pulled image "df3.azurecr.io/xxx-engine:23.6.0-123.gf12345" in 11.461509574s
Warning Failed 2s (x6 over 30s) kubelet Error: couldn't find key CDS_TOKEN in Secret cds/cds-token
Normal Pulled 2s (x5 over 29s) kubelet Container image "df3.azurecr.io/cds-engine:23.6.0-562.gf195010" already present on machine

创建cds/cds-token Secret并添加CDS_TOKEN键后，出现新的错误：

后续错误事件日志 #

Events:
Type Reason Age From Message

---

Normal Scheduled 29m default-scheduler Successfully assigned xxx/xxx-hooks-1234xl2 to aks-ncd-2810-vmss1q
Warning Failed 27m (x12 over 29m) kubelet Error: couldn't find key XXX_TOKEN in Secret xxx/xxx-token
Normal Pulled 4m26s (x119 over 29m) kubelet Container image "df3.azurecr.io/xxx-engine:23.6.0-123.gf12345" already present on machine

解决方案 #

从日志能直接看出，问题根源就是Pod要用到的Secret里缺了指定的键，按下面步骤解决：

1. 核对Secret的命名空间、名称和键名

   • 先确认Pod对应的Deployment/StatefulSet配置里，引用的Secret是不是xxx/xxx-token，要的键是不是XXX_TOKEN——Kubernetes对名称和键名是大小写敏感的，别写错。
   • 用命令查看现有Secret的内容：

     kubectl get secret xxx-token -n xxx -o yaml
     

     看输出里的data字段下有没有XXX_TOKEN这个键。

2. 补全Secret里的缺失键

   • 如果Secret还没创建，直接带键创建：

     # 直接输入值创建
     kubectl create secret generic xxx-token -n xxx --from-literal=XXX_TOKEN="你的token值"
     # 从文件读取值创建
     kubectl create secret generic xxx-token -n xxx --from-file=XXX_TOKEN=/本地文件路径/token.txt
     

   • 如果Secret已经存在，添加缺失的键：

     kubectl patch secret xxx-token -n xxx -p '{"data": {"XXX_TOKEN": "'$(echo -n "你的token值" | base64 -w 0)'"}}'
     

     注意：Secret的data字段值必须是Base64编码的，用--from-literal创建会自动编码，手动patch需要自己处理编码。

3. 检查Pod配置里的Secret引用

   • 查看Deployment的配置，确认环境变量引用Secret的部分没错：

     kubectl get deployment xxx-hooks -n xxx -o yaml
     

     找spec.template.spec.containers[*].env里的相关配置，确保secretKeyRef.name是xxx-token，secretKeyRef.key是XXX_TOKEN，和实际Secret一致。

4. 重启Pod让配置生效

   • 修改Secret后，旧Pod不会自动加载新配置，要么重启Deployment：

     kubectl rollout restart deployment xxx-hooks -n xxx
     

     要么直接删了出错的Pod，让控制器重新创建：

     kubectl delete pod xxx/xxx-hooks-1234xl2
     

内容的提问来源于stack exchange，提问作者user5539903