Azure Kubernetes集群Pod出现ImagePullBackOff问题求助

问题现象 #

Azure Kubernetes集群中部署的应用Pod突发ImagePullBackOff状态，应用此前数月运行正常，当前Pod处于Pending状态无法启动。

Pod YAML状态信息 #

执行命令：kubectl get <pod-name> -n namespace -o yaml
返回关键状态片段：

containerStatuses:
  - image: mycontainer-registry.io/company/my-app:1.0.0-integration-62c7e30532bd430477731a01a962372166fd5644
    imageID: ""
    lastState: {}
    name: dmd-base
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: Back-off pulling image "mycontainer-registry.io/company/my-app:1.0.0-integration-62c7e30532bd430477731a01a962372166fd5644"
        reason: ImagePullBackOff
  hostIP: x.x.x.53
  phase: Pending
  podIP: x.x.x.237

Pod描述错误日志 #

执行命令：kubectl describe pod <pod-name> -n namespace
返回关键错误片段：

Normal   Scheduled  85m                  default-scheduler  Successfully assigned dmd-int/app-app-base-5b4b75756c-lrcp6 to aks-agentpool-35064155-vmss00000a
Warning  Failed     85m                  kubelet            Failed to pull image "mycontainer-registry.io/company/my-app:1.0.0-integration-62c7e30532bd430477731a01a962372166fd5644": 
[rpc error: code = Unknown desc = failed to pull and unpack image "mycontainer-registry.io/company/my-app:1.0.0-integration-62c7e30532bd430477731a01a962372166fd5644": 
failed to resolve reference "mycontainer-registry.io/commpany/my-app:1.0.0-integration-62c7e30532bd430477731a01a962372166fd5644": 
failed to do request: Head "https://mycontainer-registry.azurecr.io/v2/company/my-app/manifests/1.0.0-integration-62c7e30532bd430477731a01a962372166fd5644": 
dial tcp: lookup mycontainer-registry.azurecr.io on [::1]:53: read udp [::1]:56109->[::1]:53: read: connection refused, 
rpc error: code = Unknown desc = failed to pull and unpack image "mycontainer-registry.io/company/my-app:1.0.0-integration-62c7e30532bd430477731a01a962372166fd5644": 
failed to resolve reference "mycontainer-registry.io/company/my-app:1.0.0-integration-62c7e30532bd430477731a01a962372166fd5644": 
failed to do request: Head "https://mycontainer-registry.io/v2/company/my-app/manifests/1.0.0-integration-62c7e30532bd430477731a01a962372166fd5644": 
dial tcp: lookup mycontainer-registry.io on [::1]:53: read udp [::1]:60759->[::1]:53: read: connection refused]

排查方向与步骤 #

日志核心错误dial tcp: lookup ... on [::1]:53: read udp ... connection refused表明，AKS节点的DNS解析完全失效，无法解析容器镜像仓库域名。以下是针对性排查步骤：

1. 检查节点DNS配置 #

• 登录故障节点aks-agentpool-35064155-vmss00000a，查看DNS配置：

  cat /etc/resolv.conf
  

  正常应配置Azure DNS（168.63.129.16）或其他可用DNS服务器，若显示nameserver ::1（本地回环），说明DNS配置异常。

• 验证节点域名解析能力：

  nslookup mycontainer-registry.io
  nslookup mycontainer-registry.azurecr.io
  

  若解析失败，确认节点DNS服务故障或网络无法访问DNS服务器。

2. 检查集群DNS组件状态 #

• 查看kube-dns/coredns Pod运行状态：

  kubectl get pods -n kube-system -l k8s-app=kube-dns
  kubectl describe pods -n kube-system -l k8s-app=kube-dns
  

  确保DNS Pod处于Running状态，无重启、CrashLoopBackOff等异常。

• 检查coredns配置：

  kubectl get configmap coredns -n kube-system -o yaml
  

  确认配置无错误转发规则或异常修改。

3. 检查节点网络连通性 #

• 测试节点到DNS服务器的连通性（以Azure DNS为例）：

  ping 168.63.129.16
  telnet 168.63.129.16 53
  

  若无法连通，检查节点子网的网络安全组（NSG）是否允许出站53端口（UDP/TCP）流量。

• 测试节点到镜像仓库的连通性：

  telnet mycontainer-registry.io 443
  curl -I https://mycontainer-registry.io/v2/
  

  若无法连通，检查NSG是否允许到镜像仓库的出站443端口流量，或路由表是否存在异常。

4. 验证镜像仓库与权限 #

• 确认目标镜像存在于仓库：通过Azure Portal或命令验证：

  az acr repository show --name mycontainer-registry --repository company/my-app --tag 1.0.0-integration-62c7e30532bd430477731a01a962372166fd5644
  

• 检查AKS集群访问ACR的权限：

  az aks check-acr --name <aks-cluster-name> --resource-group <rg-name> --acr mycontainer-registry.azurecr.io
  

  若权限不足，重新配置ACR与AKS的权限关联。

5. 临时修复方案 #

若DNS配置异常，可临时修改节点DNS配置并重启kubelet：

# 添加可用DNS服务器到resolv.conf
echo "nameserver 8.8.8.8" >> /etc/resolv.conf
# 重启kubelet
systemctl restart kubelet

之后删除故障Pod触发重新拉取：

kubectl delete pod <pod-name> -n namespace

内容的提问来源于stack exchange，提问作者Siya Nxuseka