GCE环境下Kubernetes Pods无法解析DNS问题求助

环境信息 #

• 节点DNS解析正常
• kube-proxy运行状态正常
• 集群采用CoreDNS作为DNS服务
• 基础设施部署在GCE平台

CoreDNS日志 #

[INFO] 10.240.0.22:35741 - 24318 "A IN google.com.cluster.local. udp 38 false 512" NXDOMAIN qr,aa,rd 131 0.000250358s
[INFO] 10.240.0.22:38607 - 32094 "A IN google.com.europe-central2-a.c.massive-pen-404512.internal. udp 72 false 512" NOERROR - 0 0.000072616s
[ERROR] plugin/errors: 2 google.com.europe-central2-a.c.massive-pen-404512.internal. A: plugin/loop: no next plugin found

Pod解析google.com结果 #

/app # nslookup google.com
Server:         10.32.0.10
Address:        10.32.0.10:53

** server can't find google.com: SERVFAIL

** server can't find google.com: SERVFAIL

Pod解析节点worker-1结果 #

/app # nslookup worker-1
Server:         10.32.0.10
Address:        10.32.0.10:53

** server can't find worker-1.cluster.local: NXDOMAIN

** server can't find worker-1.default.svc.cluster.local: NXDOMAIN

** server can't find worker-1.svc.cluster.local: NXDOMAIN

** server can't find worker-1.default.svc.cluster.local: NXDOMAIN

** server can't find worker-1.svc.cluster.local: NXDOMAIN

** server can't find worker-1.cluster.local: NXDOMAIN

** server can't find worker-1.europe-central2-a.c.massive-pen-404512.internal: SERVFAIL

** server can't find worker-1.google.internal: SERVFAIL

** server can't find worker-1.europe-central2-a.c.massive-pen-404512.internal: SERVFAIL

** server can't find worker-1.c.massive-pen-404512.internal: SERVFAIL

** server can't find worker-1.google.internal: SERVFAIL

** server can't find worker-1.c.massive-pen-404512.internal: SERVFAIL

Pod内/etc/resolv.conf配置 #

/app # cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local europe-central2-a.c.massive-pen-404512.internal c.massive-pen-404512.internal google.internal
nameserver 10.32.0.10
options ndots:5

公共DNS解析验证 #

使用公共DNS（8.8.8.8）可正常解析外部域名：

/app # nslookup example.com 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8:53

Non-authoritative answer:
Name:   example.com
Address: 93.184.216.34

Non-authoritative answer:
Name:   example.com
Address: 2606:2800:220:1:248:1893:25c8:1946

问题分析 #

从CoreDNS日志中的plugin/loop: no next plugin found错误可以明确：当CoreDNS处理GCE内部域名（如*.europe-central2-a.c.massive-pen-404512.internal）时，没有配置对应的转发逻辑，导致解析流程出现循环，最终返回SERVFAIL。

结合Pod的resolv.conf配置，搜索域包含多个GCE内部域名后缀，当Pod解析外部域名（如google.com）时，会按照ndots:5的规则依次拼接搜索域尝试解析，当尝试到GCE内部域时，CoreDNS无法处理该类域名的解析请求，进而导致整个解析失败返回SERVFAIL。

解决方案 #

调整CoreDNS配置，添加GCE内部域名转发 #

修改kube-system命名空间下的coredns ConfigMap，为GCE内部域名添加专门的转发规则，指向GCE内部DNS服务器（通常为169.254.169.254）：

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    europe-central2-a.c.massive-pen-404512.internal:53 {
        forward . 169.254.169.254
        cache 30
    }
    c.massive-pen-404512.internal:53 {
        forward . 169.254.169.254
        cache 30
    }
    google.internal:53 {
        forward . 169.254.169.254
        cache 30
    }

重启CoreDNS生效 #

执行命令重启CoreDNS Deployment，使配置生效：

kubectl rollout restart deployment coredns -n kube-system

内容的提问来源于stack exchange，提问作者Ivan