Microk8s集群Pod无法解析域名访问外网故障咨询

问题背景 #

Ubuntu服务器同时部署Microk8s与Docker：

• Docker原生启动的容器可正常ping通外网域名，网络连通性正常
• Microk8s集群内Pod直接ping公网IP连通正常，但ping外网域名返回bad address报错，属于域名解析故障

相关环境信息 #

宿主机docker0网卡状态 #

~$ sudo ip addr show docker0
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:a7:9f:15:48 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:a7ff:fe8f:1548/64 scope link
       valid_lft forever preferred_lft forever

Docker容器连通性验证结果 #

~$ sudo docker run --rm -it ubuntu:trusty bash
root@dd0af86b1209:/# ip addr show eth0
158: eth0@if159: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
root@dd0af86b1209:/# ping google.com
PING google.com (142.250.179.110) 56(84) bytes of data.
64 bytes from par21s20-in-f14.1e100.net (142.250.179.110): icmp_seq=1 ttl=108 time=3.71 ms
64 bytes from par21s20-in-f14.1e100.net (142.250.179.110): icmp_seq=2 ttl=108 time=3.70 ms
64 bytes from par21s20-in-f14.1e100.net (142.250.179.110): icmp_seq=3 ttl=108 time=3.74 ms
64 bytes from par21s20-in-f14.1e100.net (142.250.179.110): icmp_seq=4 ttl=108 time=3.75 ms
64 bytes from par21s20-in-f14.1e100.net (142.250.179.110): icmp_seq=5 ttl=108 time=3.76 ms

Microk8s Pod内异常表现 #

/ # ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 0:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: eth0@if146: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1440 qdisc noqueue state UP
    link/ether ba:03:bd:4b:66:97 brd ff:ff:ff:ff:ff:ff
    inet 172.17.159.19/32 brd 172.17.159.19 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::b803:bdff:fe44:6697/64 scope link
       valid_lft forever preferred_lft forever

/ # ping google.com
ping: bad address 'google.com'

当前UFW防火墙规则 #

Anywhere (v6) on cali+     ALLOW       Anywhere (v6)
Anywhere (v6) on cni0      ALLOW       Anywhere (v6)
Anywhere (v6) on cbr0      ALLOW       Anywhere (v6)
Anywhere (v6) on eth0      ALLOW       Anywhere (v6)

当前coredns ConfigMap配置 #

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        log . {
          class error
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . 8.8.8.8 8.8.4.4 
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors\n    health {\n      lameduck 5s\n    }\n    ready\n    log . {\n      class error\n    }\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward . 8.8.8.8 8.8.4.4 \n    cache 30\n    loop\n    reload\n    loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists","k8s-app":"kube-dns"},"name":"coredns","namespace":"kube-system"}}
  creationTimestamp: "2022-06-19T17:07:02Z"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system
  resourceVersion: "7503127"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 0735a387-6970-43ab-8490-cdf49a23f936

修复步骤 #

1. 补全UFW防火墙规则 #

当前规则仅放行了IPv6流量，未放行IPv4下CNI相关网卡的转发流量，会导致Pod到coredns的DNS请求（UDP 53端口）被拦截，执行以下命令补全规则：

sudo ufw allow in on cni0
sudo ufw allow out on cni0
sudo ufw allow in on cali+
sudo ufw allow out on cali+
sudo ufw allow in on cbr0
sudo ufw allow out on cbr0
sudo ufw default allow routed
sudo ufw reload

2. 检查coredns运行状态 #

执行命令确认coredns Pod处于正常Running状态：

microk8s kubectl get pods -n kube-system -l k8s-app=kube-dns

如果Pod状态异常，查看对应Pod日志排查启动问题，常见原因为端口占用、配置格式错误。

3. 修正coredns上游DNS配置 #

当前coredns配置将解析请求转发至8.8.8.8、8.8.4.4，若服务器网络无法访问Google DNS，会直接导致解析失败：

• 先在宿主机执行cat /etc/resolv.conf查看宿主机使用的DNS地址
• 执行microk8s kubectl edit configmap coredns -n kube-system编辑配置，将Corefile中forward . 8.8.8.8 8.8.4.4行替换为forward . /etc/resolv.conf（自动继承宿主机DNS配置），或替换为当前网络可访问的公共DNS地址（如国内环境可使用223.5.5.5、119.29.29.29）
• 保存退出后coredns会自动加载配置，也可手动重启coredns生效：microk8s kubectl rollout restart deployment coredns -n kube-system

4. 修复iptables规则冲突 #

同机部署Docker和Microk8s时，Docker生成的iptables规则可能拦截CNI转发流量，执行以下命令重建Microk8s网络规则：

microk8s kubectl rollout restart daemonset kube-proxy -n kube-system

若仍未恢复，重启Microk8s服务即可：

sudo microk8s stop && sudo microk8s start

5. 验证修复 #

进入测试Pod执行ping google.com，能正常解析IP并收到回包即修复完成。 #

内容的提问来源于stack exchange，提问作者BOUZIDI IDRISSI Mehdi