kubeadm worker节点Pod运行时出现Temporary failure in name resolution报错

VMware环境下搭建了包含1个控制平面、1个Worker节点的Kubernetes集群，通过Strimzi部署Kafka集群。控制平面节点上的客户端可正常与Kafka通信，但Worker节点上的客户端发起请求时抛出域名解析失败错误。

客户端错误日志 #

%3|1638529687.405|FAIL|apollo-prototype-765f4d8bcf-bjpf4#producer-2| [thrd:sasl_plaintext://my-cluster-kafka-bootstrap:9092/bootstrap]: sasl_plaintext://my-cluster-kafka-bootstrap:9092/bootstrap: Failed to resolve 'my-cluster-kafka-bootstrap:9092': Temporary failure in name resolution (after 20016ms in state CONNECT, 2 identical error(s) suppressed)
%3|1638529687.406|ERROR|apollo-prototype-765f4d8bcf-bjpf4#producer-2| [thrd:app]: apollo-prototype-765f4d8bcf-bjpf4#producer-2: sasl_plaintext://my-cluster-kafka-bootstrap:9092/bootstrap: Failed to resolve 'my-cluster-kafka-bootstrap:9092': Temporary failure in name resolution (after 20016ms in state CONNECT, 2 identical error(s) suppressed)

Strimzi Kafka集群配置 #

listeners:
  - name: plain
    port: 9092
    type: internal
    tls: false
    authentication:
      type: scram-sha-512
  - name: external
    port: 9094
    type: ingress
    tls: true
    authentication:
      type: scram-sha-512
    configuration:
      class: nginx
      bootstrap:
        host: localb.kafka.xxx.com
      brokers:
      - broker: 0
        host: local.kafka.xxx.com

已知排查信息 #

• 完全相同的配置在云环境中运行无任何问题
• 在两个节点上执行Telnet、nslookup均会报错
• CoreDNS日志中没有相关错误记录
• 两个节点的防火墙均已关闭

异常根因 #

Worker节点上的Calico Pod未发生崩溃，但存在bird: Netlink: Network is down报错，经排查故障由节点加载的ipvs模块不一致导致：新节点配置了ipip模块，但旧节点未加载该模块，引发Calico网络异常，最终导致域名解析失败。

Calico异常日志 #

2021-12-03 09:39:58.051 [INFO][90] felix/int_dataplane.go 1539: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"tunl0", Addrs:set.mapSet{}}
2021-12-03 09:39:58.051 [INFO][90] felix/hostip_mgr.go 85: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"tunl0", Addrs:set.mapSet{}}
2021-12-03 09:39:58.052 [INFO][90] felix/ipsets.go 130: Queueing IP set for creation family="inet" setID="this-host" setType="hash:ip"
2021-12-03 09:39:58.057 [INFO][90] felix/ipsets.go 785: Doing full IP set rewrite family="inet" numMembersInPendingReplace=3 setID="this-host"
2021-12-03 09:39:58.059 [INFO][90] felix/int_dataplane.go 1036: Linux interface state changed. ifIndex=13 ifaceName="tunl0" state="down"
2021-12-03 09:39:58.082 [INFO][90] felix/int_dataplane.go 1521: Received interface update msg=&intdataplane.ifaceUpdate{Name:"tunl0", State:"down", Index:13}
bird: Netlink: Network is down

修复步骤 #

1. 检查节点是否加载ipip模块

lsmod | grep ipip

正常输出示例：

ipip                   16384  0 
tunnel4                16384  1 ipip
ip_tunnel              24576  1 ipip

2. 卸载ipip模块

modprobe -r ipip

3. 验证模块是否卸载完成，执行以下命令无输出即为卸载成功

lsmod | grep ipip

执行完上述操作后问题即可恢复。

内容的提问来源于stack exchange，提问作者Oana