Kubernetes Pod无法互访：域名解析临时失败问题咨询

我通过以下清单在default命名空间部署了Fluent-bit DaemonSet和带NodePort Service的Elasticsearch：

部署清单文件 #

fluent-bit-daemonset.yml #

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
      version: v1
      kubernetes.io/cluster-service: "true"
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:0.14.2
        ports:
        - containerPort: 2020
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch-0.elasticsearch.default.svc.cluster.local"
        - name: FLUENT_ELASTICSEARCH_PORT 
          value: "9200"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
        - name: mnt
          mountPath: /mnt
          readOnly: true
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      - name: mnt
        hostPath:
          path: /mnt
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule

elasticsearch.yml #

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app : elasticsearch
    component: elasticsearch
    release: elasticsearch
  name: elasticsearch
spec:
  podManagementPolicy: Parallel
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app : elasticsearch
      component: elasticsearch
      release: elasticsearch
  serviceName: elasticsearch
  template:
    metadata:
      creationTimestamp: null
      labels:
        app : elasticsearch
        component: elasticsearch
        release: elasticsearch
    spec:
      containers:
      - env:
        - name: cluster.name
          value: dev-cluster
        - name: discovery.type
          value: single-node
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        - name: bootstrap.memory_lock
          value: "false"
        - name: xpack.security.enabled
          value: "false"
        image: elasticsearch:8.12.0
        imagePullPolicy: IfNotPresent
        name: elasticsearch
        ports:
        - containerPort: 9200
          name: http
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        resources:
          limits:
            cpu: 1
            memory: 3Gi
          requests:
            cpu: 250m
            memory: 512Mi
        securityContext:
          privileged: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: elasticsearch-data
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - sh
        - -c
        - chown -R 1000:1000 /usr/share/elasticsearch/data
        - sysctl -w vm.max_map_count=262144
        - chmod 777 /usr/share/elasticsearch/data
        - chomod 777 /usr/share/elasticsearch/data/node
        - chmod g+rwx /usr/share/elasticsearch/data
        - chgrp 1000 /usr/share/elasticsearch/data
        image: busybox:1.29.2
        imagePullPolicy: IfNotPresent
        name: set-dir-owner
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: elasticsearch-data
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 10
  updateStrategy:
    type: OnDelete
  volumeClaimTemplates:
  - metadata:
      creationTimestamp: null
      name: elasticsearch-data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi

elastic-service.yml #

---
apiVersion: v1
kind: Service
metadata:
  name: eks-srv
spec:
  selector:
    app: elasticsearch
    component: elasticsearch
  ports:
    - name: db
      protocol: TCP
      port: 9200
      targetPort: 9200
    - name: monitoring
      protocol: TCP
      port: 9300
      targetPort: 9300
  type: NodePort

错误日志 #

执行命令k logs pods/fluent-bit-bwcl6查看Fluent-bit Pod日志，出现大量DNS解析失败错误：

[2024/01/31 04:58:05] [ info] [engine] started (pid=1)
[2024/01/31 04:58:06] [ info] [filter_kube] https=1 host=kubernetes.default.svc.cluster.local port=443
[2024/01/31 04:58:06] [ info] [filter_kube] local POD info OK
[2024/01/31 04:58:06] [ info] [filter_kube] testing connectivity with API server...
[2024/01/31 04:58:06] [ warn] net_tcp_fd_connect: getaddrinfo(host='kubernetes.default.svc.cluster.local'): Temporary failure in name resolution
[2024/01/31 04:58:06] [error] [filter_kube] upstream connection error
[2024/01/31 04:58:06] [ warn] [filter_kube] could not get meta for POD fluent-bit-bwcl6
[2024/01/31 04:58:06] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/01/31 04:58:11] [ warn] net_tcp_fd_connect: getaddrinfo(host='kubernetes.default.svc.cluster.local'): Temporary failure in name resolution
[2024/01/31 04:58:11] [error] [filter_kube] upstream connection error
[2024/01/31 04:58:21] [ warn] net_tcp_fd_connect: getaddrinfo(host='kubernetes.default.svc.cluster.local'): Temporary failure in name resolution
[2024/01/31 04:58:21] [error] [filter_kube] upstream connection error
[2024/01/31 04:58:21] [ warn] net_tcp_fd_connect: getaddrinfo(host='kubernetes.default.svc.cluster.local'): Temporary failure in name resolution
[2024/01/31 04:58:21] [error] [filter_kube] upstream connection error
[2024/01/31 04:58:31] [ warn] net_tcp_fd_connect: getaddrinfo(host='kubernetes.default.svc.cluster.local'): Temporary failure in name resolution
[2024/01/31 04:58:31] [error] [filter_kube] upstream connection error
getaddrinfo(host='kubernetes.default.svc.cluster.local'): Temporary failure in name resolution
[2024/01/31 04:58:41] [error] [filter_kube] upstream connection error
[2024/01/31 04:58:51] [ warn] net_tcp_fd_connect: getaddrinfo(host='kubernetes.default.svc.cluster.local'): Temporary failure in name resolution
[2024/01/31 04:58:51] [error] [filter_kube] upstream connection error
[2024/01/31 04:59:16] [ warn] net_tcp_fd_connect: getaddrinfo(host='elasticsearch-0.elasticsearch.default.svc.cluster.local'): Temporary failure in name resolution
[2024/01/31 04:59:21] [ warn] net_tcp_fd_connect: getaddrinfo(host='elasticsearch-0.elasticsearch.default.svc.cluster.local'): Temporary failure in name resolution
[2024/01/31 04:59:27] [ warn] net_tcp_fd_connect: getaddrinfo(host='elasticsearch-0.elasticsearch.default.svc.cluster.local'): Temporary failure in name resolution
[2024/01/31 04:59:37] [ warn] net_tcp_fd_connect: getaddrinfo(host='kubernetes.default.svc.cluster.local'): Temporary failure in name resolution
[2024/01/31 04:59:37] [error] [filter_kube] upstream connection error
[2024/01/31 04:59:37] [ warn] net_tcp_fd_connect: getaddrinfo(host='kubernetes.default.svc.cluster.local'): Temporary failure in name resolution
[2024/01/31 04:59:37] [error] [filter_kube] upstream connection error
[2024/01/31 04:59:47] [ warn] net_tcp_fd_connect: getaddrinfo(host='kubernetes.default.svc.cluster.local'): Temporary failure in name resolution
[2024/01/31 04:59:47] [error] [filter_kube] upstream connection error
[2024/01/31 04:59:47] [ warn] net_tcp_fd_connect: getaddrinfo(host='kubernetes.default.svc.cluster.local'): Temporary failure in name resolution
[2024/01/31 04:59:47] [error] [filter_kube] upstream connection error
[2024/01/31 04:59:57] [ warn] net_tcp_fd_connect: getaddrinfo(host='elasticsearch-0.elasticsearch.default.svc.cluster.local'): Temporary failure in name resolution

问题原因分析 #

1. Elasticsearch域名配置不匹配：Fluent-bit使用的elasticsearch-0.elasticsearch.default.svc.cluster.local是StatefulSet对应的Headless Service域名，但你的StatefulSet指定serviceName: elasticsearch，却未创建同名的Headless Service，实际运行的Service名称是eks-srv，导致该域名无法解析。
2. 集群DNS服务异常：kubernetes.default.svc.cluster.local是K8s API Server的默认域名，该域名解析失败说明集群CoreDNS组件可能未正常运行，或Pod的DNS配置存在问题。
3. Fluent-bit版本过旧：使用的fluent/fluent-bit:0.14.2是多年前的老版本，存在DNS解析相关的已知bug，与新版K8s集群兼容性较差。

解决步骤 #

1. 修复Elasticsearch访问域名 #

选择以下两种方式之一：

• 方式一：创建Headless Service：新增与StatefulSetserviceName匹配的Headless Service：

  apiVersion: v1
  kind: Service
  metadata:
    name: elasticsearch
  spec:
    selector:
      app: elasticsearch
      component: elasticsearch
    ports:
    - port: 9200
      name: http
    - port: 9300
      name: transport
    clusterIP: None # 标记为Headless Service
  

• 方式二：修改Fluent-bit配置：将FLUENT_ELASTICSEARCH_HOST的值改为现有Service的域名eks-srv.default.svc.cluster.local。

2. 检查并修复CoreDNS状态 #

执行命令查看CoreDNS Pod运行状态：

kubectl get pods -n kube-system -l k8s-app=kube-dns

若Pod未处于Running状态，查看CoreDNS日志排查故障，确保集群DNS服务正常运行。

3. 升级Fluent-bit版本 #

将Fluent-bit镜像更新为最新稳定版（如fluent/fluent-bit:2.2.0），老版本的DNS解析逻辑存在缺陷，升级后可解决大部分兼容性问题。

4. 验证Pod DNS配置 #

进入Fluent-bit Pod查看DNS配置：

kubectl exec -it fluent-bit-bwcl6 -- cat /etc/resolv.conf

确认nameserver指向K8s集群的DNS Service IP（通常为10.96.0.10），若配置异常，检查Pod的dnsPolicy是否为默认的ClusterFirst。

内容的提问来源于stack exchange，提问作者Behnia FB