AKS集群中同StatefulSet内Pod主机名解析失败求助

问题背景 #

在AKS的default命名空间部署了副本数为2的StatefulSet运行JBoss应用，搭配三个同命名空间的Service：

• demo-app-hs（Headless Service）：无ClusterIP，关联demo-app-depl-0和demo-app-depl-1两个Pod
• demo-app-service（普通Service）：带ClusterIP和ExternalIP，关联上述两个Pod
• demo-app-service-lb（Azure LoadBalancer类型Service）：带ClusterIP和ExternalIP，关联上述两个Pod

第一个Pod demo-app-depl-0运行正常，但第二个Pod demo-app-depl-1启动后日志报错无法连接主Pod：

[exec] 2023-11-24 04:45:13.378+0000 ERROR [org.apache.activemq.artemis.core.client:877] {} (Thread-28 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@21918ea4)) AMQ214016: Failed to create netty connection: java.net.UnknownHostException: demo-app-depl-0

排查信息 #

1. Pod内DNS配置
   进入demo-app-depl-1查看/etc/resolv.conf内容：

search default.svc.cluster.local svc.cluster.local cluster.local

nameserver 10.0.0.10

options ndots:5

2. 域名解析测试

• 执行kubectl exec -i -t demo-app-depl-1 -- nslookup default.svc.cluster.local返回无结果：

Server: 10.0.0.10

Address: 10.0.0.10#53

*** Can't find default.svc.cluster.local: No answer

• 执行kubectl exec -i -t demo-app-depl-1 -- nslookup demo-app-hs.default.svc.cluster.local可正常解析到两个Pod的IP：

Server: 10.0.0.10

Address: 10.0.0.10#53

Name: demo-app-hs.default.svc.cluster.local

Address: 10.244.2.6

Name: demo-app-hs.default.svc.cluster.local

Address: 10.244.2.7

3. 临时解决方法
   手动修改demo-app-depl-1的/etc/resolv.conf添加demo-app-hs.default.svc.cluster.local可临时恢复解析，但非长久之计，且此前相同部署方式无需此操作。

相关配置文件 #

Headless Service配置 #

apiVersion: v1
kind: Service
metadata:
  name: {{ include "deploy.fullname" . }}-hs
  labels:
    {{- include "deploy.labels" . | nindent 4 }}
spec:
  selector:
    app: {{ include "deploy.fullname" . }}-app-label
#  type: LoadBalancer
  clusterIP: None
  ports:
    ...

StatefulSet配置 #

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: {{ include "deploy.fullname" . }}-depl
  namespace: {{ .Values.application.namespace }}
  labels:
    {{- include "deploy.labels" . | nindent 4 }}
    app: {{ include "deploy.fullname" . }}-app-label
    date: "{{ now | unixEpoch }}"
spec:
  replicas: {{ .Values.application.replicas }}
  serviceName: {{ include "deploy.fullname" . }}-hs
  selector:
    matchLabels:
      {{- include "deploy.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "deploy.selectorLabels" . | nindent 8 }}
        app: {{ include "deploy.fullname" . }}-app-label
    spec:
      serviceAccountName: {{ include "deploy.fullname" . }}-clustering-service-account
      securityContext:
        fsGroup: 1067
      containers:
        - image: {{ .Values.application.image }}:{{ .Values.application.version }}
          name: {{ .Values.application.containerName }}
          command: ["ant"]
          args: ['run', '-Dcontext-root={{ include "deploy.contextRoot" . }}', '-Dfile.encoding=utf8', '-Denv-name={{ .Values.application.envName }}', '-Dcacerts=/opt/jboss/standalone/configuration/cacerts']
          imagePullPolicy: {{ .Values.application.imagePullPolicy }}
          {{ if .Values.application.processLarge }}
          resources:
            requests:
              cpu: 2000m
              memory: 4096Mi
            limits:
              cpu: 3000m
              memory: 8192Mi
          {{ end }}
          ports:
            ...
          env:
            - name: ENV_NAME
              valueFrom:
                configMapKeyRef:
                  name: {{ include "deploy.fullname" . }}-config-map
                  key: env-name
            # ... other env variables ...
          volumeMounts:
            - name: jboss-data-vol
              mountPath: /opt/jboss/standalone/data
            - name: jboss-log-vol
              mountPath: /opt/jboss/standalone/log
         ...

可能的解决方向 #

1. 检查CoreDNS状态

   • 执行kubectl get pods -n kube-system -l k8s-app=kube-dns确认CoreDNS Pod是否全部正常运行
   • 查看CoreDNS日志：kubectl logs -n kube-system -l k8s-app=kube-dns，排查是否有解析异常、权限或资源不足问题

2. 验证标签匹配

   • 确认StatefulSet Pod的标签app: {{ include "deploy.fullname" . }}-app-label与Headless Service的selector完全一致，标签不匹配会导致DNS记录无法生成
   • 执行kubectl describe pod demo-app-depl-1查看Pod的标签，对比Headless Service的selector

3. 持久化DNS搜索域配置
   在StatefulSet的spec.template.spec中添加dnsConfig字段，让Pod启动时自动包含所需搜索域：

   dnsConfig:
     searches:
       - default.svc.cluster.local
       - svc.cluster.local
       - cluster.local
       - demo-app-hs.default.svc.cluster.local
   

4. 排查集群配置变更
   确认近期是否有AKS集群版本升级、CoreDNS版本更新或kube-dns配置修改，这些变更可能影响域名解析逻辑

5. 确认StatefulSet的ServiceName配置
   检查StatefulSet的serviceName是否正确指向Headless Service的名称（{{ include "deploy.fullname" . }}-hs），该字段是StatefulSet生成稳定DNS记录的核心配置

内容的提问来源于stack exchange，提问作者Brian Lui