Azure Kubernetes Services负载均衡器入站UDP连接异常排查求助

问题背景 #

对Kubernetes及Azure Kubernetes Services（AKS）不熟悉，现有AKS集群中部署了telemetry asterix adapter Pod，用于通过公网IP接收ADSB传感器的UDP数据。已基于微软通用YAML创建同命名空间的公网IP与LoadBalancer Service，可ping通该公网IP，ADSB传感器链路已按承包商配置完成，但Pod日志中无任何数据包记录。当前Service使用源端口1025、目标端口6000，与Pod内NettyUDP监听端口一致，且已通过selector配置Service与Pod的关联。已尝试修改Service YAML、移除源IP白名单、用其他设备发送测试UDP包、调整NSG等操作，问题仍未解决，怀疑LoadBalancer Service未正确关联目标Pod。

相关配置 #

Service配置 #

kind: Service
apiVersion: v1
metadata:
  name: telemetry-asterix-adapter-svc
  namespace: utm
  uid: fac3e2f1-50e1-49f3-9624-2b49fe5bec39
  resourceVersion: '15394560'
  creationTimestamp: '2025-03-04T18:39:58Z'
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"telemetry-asterix-adapter-svc","namespace":"utm"},"spec":{"loadBalancerSourceRanges":["71.###.###.###/32","71.###.###.###/32"],"ports":[{"port":1025,"protocol":"UDP","targetPort":6000}],"selector":{"app":"telemetry-asterix-adapter"},"type":"LoadBalancer"}}
  finalizers:
    - service.kubernetes.io/load-balancer-cleanup
  managedFields:
    - manager: cloud-controller-manager
      operation: Update
      apiVersion: v1
      time: '2025-03-11T16:08:39Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers:
            .: {}
            v:"service.kubernetes.io/load-balancer-cleanup": {}
        f:status:
          f:loadBalancer:
            f:ingress: {}
      subresource: status
    - manager: kubectl-client-side-apply
      operation: Update
      apiVersion: v1
      time: '2025-03-11T20:13:05Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:spec:
          f:allocateLoadBalancerNodePorts: {}
          f:externalTrafficPolicy: {}
          f:internalTrafficPolicy: {}
          f:loadBalancerSourceRanges: {}
          f:ports:
            .: {}
            k:{"port":1025,"protocol":"UDP"}:
              .: {}
              f:port: {}
              f:protocol: {}
              f:targetPort: {}
          f:selector: {}
          f:sessionAffinity: {}
          f:type: {}
spec:
  ports:
    - protocol: UDP
      port: 1025
      targetPort: 6000
      nodePort: 31780
  selector:
    app: telemetry-asterix-adapter
  clusterIP: 10.0.203.107
  clusterIPs:
    - 10.0.203.107
  type: LoadBalancer
  sessionAffinity: None
  loadBalancerSourceRanges:
    - 71.###.###.###/32
    - 71.###.###.###/32
  externalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  allocateLoadBalancerNodePorts: true
  internalTrafficPolicy: Cluster
status:
  loadBalancer:
    ingress:
      - ip: 62.##.##.###
        ipMode: VIP

Pod清单 #

Name:             telemetry-asterix-adapter-f8bb6f48d-2mqf6
Namespace:        utm
Priority:         0
Service Account:  default
Node:             aks-nodepool1-25615987-vmss000001/10.64.80.12
Start Time:       Thu, 13 Mar 2025 13:09:14 +0000
Labels:           app=telemetry-asterix-adapter
                  pod-template-hash=f8bb6f48d
Annotations:      kubectl.kubernetes.io/restartedAt: 2025-03-13T13:09:13Z
Status:           Running
IP:               10.64.82.134
IPs:
  IP:           10.64.82.134
Controlled By:  ReplicaSet/telemetry-asterix-adapter-f8bb6f48d
Containers:
  telemetry-asterix-adapter:
    Container ID:   containerd://88a01df213e0ec4732dee857798f61d73e9296b9f24ab4b1f61d7a6425c75e93
    Image:          crfusademousgv634.azurecr.us/utm-services/telemetry-asterix:3.5.0
    Image ID:       crfusademousgv634.azurecr.us/utm-services/telemetry-asterix@sha256:4c44d3b8946c6cecaa28d6637104b3f336776a4062f372a33a53238cec3a132f
    Ports:          6000/UDP, 8080/TCP
    Host Ports:     0/UDP, 0/TCP
    State:          Running
      Started:      Thu, 13 Mar 2025 13:09:15 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  512Mi
    Requests:
      memory:  512Mi
    Environment Variables from:
      telemetry-asterix-adapter  ConfigMap  Optional: false
    Environment:                 <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-g6r4q (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  kube-api-access-g6r4q:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

Pod日志 #

.   ____          _            __ _ _
 /\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.7.9)

2025-03-13 13:09:20.002  INFO 1 --- [           main] c.f.s.t.a.AsterixAdapterApp              : Starting AsterixAdapterApp v3.5.0 using Java 11.0.16 on telemetry-asterix-adapter-f8bb6f48d-2mqf6 with PID 1 (/opt/adapter/adapter.jar started by ? in /opt/adapter)
2025-03-13 13:09:20.018 DEBUG 1 --- [           main] c.f.s.t.a.AsterixAdapterApp              : Running with Spring Boot v2.7.9, Spring v5.3.25
2025-03-13 13:09:20.019  INFO 1 --- [           main] c.f.s.t.a.AsterixAdapterApp              : No active profile set, falling back to 1 default profile: "default"
2025-03-13 13:09:26.636  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2025-03-13 13:09:26.682  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2025-03-13 13:09:26.683  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.71]
2025-03-13 13:09:26.968  INFO 1 --- [           main] a.c.c.C.[.[.[/telemetry-asterix-adapter] : Initializing Spring embedded WebApplicationContext
2025-03-13 13:09:26.969  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 6760 ms
2025-03-13 13:09:28.505  INFO 1 --- [           main] c.f.s.t.asterixadapter.grpc.GrpcClient   : Create gRPC client at address: telemetry-manager-ng.utm.svc.cluster.local:8081
2025-03-13 13:09:38.298  INFO 1 --- [           main] o.a.c.c.s.CamelHttpTransportServlet      : Initialized CamelHttpTransportServlet[name=CamelServlet, contextPath=/telemetry-asterix-adapter]
2025-03-13 13:09:38.304  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path '/telemetry-asterix-adapter'
2025-03-13 13:09:40.204  INFO 1 --- [           main] o.a.c.component.netty.NettyComponent     : Creating shared NettyConsumerExecutorGroup with 3 threads
2025-03-13 13:09:40.551  INFO 1 --- [           main] c.n.SingleUDPNettyServerBootstrapFactory : ConnectionlessBootstrap binding to 0.0.0.0:6000
2025-03-13 13:09:40.837  INFO 1 --- [           main] o.a.camel.component.netty.NettyConsumer  : Netty consumer bound to: 0.0.0.0:6000
2025-03-13 13:09:40.841  INFO 1 --- [           main] o.a.c.impl.engine.AbstractCamelContext   : Routes startup (total:2 started:2)
2025-03-13 13:09:40.841  INFO 1 --- [           main] o.a.c.impl.engine.AbstractCamelContext   :     Started route1 (netty://UDP://0.0.0.0:6000)
2025-03-13 13:09:40.841  INFO 1 --- [           main] o.a.c.impl.engine.AbstractCamelContext   :     Started route2 (rest://post:telemetry)
2025-03-13 13:09:40.841  INFO 1 --- [           main] o.a.c.impl.engine.AbstractCamelContext   : Apache Camel 3.14.1 (camel-1) started in 2s483ms (build:178ms init:1s560ms start:745ms)
2025-03-13 13:09:40.980  INFO 1 --- [           main] c.f.s.t.a.AsterixAdapterApp              : Started AsterixAdapterApp in 23.126 seconds (JVM running for 25.782)

排查步骤 #

1. 确认Service与Pod的关联状态 #

执行命令查看Service对应的Endpoints：

kubectl get endpoints telemetry-asterix-adapter-svc -n utm

如果输出中包含Pod的IP（10.64.82.134），说明Service与Pod的标签匹配正常；如果Endpoints为空，检查Pod的labels是否与Service的selector完全一致。

2. 验证集群内UDP连通性 #

在utm命名空间创建临时测试Pod，测试集群内UDP流量是否能到达目标Pod：

kubectl run -it --rm udp-test --image=busybox -n utm

进入Pod后，发送测试UDP包到Service的ClusterIP和端口：

nc -u 10.0.203.107 1025

输入任意内容后回车，同时查看目标Pod的日志。如果集群内能收到包，说明问题出在公网到集群的链路；如果收不到，排查集群内部网络策略或Pod监听配置。

3. 检查Pod内部端口监听 #

进入目标Pod，确认6000端口正常监听：

kubectl exec -it telemetry-asterix-adapter-f8bb6f48d-2mqf6 -n utm -- netstat -ulpn

确认输出中存在0.0.0.0:6000的UDP监听记录，与日志中显示的绑定地址一致。

4. 验证NodePort的UDP连通性 #

找到Pod所在Node的公网IP，从外部设备向该IP的NodePort（31780）发送UDP测试包，查看Pod日志是否有记录：

• 如果能收到，说明Azure LoadBalancer到Node的转发存在问题，检查LoadBalancer的配置或Azure侧的路由规则；
• 如果收不到，检查Node所在子网的NSG是否允许UDP 31780端口入站，以及Node内部的防火墙规则。

5. 抓包定位流量丢失点 #

• 传感器侧抓包：确认UDP包已发送至LoadBalancer公网IP（62.##.##.###）的1025端口；
• AKS Node侧抓包：在Pod所在Node上执行抓包命令，查看是否收到UDP包：

  kubectl exec -it aks-nodepool1-25615987-vmss000001 -n kube-system -- tcpdump -i any udp port 31780
  

• Pod侧抓包：进入Pod内部抓包，确认流量是否到达Pod：

  kubectl exec -it telemetry-asterix-adapter-f8bb6f48d-2mqf6 -n utm -- tcpdump -i any udp port 6000
  

通过抓包可以精准定位流量丢失的环节，是公网链路、LoadBalancer、Node还是Pod内部的问题。

6. 检查Azure LoadBalancer配置 #

登录Azure门户，查看该LoadBalancer的规则：

• 确认存在UDP协议的转发规则，将公网端口1025映射到NodePort 31780；
• 检查loadBalancerSourceRanges是否包含传感器的IP（如果未移除白名单），或者是否已正确移除限制。

内容的提问来源于stack exchange，提问作者Evan Albrecht