Helm部署GCP K8s的SpringBoot应用无法通过LoadBalancer访问

问题背景 #

部署了单端点Spring Boot Web应用，通过Helm安装到GCP Kubernetes集群，应用监听8080端口，创建LoadBalancer类型Service暴露服务，Deployment、Pod、Service资源均显示运行正常，但通过外部IP无法访问应用。
相关配置截图：

现有Deployment配置 #

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: demo
    meta.helm.sh/release-namespace: springboot-demoweb
  creationTimestamp: "2022-07-15T09:20:05Z"
  generation: 1
  labels:
    app: springboot-demoweb
    app.kubernetes.io/managed-by: Helm
    chart: springboot-demoweb-0.1.0
    heritage: Helm
    release: demo
  name: demo-springboot-demoweb
  namespace: springboot-demoweb
  resourceVersion: "514983"
  uid: c0381302-7f1d-44c2-a763-fc4a743395fd
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: springboot-demoweb
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: springboot-demoweb
        release: demo
    spec:
      containers:
      - image: rohit2502/springboot-helm-chart:latest
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: 8080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: springboot-demoweb
        ports:
        - containerPort: 8080
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: 8080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2022-07-15T09:20:25Z"
    lastUpdateTime: "2022-07-15T09:20:25Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2022-07-15T09:20:05Z"
    lastUpdateTime: "2022-07-15T09:20:25Z"
    message: ReplicaSet "demo-springboot-demoweb-7dc46847bc" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

现有Service配置 #

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/neg: '{"ingress":true}'
  creationTimestamp: "2022-07-15T09:37:43Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app: springboot-demoweb
    app.kubernetes.io/managed-by: Helm
    chart: springboot-demoweb-0.1.0
    heritage: Helm
    release: demo
  name: demo-springboot-demoweb-service
  namespace: springboot-demoweb
  resourceVersion: "544118"
  uid: 72e80300-b107-4329-852b-a3592456225d
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.8.4.82
  clusterIPs:
  - 10.8.4.82
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - nodePort: 31905
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: springboot-demoweb
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 104.197.133.49

排查方向（按优先级排序） #

• 访问方式校验：当前Service配置对外暴露80端口，转发到Pod的8080端口，访问时直接使用http://104.197.133.49即可，不要额外添加8080端口，这是最高发的配置错误。
• 后端端点校验：执行命令kubectl get endpoints -n springboot-demoweb demo-springboot-demoweb-service，确认返回的ENDPOINTS列表中存在对应Pod的IP+8080端口，如果列表为空说明Service标签和Pod标签不匹配，流量无法转发到后端。
• 冗余注解清理：当前Service携带的cloud.google.com/neg: '{"ingress":true}'注解是专供GCP Ingress资源使用的，会创建网络端点组接管流量转发，普通LoadBalancer Service不需要这个注解，直接删除该注解后等待1-2分钟再测试访问。
• 集群内连通性逐层验证：
  1. 启动临时调试Pod：kubectl run -it --rm debug --image=curlimages/curl -n springboot-demoweb -- sh
  2. 直接访问Pod IP+8080端口，确认应用本身能正常返回hello world，排除应用绑定127.0.0.1、context-path配置错误、端口监听异常等问题
  3. 访问Service ClusterIP 10.8.4.82的80端口，确认Service转发规则正常
  4. 访问任意集群节点IP+31905端口，确认NodePort转发正常
• GCP防火墙规则核查：GCP VPC默认不会自动放开LoadBalancer的入站流量，需要到VPC防火墙页面确认存在允许80端口入站、来源为0.0.0.0/0（测试阶段使用，生产按需收紧网段）的规则，且规则应用到了集群节点所在的目标标签。
• 负载均衡健康检查核查：到GCP负载均衡控制台查看对应后端服务的健康检查状态，如果健康检查失败，负载均衡不会转发流量。需要确认健康检查的路径、端口和应用实际配置一致，当前应用健康检查路径为/、端口为8080，要和GCP侧配置对齐。
• 网络策略校验：确认集群内没有配置NetworkPolicy规则拦截8080端口的入站流量、拦截LoadBalancer健康检查网段的流量。

常见遗漏配置 #

• Spring Boot默认绑定0.0.0.0地址，如果手动修改过server.address配置为127.0.0.1，会导致外部无法访问，需要改回0.0.0.0
• 如果配置了server.servlet.context-path，存活探针、就绪探针、访问路径都需要加上对应前缀，否则会出现探针显示正常但实际访问404的问题
• 私有GKE集群需要确认负载均衡的访问策略没有限制公网来源，且控制平面和节点之间的连通性正常

内容的提问来源于stack exchange，提问作者Rohit