AWS EKS环境中Consul API网关负载均衡器目标持续异常问题

问题背景 #

在AWS EKS集群中配置跨命名空间的Consul API Gateway及Keycloak的HttpRoute后，生成的AWS负载均衡器显示注册目标持续处于不健康状态，无法通过API网关正常路由到Keycloak服务。

相关配置文件 #

Consul-server-and-ui-deployment.yaml #

---
global:
  name: consul  
  imagePullSecrets:
    - name: "***"
  image: /hashicorp/consul:1.12.0
  imageEnvoy: /envoyproxy/envoy:v1.22.2
  imageK8S: /hashicorp/consul-k8s-control-plane:0.44.0
  metrics:
    enabled: true
server:
  replicas: 3
  bootstrapExpect: 3
  extraLabels:
    tags.datadoghq.com/source: consul
    tags.datadoghq.com/service: consul-server
  annotations: |
    "tags.datadoghq.com/source":                              "consul"
    "tags.datadoghq.com/service":                             "consul-server"
  updatePartitions: 3
  disruptionBudget:
    maxUnavailable: 1
  tolerations: >
    -
    key:                                                    "taint_for_consul_xor_vault"
      operator:                                               "Equal"
      value:                                                  "true"
      effect:                                                 "NoSchedule"
  nodeSelector: |
    'eks.amazonaws.com/capacityType':                         'ON_DEMAND'
    'purpose':                                                'consul-server'
  storage: 50G
  priorityClassName: high-priority
client:
  enabled: true
  grpc: true
  exposeGossipPorts: false
  priorityClassName: high-priority
  resources:
    requests:
      memory: 1G
      cpu: "1"
    limits:
      memory: 1G
      cpu: "1"
  healthChecks:
    enabled: true
  hostNetwork: false
  extraConfig: |
    {
      "advertise_reconnect_timeout":                          "15m",
      "limits":                                               {
        "http_max_conns_per_client":                          -1
      }
    }
  extraLabels:
    tags.datadoghq.com/source: consul
    tags.datadoghq.com/service: consul-client
  annotations: |
    "tags.datadoghq.com/source":                              "consul"
    "tags.datadoghq.com/service":                             "consul-client"
  updateStrategy: |
    type:                                                     RollingUpdate
    rollingUpdate:
      maxUnavailable:                                         1
ui:
  enabled: true
  service:
    type: LoadBalancer
    additionalSpec: "'ports': [{'name': 'http', 'protocol': 'TCP', 'port': 8500,
      'targetPort': 8500}, {'name': 'https', 'protocol': 'TCP', 'port': 8501,
      'targetPort': 8501}]"
    annotations: |
      'service.beta.kubernetes.io/aws-load-balancer-scheme':  'internal'
      'service.beta.kubernetes.io/aws-load-balancer-type':    'nlb-ip'
controller:
  enabled: true
prometheus:
  enabled: true
grafana:
  enabled: true
terminatingGateways:
  enabled: true
  priorityClassName: high-priority
  defaults:
    replicas: 2
apiGateway:
  enabled: true
  image: /hashicorp/consul-api-gateway:0.3.0
  controller:
    replicas: 2
    priorityClassName: high-priority
connectInject:
  enabled: true
  priorityClassName: high-priority
  transparentProxy:
    defaultEnabled: true
  default: false
syncCatalog:
  enabled: true
  priorityClassName: high-priority
  default: false
  toConsul: true
  toK8S: false

Consul-api-gateway-deployment.yaml #

apiVersion:         api-gateway.consul.hashicorp.com/v1alpha1
kind:               GatewayClassConfig
metadata:
  name:             consul-common-gateway-class-config
spec:
  logLevel:         'info'  
  copyAnnotations:
    service:
      - service.beta.kubernetes.io/aws-load-balancer-scheme
      - service.beta.kubernetes.io/aws-load-balancer-type
  consul:
    scheme:         'http'
    ports:
      http:         8500
      grpc:         8502
  serviceType:      LoadBalancer
---
apiVersion:         gateway.networking.k8s.io/v1alpha2
kind:               GatewayClass
metadata:
  name:             consul-common-gateway-class
spec:
  controllerName:   'hashicorp.com/consul-api-gateway-controller'
  parametersRef:
    group:          api-gateway.consul.hashicorp.com
    kind:           GatewayClassConfig
    name:           consul-common-gateway-class-config
---
apiVersion:                                                 gateway.networking.k8s.io/v1alpha2
kind:                                                       Gateway
metadata:
  name:                                                     common-api-gateway
  annotations:
     'service.beta.kubernetes.io/aws-load-balancer-scheme':  'internal'
     'service.beta.kubernetes.io/aws-load-balancer-type':    'nlb-ip'           
spec:
  gatewayClassName:                                         consul-common-gateway-class
  listeners:  
  - protocol:                                               HTTP
    port:                                                   80
    name:                                                   http
    allowedRoutes:
      namespaces:
        from:                                               All

HttpRoute-keycloak-deployment.yaml #

---
apiVersion:       gateway.networking.k8s.io/v1alpha2
kind:             HTTPRoute
metadata:
  name:           gateway-keycloak-route
  namespace:      app-ns
spec:
  parentRefs:
  - name:         common-api-gateway
    namespace:    consul
  rules:
  - matches:
    - path:
        type:     PathPrefix
        value:    /auth
    backendRefs:
    - kind:       Service
      name:       keycloak
      port:       80
      namespace:  app-ns
---
apiVersion:       gateway.networking.k8s.io/v1alpha2
kind:             ReferencePolicy
metadata:
  name:           reference-policy-keycloak
spec:
  from:
    - group:      gateway.networking.k8s.io
      kind:       HTTPRoute
      namespace:  app-ns
  to:
    - group:      ""
      kind:       Service
      name:       keycloak

排查与修复步骤 #

1. 检查Consul API Gateway Pod状态与健康探针 #

• 执行kubectl get pods -n consul查看网关相关Pod是否正常运行，若出现CrashLoopBackOff，查看日志定位问题：kubectl logs <gateway-pod-name> -n consul
• 确认Pod的存活/就绪探针配置，Consul API Gateway默认使用:8080/health作为健康检查端点，需确保Pod内该端口可正常访问

2. 修正AWS NLB健康检查配置 #

• 登录AWS控制台找到对应NLB，查看目标组的健康检查设置：
  • 默认健康检查路径为/，但Consul API Gateway的健康端点是/health，需手动修改目标组的健康检查路径为/health，端口保持80
  • 确认健康检查协议为HTTP，超时时间设为5秒、间隔时间设为10秒，匹配服务实际响应能力

3. 完善跨命名空间路由权限配置 #

当前ReferencePolicy未指定命名空间，需补充目标服务所在的命名空间，修改后的配置如下：

---
apiVersion:       gateway.networking.k8s.io/v1alpha2
kind:             ReferencePolicy
metadata:
  name:           reference-policy-keycloak
  namespace:      app-ns
spec:
  from:
    - group:      gateway.networking.k8s.io
      kind:       HTTPRoute
      namespace:  app-ns
  to:
    - group:      ""
      kind:       Service
      name:       keycloak

4. 验证Keycloak服务可访问性 #

• 在Consul API Gateway Pod内执行curl http://keycloak.app-ns.svc.cluster.local:80/auth，确认Keycloak服务能正常响应
• 若无法访问，检查Keycloak Pod状态是否正常、Service端口配置是否与实际容器端口一致

5. 确认Consul组件通信正常 #

• 查看Consul API Gateway控制器日志：kubectl logs <gateway-controller-pod-name> -n consul，确认是否能正常连接Consul Server（端口8500/8502）
• 执行kubectl exec <consul-server-pod-name> -n consul -- consul members，检查Consul集群健康状态

6. 升级Consul API Gateway版本（可选） #

当前使用的consul-api-gateway:0.3.0版本较旧，存在NLB适配相关的已知问题，建议升级到1.x系列稳定版本，同时同步升级Consul及consul-k8s-control-plane版本，确保组件版本兼容性

内容的提问来源于stack exchange，提问作者PaulAndrew