AWS EKS Target Group就绪检测连接拒绝问题求助

环境信息 #

• kubectl版本：1.23.6
• eksctl版本：0.165.0
• 集群架构：通过eksctl搭建AWS EKS集群，部署运行在3000端口的NestJS应用，搭配AWS Load Balancer Controller
• 当前问题：Target Group就绪检测返回连接拒绝，Pod状态显示不健康；但本地通过端口转发可正常访问应用

Pod描述信息 #

kubectl describe pod techcare-apigw-7f5487b86c-flf8p -n techcare                                                                                                               ✔  18.10.0 Node  18.10.0 Node  nbozic@techcare.eu-central-1.eksctl.io ○  12:21:22  
Name:         techcare-apigw-7f5487b86c-flf8p
Namespace:    techcare
Priority:     0
Node:         ip-192-168-93-25.eu-central-1.compute.internal/192.168.93.25
Start Time:   Sun, 10 Dec 2023 12:15:07 +0100
Labels:       app=techcare-apigw
              pod-template-hash=7f5487b86c
Annotations:  <none>
Status:       Running
IP:           192.168.72.84
IPs:
  IP:           192.168.72.84
Controlled By:  ReplicaSet/techcare-apigw-7f5487b86c
Containers:
  techcare-apigw:
    Container ID:   containerd://2f2b2cf4870d783d9572307f8f3729f4e3fdd40722771d91b1e92b65eb3fcc76
    Image:          435109652403.dkr.ecr.eu-central-1.amazonaws.com/api-gateway:1.0.66
    Image ID:       435109652403.dkr.ecr.eu-central-1.amazonaws.com/api-gateway@sha256:1cdeab8e0a1db87ad79652b5887d0b961498dcf43a7f06a3ff4e05aec792306a
    Port:           3000/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sun, 10 Dec 2023 12:15:08 +0100
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     400m
      memory:  500Mi
    Requests:
      cpu:      400m
      memory:   500Mi
    Liveness:   http-get http://:3000/health delay=180s timeout=180s period=180s #success=1 #failure=180
    Readiness:  http-get http://:3000/healthz delay=180s timeout=180s period=180s #success=1 #failure=3
    Environment:
      PORT:  3000
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n28wz (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-n28wz:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  12m                   default-scheduler  Successfully assigned techcare/techcare-apigw-7f5487b86c-flf8p to ip-192-168-93-25.eu-central-1.compute.internal
  Normal   Pulling    12m                   kubelet            Pulling image "435109652403.dkr.ecr.eu-central-1.amazonaws.com/api-gateway:1.0.66"
  Normal   Pulled     12m                   kubelet            Successfully pulled image "435109652403.dkr.ecr.eu-central-1.amazonaws.com/api-gateway:1.0.66" in 131.403771ms (131.413766ms including waiting)
  Normal   Created    12m                   kubelet            Created container techcare-apigw
  Normal   Started    12m                   kubelet            Started container techcare-apigw
  Warning  Unhealthy  16s (x10 over 8m13s)  kubelet            Readiness probe failed: Get "http://192.168.72.84:3000/healthz": dial tcp 192.168.72.84:3000: connect: connection refused
  Warning  Unhealthy  16s (x3 over 6m16s)   kubelet            Liveness probe failed: Get "http://192.168.72.84:3000/health": dial tcp 192.168.72.84:3000: 
connect: connection refused

Service描述信息 #

kubectl describe service techcare-apigw-s -n techcare                                                                                                                    ✔  16s  18.10.0 Node  18.10.0 Node  nbozic@techcare.eu-central-1.eksctl.io ○  12:45:06  

Name:                     techcare-apigw-s
Namespace:                techcare
Labels:                   app=techcare-apigw-s
                          app.kubernetes.io/name=techcare-apigw-s
Annotations:              <none>
Selector:                 app=techcare-apigw
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.100.86.89
IPs:                      10.100.86.89
Port:                     <unset>  80/TCP
TargetPort:               3000/TCP
NodePort:                 <unset>  32437/TCP
Endpoints:                
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Ingress描述信息 #

kubectl describe ing techcare-apigw -n techcare                                                                                                                                ✔  18.10.0 Node  18.10.0 Node  nbozic@techcare.eu-central-1.eksctl.io ○  12:56:08  

Name:             techcare-apigw
Labels:           <none>
Namespace:        techcare
Address:          k8s-techcare-techcare-398c60dce3-83765040.eu-central-1.elb.amazonaws.com
Default backend:  techcare-apigw-s:80 ()
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /   techcare-apigw-s:80 ()
Annotations:  alb.ingress.kubernetes.io/scheme: internet-facing
              alb.ingress.kubernetes.io/target-type: instance
Events:
  Type    Reason                  Age                From     Message
  ----    ------                  ----               ----     -------
  Normal  SuccessfullyReconciled  46m (x2 over 46m)  ingress  Successfully reconciled

资源清单文件 #

apiVersion: v1
kind: Service
metadata:
  name: techcare-apigw-s
  namespace: techcare # Specify the correct namespace here
  labels:
    app.kubernetes.io/name: techcare-apigw-s
    app: techcare-apigw-s
spec:
  selector:
    app: techcare-apigw
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
  type: NodePort

apiVersion: apps/v1
kind: Deployment
metadata:
  name: techcare-apigw
  labels:
    app.kubernetes.io/name: techcare-apigw
  annotations:
    linkerd.io/inject: enabled
  namespace: techcare
spec:
  replicas: 1
  selector:
    matchLabels:
      app: techcare-apigw
  template:
    metadata:
      labels:
        app: techcare-apigw
    spec:
      containers:
        - image: 435109652403.dkr.ecr.eu-central-1.amazonaws.com/api-gateway:1.0.66
          imagePullPolicy: Always
          name: techcare-apigw
          ports:
            - containerPort: 3000
          env:
            - name: PORT
              value: "3000"
          resources:
            requests:
              memory: "500Mi"  # Set the requested memory
              cpu: "0.4"      # Set the requested CPU
            limits:
              memory: "500Mi"  # Set the requested memory
              cpu: "0.4"       # Set the CPU limit
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 180
            periodSeconds: 180
            timeoutSeconds: 180
            failureThreshold: 180
          readinessProbe:
            httpGet:
              path: /healthz
              port: 3000
            timeoutSeconds: 180
            initialDelaySeconds: 180
            periodSeconds: 180


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: techcare-apigw
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: instance
spec:
  ingressClassName: alb # Corrected Ingress class name
  defaultBackend: # Define default backend here
    service:
      name: techcare-apigw-s  # Replace with the name of your default backend service
      port:
        number: 80  # Replace with the port number of your default backend service
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: techcare-apigw-s
                port:
                  number: 80

测试结果 #

执行端口转发命令 kubectl port-forward techcare-apigw-7f5487b86c-flf8p -n techcare 2000:3000 后，可正常获取200响应。

内容的提问来源于stack exchange，提问作者Nikola Božić