AWS EKS Fargate探针报错rpc error及kubectl exec无法访问问题求助

我使用AWS EKS Fargate部署业务，应用Deployment YAML文件后前10分钟运行正常，但之后无法通过kubectl exec <podname> -- bash访问Pod。执行kubectl describe pod <podname>后，readinessProbe和livenessProbe均返回如下报错：

Events:
  Type     Reason           Age                 From               Message
  ----     ------           ----                ----               -------
  Warning  LoggingDisabled  16m                 fargate-scheduler  Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found
  Normal   Scheduled        15m                 fargate-scheduler  Successfully assigned k8s-fargate/k8s-api-5765846f76-d7nws to fargate-ip-10-0-130-250.ap-east-1.compute.internal
  Normal   Pulling          15m                 kubelet            Pulling image "awsaccid.dkr.ecr.ap-east-1.amazonaws.com/k8s-api-test:1.0.0"
  Normal   Pulled           14m                 kubelet            Successfully pulled image "awsaccid.dkr.ecr.ap-east-1.amazonaws.com/k8s-api-test:1.0.0" in 1m18.703187993s
  Normal   Created          14m                 kubelet            Created container k8s-api
  Normal   Started          14m                 kubelet            Started container k8s-api
  Warning  Unhealthy        2m18s               kubelet            Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "c2a2e9750a44684104a7e76a92bf7abe814ba29f306b092a48e17b90aab7f2dd": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: resource temporarily unavailable: unknown
  Warning  Unhealthy        2m13s               kubelet            Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "bcb60f638e2c364adc8694bc12f00660e2b0d7647d3861d3462727976d2df08c": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: resource temporarily unavailable: unknown
  Warning  Unhealthy        2m8s                kubelet            Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "988d29870b88fdcaae3cedf1071e79d2a786638c801364d71b6c7886f0be79e1": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: resource temporarily unavailable: unknown

此外，即使Pod处于不健康状态，livenessProbe也未触发Pod重启。

Deployment配置如下：

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: k8s-fargate
  name: k8s-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: k8s-api
  template:
    metadata:
      labels:
        app: k8s-api
    spec:
      volumes:
        - name: k8s-properties
          configMap: 
            name: k8s-properties
      containers:
        - name: k8s-api
          image: awsaccountid.dkr.ecr.ap-east-1.amazonaws.com/k8s-test:1.0.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8443
          resources:
            requests:
              memory: "1024Mi"
              cpu: "200m"
            limits:
              memory: "2500Mi"
              cpu: "1000m"
          volumeMounts:
            - name: k8s-properties
              mountPath: "/usr/local/folder"
              readOnly: false
          livenessProbe:
            exec:
              command:
              - cat
              - /usr/local/folder/file
            initialDelaySeconds: 5
            periodSeconds: 30
          readinessProbe:
            exec:
              command:
              - cat
              - /usr/local/folder/file
            initialDelaySeconds: 5
            periodSeconds: 5     

问题根源 #

1. 资源耗尽：报错resource temporarily unavailable核心原因是容器无法创建新进程（exec命令、探针的cat操作都需要启动新进程），大概率是CPU/内存资源被占满，或者PID资源耗尽。
2. 探针配置缺陷：当前livenessProbe未明确failureThreshold（默认3次失败才重启），且readinessProbe探测周期过短（5秒一次），频繁的探针会加剧资源消耗，形成恶性循环。
3. Fargate资源限制：Fargate对CPU/内存采用硬限制，若应用存在内存泄漏、CPU死循环等问题，会逐步耗尽资源，导致无法分配新进程所需资源。

解决步骤 #

1. 排查实时资源占用 #

• 执行kubectl top pod <podname> -n k8s-fargate，查看Pod的CPU和内存占用是否接近或超过配置的limits值。
• 若无法exec进入容器，先配置AWS日志收集（见步骤4），通过CloudWatch查看容器日志，定位应用是否有内存泄漏、进程异常等问题。

2. 调整探针配置 #

修改Deployment的探针参数，减少探测频率，明确重启条件：

livenessProbe:
  exec:
    command:
    - cat
    - /usr/local/folder/file
  initialDelaySeconds: 10  # 延长初始化等待，避免应用未完全启动就触发探测
  periodSeconds: 60        # 降低探测频率，减少资源消耗
  failureThreshold: 3      # 累计3次失败触发Pod重启
  timeoutSeconds: 5        # 设置探针超时，避免长时间占用资源
readinessProbe:
  exec:
    command:
    - cat
    - /usr/local/folder/file
  initialDelaySeconds: 10
  periodSeconds: 10        # 适当延长readiness探测周期
  failureThreshold: 2
  timeoutSeconds: 3

3. 优化资源配置 #

• 如果内存占用持续接近2500Mi，说明内存限制不足，可适当提高limits.memory；同时检查应用代码，排查是否存在内存泄漏。
• 若CPU使用率长期接近1000m，提高limits.cpu，或通过日志分析CPU占用高的具体进程，优化代码逻辑。

4. 配置Fargate日志收集 #

创建aws-logging ConfigMap，开启容器日志到CloudWatch的收集，方便后续排查：

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-logging
  namespace: kube-system
data:
  fluentd.conf: |
    <match **>
      @type cloudwatch_logs
      log_group_name /aws/eks/你的集群名称/fargate-logs
      log_stream_prefix k8s-fargate-
      region ap-east-1
      auto_create_stream true
    </match>

5. 排查PID资源耗尽 #

若CPU/内存占用正常，可通过容器日志或工具检查是否存在进程泄漏：

• 应用若频繁创建子进程且未及时回收，会耗尽容器的PID配额，导致无法创建新进程。需优化应用的进程管理逻辑，避免不必要的进程创建。

内容的提问来源于stack exchange，提问作者Andrew