排查EKS节点组中AWS Secrets Manager访问异常问题

环境与现象 #

拥有一个Amazon EKS集群，节点组包含两台配置一致的EC2实例，使用AWS Secrets Manager存储密钥供集群应用访问。

异常现象 #

一台节点上的Pod可正常拉取Secrets Manager密钥，另一台节点上的Pod访问时报错：

I0829 04:57:10.874010       1 auth.go:123] Role ARN for dev-ns:sm-sa is arn:aws:iam::91xxxxxx0031:role/aws-eks-oidc-role
W0829 04:58:45.144719       1 secrets_manager_provider.go:84] ap-south-1: Failed fetching secret eks-ssm-dev: WebIdentityErr: failed to retrieve credentials
caused by: RequestError: send request failed

执行kubectl top nodes确认：node1上的Pod正常运行，node2上的Pod无法启动。

已完成的前置检查 #

1. IAM角色与权限验证 #

• IAM角色：aws-eks-oidc-role
• 权限策略：已确认角色附加策略包含secretsmanager:GetSecretValue和secretsmanager:DescribeSecret权限，曾测试Secrets Manager全访问权限
• 可信实体配置：

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::91xxxxxx0031:oidc-provider/oidc.eks.ap-south-1.amazonaws.com/id/3FFF1AB9xxxxx304xxxxxF74F5F"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.ap-south-1.amazonaws.com/id/3FFF1AB9xxxxx304xxxxxF74F5F:sub": [
                        "system:serviceaccount:kube-system:aws-load-balancer-controller",
                        "system:serviceaccount:dev-ns:sm-sa"
                    ],
                    "oidc.eks.ap-south-1.amazonaws.com/id/3FFF1AB9xxxxx304xxxxxF74F5F:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

• 服务账号配置：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sm-sa
  namespace: dev-ns
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::91xxxxxx0031:role/aws-eks-oidc-role

2. Secrets Manager CSI驱动部署 #

• 驱动安装命令：

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system --set syncSecret.enabled=true --set enableSecretRotation=true

• AWS提供商安装命令：

helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws

• 驱动Pod状态：
  

3. 应用与SecretProviderClass配置 #

• Deployment示例：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-deployment
  namespace: dev-ns
spec:
  replicas: 10
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      serviceAccountName: sm-sa
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          volumeAttributes:
            secretProviderClass: "example-secret-provider" 
      containers:
      - name: example-container
        image: your-docker-image:latest
        volumeMounts:
        - name: secrets-store-inline
          mountPath: /mnt/secrets
          readOnly: true
        env:
        - name: MY_SECRET_KEY1
          valueFrom:
            secretKeyRef:
              name: example-secret
              key: my-secret-data-key1
        - name: MY_SECRET_KEY2
          valueFrom:
            secretKeyRef:
              name: example-secret
              key: my-secret-data-key2

• SecretProviderClass配置：

apiVersion: secrets-store.csi.k8s.io/v1
kind: SecretProviderClass
metadata:
  name: example-secret-provider
  namespace: dev-ns
spec:
  provider: "aws"
  secretObjects:
    - secretName: example-secret
      type: "Opaque"
      data:
        - objectName: "my-secret-key1"
          key: "my-secret-data-key1"
        - objectName: "my-secret-key2"
          key: "my-secret-data-key2"
  parameters:
    objects: |
      - objectName: "my-aws-secret-name"
        objectType: "secretsmanager"
        jmesPath:
          - path: "my-secret-key1"
            objectAlias: "my-secret-data-key1"
          - path: "my-secret-key2"
            objectAlias: "my-secret-data-key2"

根据报错RequestError: send request failed，核心是故障节点无法与AWS服务建立连接，结合单节点异常的特性，按以下顺序排查：

1. 验证故障节点的网络连通性 #

登录故障EC2节点，执行以下测试：

• 测试Secrets Manager端点连通性：

  curl -v https://secretsmanager.ap-south-1.amazonaws.com
  

• 测试STS服务端点（WebIdentity认证依赖）：

  curl -v https://sts.amazonaws.com
  

若连接失败，检查网络配置：

• 安全组：确认出站规则允许HTTPS（443）访问AWS服务，可临时开放所有出站流量测试
• NACL：确认子网出站规则允许443端口流量
• VPC终端节点：若使用终端节点，检查Secrets Manager/STS终端节点是否正常，且故障节点子网已关联终端节点路由

2. 检查节点CSI驱动运行状态 #

• 查看故障节点上的CSI驱动Pod：

  kubectl get pods -n kube-system -o wide | grep -E "(secrets-store|secrets-provider-aws)" | grep <故障节点名称>
  

• 若Pod异常，重启对应Pod：

  kubectl delete pod <异常Pod名称> -n kube-system
  

• 查看驱动Pod日志，定位节点级错误：

  kubectl logs <secrets-provider-aws-Pod名称> -n kube-system
  

3. 排查节点实例角色干扰 #

虽然使用了IRSA，但节点实例角色可能存在冲突：

• 查看节点绑定的IAM实例角色：

  curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/
  

• 若实例角色存在，检查其是否具备Secrets Manager权限，或临时移除实例角色测试（需确保不影响其他依赖）

4. 验证节点OIDC配置 #

• 检查kubelet的OIDC配置：

  cat /var/lib/kubelet/config.yaml | grep -E "provider-url|audience"
  

• 确认节点能解析OIDC端点：

  nslookup oidc.eks.ap-south-1.amazonaws.com
  

5. 强制重启故障节点 #

若以上检查均无异常，重启故障EC2实例修复节点级网络或kubelet异常：

aws ec2 reboot-instances --instance-ids <故障节点实例ID>

内容的提问来源于stack exchange，提问作者Manu