使用CSI驱动在EKS中从AWS Secrets Manager获取密钥报错排查

执行POC测试时，尝试将AWS Secrets Manager中的密钥导入EKS Pod，遇到以下报错：

MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/busyboxsecret, err: rpc error: code = Unknown desc = Failed to fetch secret from all regions: arn:aws:secretsmanager:ap-south-1:121000000000:secret:test/secret-ihX9W4

配置文件 #

ServiceAccount.yml #

apiVersion: v1
kind: ServiceAccount
metadata:
  name: secret-manager-service-account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::121000000000:role/eksctl-sandbox-addon-iamserviceaccount-defau-Role1-M5ALRCJ7CZ8E

SecretProviderClass.yml #

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: secrets-provider
spec:
  provider: aws
  secretObjects:
    - secretName: testsecret
      type: Opaque
      data:
        - objectName: "MySecretPassword"
          key: password
  parameters:
    objects: |
      - objectName: arn:aws:secretsmanager:ap-south-1:121756462868:secret:test/secret-ihX9W4
        objectAlias: "MySecretPassword"

PodDefinition.yml #

apiVersion: v1
kind: Pod
metadata:
  name: busyboxsecret
  labels:
    name: busybox
    app: busybox
spec:
  serviceAccountName: secret-manager-service-account
  containers:
  - name: busyboxsecret
    image: busybox
    command: 
      - "sleep"
      - "3600"
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
    env:
    - name: DATABASE_PASSWORD
      valueFrom:
        secretKeyRef:
          name: testsecret # Name of the secret in Kubernetes Secrets, which has been set in the secretProviderClass 
          key: password
  volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "secrets-provider" 

IAM角色配置 #

权限策略 #

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "kms:CallerAccount": [
                        "AWS_ACCOUNT_ID"
                    ],
                    "kms:ViaService": [
                        "secretsmanager.ap-south-1.amazonaws.com"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "*"
        }
    ]
}

信任关系 #

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::121000000000:oidc-provider/oidc.eks.ap-south-1.amazonaws.com/id/E0A69D78D61DE240EB05F3266AB90640"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.ap-south-1.amazonaws.com/id/E0A69D78D61DE240EB05F3266AB90640:sub": "system:serviceaccount:default:secret-store-service-account",
                    "oidc.eks.ap-south-1.amazonaws.com/id/E0A69D78D61DE240EB05F3266AB90640:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

排查关键点 #

• 账号ID不匹配：报错信息中的Secret ARN属于账号121000000000，但SecretProviderClass.yml中配置的Secret ARN属于账号121756462868，两者不一致。需确认Secret所在的正确AWS账号，统一所有配置中的账号ID。
• 信任关系中的ServiceAccount名称错误：IAM角色信任关系里的sub字段指定的ServiceAccount是secret-store-service-account，但实际创建的ServiceAccount名称是secret-manager-service-account，导致Pod无法通过OIDC AssumeRole。需将信任关系中的sub值改为system:serviceaccount:default:secret-manager-service-account。
• KMS策略中的占位符未替换：IAM权限策略里的kms:CallerAccount使用了占位符AWS_ACCOUNT_ID，需替换为实际的AWS账号ID（如121000000000）。
• 验证Secret ARN正确性：确认报错中的Secret ARNarn:aws:secretsmanager:ap-south-1:121000000000:secret:test/secret-ihX9W4在对应账号和区域中存在，且未被删除或禁用。

内容的提问来源于stack exchange，提问作者Jaipal Solanki