生产环境EKS部署失败：Secret存储卷挂载权限错误排查

集群现状 #

• US-East-2区域的QA/Dev集群：所有部署操作正常执行
• US-East-1区域的生产集群：Secret共享映射时出现错误，配置与QA/Dev集群一致

错误日志 #

MountVolume.SetUp failed for volume "secrets-store-inline-db" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod enterprise-production/enterprise-web-589994cf57-65h4q, err: rpc error: code = Unknown desc = us-east-1: Failed fetching secret arn:aws:secretsmanager:us-east-1:XXXXXXXXXXX:secret:rds-xxx-xxx-xxx: WebIdentityErr: failed to retrieve credentials caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity status code: 403, request id: bd09ac7c-c0e8-49f4-98e6-905c8174c1c7
Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline-db], unattached volumes=[secrets-store-inline-db kube-api-access-d8rgs aws-iam-token]: timed out waiting for the condition
(combined from similar events): MountVolume.SetUp failed for volume "secrets-store-inline-db" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod enterprise-production/enterprise-web-589994cf57-65h4q, err: rpc error: code = Unknown desc = us-east-1: Failed fetching secret arn:aws:secretsmanager:us-east-1:xxxxxxx:secret:rds-xxxxx-xxxx-XXXX: WebIdentityErr: failed to retrieve credentials caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity status code: 403, request id: bd157f5d-5852-49d2-806b-3dd2f1e18583

已配置资源 #

IAM角色信任策略（关联ServiceAccount） #

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::xxxxxxxx:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/xxxxxxxxxx"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/xxxxxx:aud": "sts.amazonaws.com",
"oidc.eks.us-east-1.amazonaws.com/id/xxxxxx:sub": "system:serviceaccount:*:*"
}
}
}
]
}

生产环境Values.yaml配置 #

image:
repositoryBase: xxxxxxx.dkr.ecr.us-east-1.amazonaws.com

serviceAccount:
# 要添加到ServiceAccount的注解
annotations: {
"eks.amazonaws.com/role-arn": "arn:aws:iam::xxxxxx:role/eksctl-Prod-Cluster-addon-iamserviceaccount-Role1-8JPAZJ72W4N1"
}

ingress:
cert: "arn:aws:acm:us-east-1:xxxxxx:certificate/0acb6fce-e94a-48f9-9405-4470dd28ecb1"
group: "prod-alb"

environment:
target: production
env: [
{ name: REDIS_HOST, value: "prod-xxxx-redis.mtalpw.ng.0001.use1.cache.amazonaws.com" }
]

secrets:
- name: prod-db
  secretname: rds-xxx-xxxx
  region: us-east-1
  object: arn:aws:secretsmanager:us-east-1:xxxx:secret:rds-xxx-xxx-xxx
  paths:
    - path: "username"
      objectAlias: "db_username"
    - path: "password"
      objectAlias: "db_password"
    - path: "host"
      objectAlias: "db_host"
    - path: "port"
      objectAlias: "db_port"
  data:
    - key: "DB_USERNAME"
      objectName: "db_username"
    - key: "DB_PASSWORD"
      objectName: "db_password"
    - key: "DB_HOST"
      objectName: "db_host"
    - key: "DB_PORT"
      objectName: "db_port"

volumes:
- name: secrets-store-inline-db
  csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
      secretProviderClass: "prod-db"

已尝试操作 #

• 创建新的ServiceAccount
• 修改Values.yaml中的其他配置项

排查建议 #

1. 验证OIDC提供商与集群的匹配性

   • 执行命令aws eks describe-cluster --name <prod-cluster-name> --region us-east-1 --query "cluster.identity.oidc.issuer"获取集群OIDC issuer，确认与信任策略中的oidc.eks.us-east-1.amazonaws.com/id/xxxxxx完全匹配（注意末尾的ID部分不能有误）。

2. 检查IAM角色的权限边界

   • 查看关联IAM角色eksctl-Prod-Cluster-addon-iamserviceaccount-Role1-8JPAZJ72W4N1的权限策略，确保包含secretsmanager:GetSecretValue操作，且资源范围覆盖目标密钥ARN。
   • 检查该角色是否设置了权限边界，若有，需确保边界允许sts:AssumeRoleWithWebIdentity和secretsmanager:GetSecretValue操作。

3. 验证ServiceAccount注解的准确性

   • 执行命令kubectl describe sa <serviceaccount-name> -n enterprise-production查看注解内容，确认eks.amazonaws.com/role-arn指向的IAM角色ARN无拼写错误。

4. 检查SecretProviderClass配置

   • 执行命令kubectl get secretproviderclass prod-db -o yaml，检查其中的roleArn字段是否与IAM角色ARN一致，且objects部分的Secret ARN正确。

5. 验证CSI驱动的安装状态

   • 执行命令kubectl get pods -n kube-system | grep secrets-store，确保所有驱动Pod处于Running状态。

内容的提问来源于stack exchange，提问作者Andrew Evers