Keda Operator使用SQS Scaler时遭遇AWS访问权限拒绝问题排查

使用Keda的SQS Scaler对Deployment进行扩缩容，ScaledObject配置identityOwner: operator。已创建带指定权限和信任关系的IAM角色，并将角色注解到keda命名空间的keda-operator ServiceAccount。在test-keda命名空间部署ScaledObject后，keda-operator Pod日志报AccessDenied错误。尝试将identityOwner改为pod后仍权限拒绝，错误显示节点角色尝试获取Deployment的IRSA角色权限。

权限配置 #

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "KedaSQSReadPermission",
            "Effect": "Allow",
            "Action": "sqs:GetQueueAttributes",
            "Resource": "arn:aws:sqs:<region-name>:<account-id>:<test-queue>"
        },
        {
            "Sid": "KedaSQSListPermission",
            "Effect": "Allow",
            "Action": "sqs:ListQueues",
            "Resource": ["*"]
        }
    ]
}

信任关系配置 #

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<account-id>:oidc-provider/oidc.eks.<region-name>.amazonaws.com/id/<oidc-id>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.<region-name>.amazonaws.com/id/<oidc-id>:sub": "system:serviceaccount:keda:keda-operator"
                }
            }
        }
    ]
}

keda-operator ServiceAccount配置 #

{
  "apiVersion": "v1",
  "kind": "ServiceAccount",
  "metadata": {
    "annotations": {
      "eks.amazonaws.com/role-arn": "arn:aws:iam::<account-id>:role/<role-name>",
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"ServiceAccount\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/name\":\"keda-operator\",\"app.kubernetes.io/part-of\":\"keda-operator\",\"app.kubernetes.io/version\":\"2.7.0\"},\"name\":\"keda-operator\",\"namespace\":\"keda\"}}\n"
    },
    "creationTimestamp": "2022-07-14T13:49:40Z",
    "labels": {
      "app.kubernetes.io/name": "keda-operator",
      "app.kubernetes.io/part-of": "keda-operator",
      "app.kubernetes.io/version": "2.7.0"
    },
    "name": "keda-operator",
    "namespace": "keda",
    "resourceVersion": "174401872",
    "uid": "a5862afb-fcd1-49e3-8ebb-6c81f4fd5014"
  },
  "secrets": [
    {
      "name": "keda-operator-token-wzqsz"
    }
  ]
}

ScaledObject配置 #

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
    name: scaled-object-test
    namespace: test-keda
    labels:
        name: test-app
spec:
  scaleTargetRef:
    kind: Deployment
    name: test-app
  pollingInterval: 5
  cooldownPeriod: 10
  maxReplicaCount: 8
  minReplicaCount: 0
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.<region-name>.amazonaws.com/<account-id>/<queue-name>
        queueLength: "10"
        awsRegion: "<region-name>"
        identityOwner: operator

错误日志 #

error: "AccessDenied: Access to the resource https://sqs.<region-name>.amazonaws.com/ is denied.
	status code: 403, request id: 87fd482d-89cf-54ef-af3e-13a25f4d9e23"

排查提示 #

• 检查IAM角色权限中的SQS资源ARN是否与实际队列完全匹配：日志中拒绝访问的是队列服务根地址，可能是权限配置里的队列ARN拼写错误（比如<test-queue>占位符未替换为实际队列名），导致Keda尝试访问时权限不匹配。
• 验证keda-operator Pod是否正确挂载了IRSA角色：执行kubectl exec -n keda <keda-operator-pod-name> -- env | grep AWS，查看是否存在AWS_WEB_IDENTITY_TOKEN_FILE和AWS_ROLE_ARN环境变量，确认IRSA配置生效。
• 检查信任关系中的OIDC条件是否准确：确认OIDC提供商URL、ServiceAccount的sub字段（system:serviceaccount:keda:keda-operator）与实际环境完全一致，避免拼写错误或命名空间/ServiceAccount名称不匹配。
• 当identityOwner: pod时，需确保被扩缩容的Deployment对应的ServiceAccount已配置正确的IRSA角色注解，且该角色具备SQS访问权限，而非依赖节点角色。此时节点角色尝试获取权限说明Pod级IRSA未正确配置。
• 检查Keda版本兼容性：当前使用的Keda 2.7.0是否与EKS集群版本兼容，部分版本可能存在IRSA适配问题，可尝试升级Keda到稳定版本。
• 查看CloudTrail日志：在AWS控制台搜索对应的403请求ID，确认请求使用的角色是否为预期的keda-operator IRSA角色，以及具体被拒绝的操作和资源，定位权限缺失点。

内容的提问来源于stack exchange，提问作者Vineet