AWS EKS上cert-manager基于Route53的DNS01验证失败

错误信息 #

Warning  PresentError  35s                 cert-manager-challenges  Error presenting challenge: failed to change Route 53 record set: AccessDenied: User: arn:aws:sts::ACCOUNT:assumed-role/cert-manager/1633841314544392345 is not authorized to perform: route53:ChangeResourceRecordSets on resource: arn:aws:route53:::hostedzone/ZONEIDbecause no identity-based policy allows the route53:ChangeResourceRecordSets action

配置详情 #

Helm安装cert-manager的values配置 #

namespace: "cert-manager"
serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: "cert-manager-service-account"
  # Optional additional annotations to add to the controller's ServiceAccount 
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNTID:role/cert-manager
  # Automount API credentials for a Service Account.
  # Optional additional labels to add to the controller's ServiceAccount
  # labels: {}
  automountServiceAccountToken: true  

# Add SecurityContext fsgroup
securityContext:
  runAsNonRoot: true
  fsGroup: 1001

AWS IAM角色配置 #

权限策略

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "route53:GetChange",
      "Resource": "arn:aws:route53:::change/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets"
      ],
      "Resource": "arn:aws:route53:::hostedzone/ZONEID/*"
    },
    {
      "Effect": "Allow",
      "Action": "route53:ListHostedZonesByName",
      "Resource": "*"
    }
  ]
}

信任策略

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Principal": {
        "Federated": "arn:aws:iam::ACCOUNTID:oidc.region.amazonaws.com/id/EKS-HASH"
      },
      "Condition": {
        "StringEquals": {
          "oidc.eks.region.amazonaws.com/id/EKS-HASH:sub": "system:serviceaccount:cert-manager:cert-manager-service-account"
        }
      }
    }
  ]
}

cert-manager CRD资源配置 #

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-playground
spec:
  acme:
    email: email.com
    privateKeySecretRef:
      name: letsencrypt-playground
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    solvers:
      - dns01:
          route53:
            region: eu-west-1
            hostedZoneID: ZONEID
            role: arn:aws:iam::ACCOUNT:role/cert-manager
        selector:
          dnsZones:
            - "domain"

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: letsencrypt-playground
  namespace: istio-system
spec:
  secretName: letsencrypt-playground
  commonName: hello.domain
  duration: 168h # 7d
  renewBefore: 120h # 5d
  dnsNames:
  - hello.domain
  issuerRef:
    name: letsencrypt-playground
    # We can reference ClusterIssuers by changing the kind here.
    # The default value is Issuer (i.e. a locally namespaced Issuer)
    kind: ClusterIssuer
    group: cert-manager.io

当前状态 #

已移除ClusterIssuer中的role配置（ServiceAccount已关联IAM角色），但权限拒绝错误仍存在；Certificate资源事件显示已创建证书请求，但目标Secret未生成：

Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    3m4s  cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  3m3s  cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "letsencrypt-playground-2g4mq"
  Normal  Requested  3m3s  cert-manager-certificates-request-manager  Created new CertificateRequest resource "letsencrypt-playground-vjtpv"

排查方案 #

• 修正IAM策略资源ARN格式：Route53托管区的资源ARN应为arn:aws:route53:::hostedzone/ZONEID，无需追加/*。修改权限策略中对应Resource字段，因为route53:ChangeResourceRecordSets操作的目标是托管区本身，而非其子资源。
• 验证OIDC身份提供商与信任策略匹配：确认EKS集群已正确创建OIDC提供商，信任策略中的Federated ARN与集群OIDC提供商ARN完全一致；检查sub字段是否准确匹配system:serviceaccount:cert-manager:cert-manager-service-account（注意namespace和ServiceAccount名称）。
• 检查ServiceAccount注解生效情况：执行命令kubectl get serviceaccount cert-manager-service-account -n cert-manager -o yaml，确认eks.amazonaws.com/role-arn注解存在且值正确，同时确认automountServiceAccountToken为true。
• 查看挑战Pod日志：找到名称包含cert-manager-challenges的Pod，执行kubectl logs <pod-name> -n cert-manager获取详细错误日志，排查是STS角色假设失败还是权限策略不匹配。
• 检查IAM角色权限边界：若IAM角色设置了权限边界，需确保边界允许route53:ChangeResourceRecordSets等相关操作，否则即使附加了策略也会被拒绝。
• 确认托管区ID正确性：核对ClusterIssuer和IAM策略中的ZONEID与Route53控制台显示的托管区ID是否一致，避免ID错误导致权限匹配失败。
• 重启cert-manager控制器：修改ServiceAccount或IAM配置后，执行kubectl rollout restart deployment cert-manager -n cert-manager重启控制器，确保新配置生效。

内容的提问来源于stack exchange，提问作者andrew clark