升级Confluent Schema Registry至7.9.2引发AWS MSK Kafka IAM授权错误

背景 #

在AWS环境使用MSK运行Kafka，搭配容器化的Confluent Schema Registry管理Avro Schema。将基础镜像从confluentinc/cp-schema-registry:7.6.2升级到confluentinc/cp-schema-registry:7.9.2时，新版本启动触发IAM授权错误，旧版本运行正常。

错误日志 #

INFO initializing Kafka metrics collector (org.apache.kafka.common.telemetry.internals.KafkaMetricsCollector)
INFO Creating SSL context for keystore : null (org.apache.kafka.common.security.ssl.DefaultSslEngineFactory)
INFO Kafka version: 7.9.2-ce (org.apache.kafka.common.utils.AppInfoParser)
INFO Kafka commitId: 0a1b2c3d4e5f (org.apache.kafka.common.utils.AppInfoParser)
INFO Kafka startTimeMs: 1752156499999 (org.apache.kafka.common.utils.AppInfoParser)
INFO [Producer clientId=producer-1] Cluster ID: abcdefg123456 (org.apache.kafka.clients.Metadata)
INFO [Producer clientId=producer-1] Abortable authorization error: Cluster authorization failed..  Transition the producer state to ABORTABLE_ERROR (org.apache.kafka.clients.producer.internals.TransactionManager)
INFO [Producer clientId=producer-1] Transiting to abortable error state due to org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed. (org.apache.kafka.clients.producer.internals.TransactionManager)

现有IAM权限配置（7.6.2版本可用） #

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:Describe*",
        "kafka-cluster:WriteData",
        "kafka-cluster:AlterGroup",
        "kafka-cluster:AlterTopic",
        "kafka-cluster:CreateTopic",
        "kafka-cluster:ReadData"
      ],
      "Resource": [
        "arn:aws:kafka:us-west-1:${aws_account_id}:topic/Test-kafka-${environment_name}/*/_schemas",
        "arn:aws:kafka:us-west-1:${aws_account_id}:topic/Test-kafka-${environment_name}/*/__consumer_offsets",
        "arn:aws:kafka:us-west-1:${aws_account_id}:group/Test-kafka-${environment_name}/*",
        "arn:aws:kafka:us-west-1:${aws_account_id}:cluster/Test-kafka-${environment_name}/*"
      ]
    }
  ]
}

已尝试添加更宽泛的集群资源"arn:aws:kafka:us-west-1:${aws_account_id}:cluster/Test-kafka-${environment_name}"，但问题未解决。

Schema Registry Dockerfile内容 #

FROM confluentinc/cp-schema-registry:7.6.2

ENV AWS_MSK_IAM_VERSION=2.3.0
ENV AWS_MSK_IAM_JAR=aws-msk-iam-auth-${AWS_MSK_IAM_VERSION}-all.jar

USER root

RUN mkdir /work-dir

# Download AWS_MSK_IAM_JAR
RUN wget -O /work-dir/${AWS_MSK_IAM_JAR} https://github.com/aws/aws-msk-iam-auth/releases/download/v${AWS_MSK_IAM_VERSION}/${AWS_MSK_IAM_JAR}

RUN cp /work-dir/${AWS_MSK_IAM_JAR} /usr/share/java/cp-base-new/${AWS_MSK_IAM_JAR}
RUN cp /work-dir/${AWS_MSK_IAM_JAR} /usr/share/java/schema-registry/${AWS_MSK_IAM_JAR}

USER appuser

解决思路 #

• 补充Metrics Collector所需集群权限：7.9.2版本新增的Kafka Metrics Collector需要额外集群级权限，建议添加kafka-cluster:AlterCluster或更具体的kafka-cluster:DescribeClusterDynamicConfigs权限，适配遥测组件的配置读取需求。
• 升级AWS MSK IAM Auth版本：当前使用的aws-msk-iam-auth:2.3.0可能与7.9.2版本Kafka客户端不兼容，升级到最新稳定版（如2.4.0+），确保IAM认证逻辑适配新版本权限检查规则。
• 添加遥测主题权限：Metrics Collector可能尝试写入内部遥测主题（如_kafka-telemetry），需为该主题添加kafka-cluster:WriteData和kafka-cluster:CreateTopic权限，资源ARN格式参考现有主题配置。
• 验证集群ARN格式：MSK集群ARN标准格式为arn:aws:kafka:region:account-id:cluster/cluster-name/cluster-uuid，检查现有配置是否遗漏UUID部分或格式错误。
• 启用IAM访问日志：在AWS IAM中开启访问日志，查看具体被拒绝的权限动作和资源，精准定位缺失的权限项。

内容的提问来源于stack exchange，提问作者user3809888