启用exactly_once_v2的Spring Boot Kafka Streams对接AWS MSK授权失败求助

1. 修复IAM策略的语法错误 #

你的策略1存在JSON语法错误："kafka-cluster:WriteDataIdempotently" 末尾缺少逗号，导致AWS无法正确解析策略，权限未生效。修正后的策略1如下：

{
    "Effect": "Allow",
    "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:AlterCluster",
        "kafka-cluster:DescribeCluster",
        "kafka-cluster:DescribeClusterDynamicConfiguration",
        "kafka-cluster:ReadData",
        "kafka-cluster:WriteData",
        "kafka-cluster:*Topic*",
        "kafka-cluster:WriteDataIdempotently",
        "kafka-cluster:DescribeTransactionalId",
        "kafka-cluster:AlterTransactionalId",
        "kafka-cluster:AlterGroup",
        "kafka-cluster:DescribeGroup"
    ],
    "Resource": [
        "arn:aws:kafka:<region>:<account-id>:cluster/my-cluster/*",
        "arn:aws:kafka:<region>:<account-id>:topic/my-cluster/*",
        "arn:aws:kafka:<region>:<account-id>:transactional-id/my-cluster/*",
        "arn:aws:kafka:<region>:<account-id>:group/my-cluster/*"
    ]
}

注意：替换<region>和<account-id>为实际AWS区域和账号ID，同时移除Resource数组最后一项的多余逗号。

2. 确认TransactionalId资源ARN格式正确 #

Kafka Streams自动生成的TransactionalId格式为 <application-id>-<UUID>-<thread-number>，策略中必须用通配符*覆盖这类格式，正确的TransactionalId资源ARN应为：
arn:aws:kafka:<region>:<account-id>:transactional-id/my-cluster/*

3. 验证MSK集群事务配置 #

确保MSK Broker满足事务运行的基础配置要求：

• transaction.state.log.replication.factor 设置为至少3（与你的streams replicationFactor一致）
• transaction.state.log.min.isr 设置为至少2
• 确认集群已启用事务支持（MSK默认可能未开启，需在集群配置中检查并开启）

4. 显式配置Streams的安全参数 #

虽然spring.kafka.properties下的安全参数通常会被Streams复用，但可以显式将IAM认证参数添加到spring.kafka.streams.properties中，确保事务生产者能正确使用认证：

spring:
  kafka:
    producer:
      bootstrap-servers: <urls>
    properties:
      security.protocol: SASL_SSL
      sasl.mechanism: AWS_MSK_IAM
      sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required;
      sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler
    streams:
      application-id: my-application
      bootstrap-servers: <urls>
      replicationFactor: 3
      properties:
        security.protocol: SASL_SSL
        sasl.mechanism: AWS_MSK_IAM
        sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required;
        sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler
        acks: all
        retries: 3
        processing:
          guarantee: exactly_once_v2
        num:
          stream:
            threads: 3

5. 确认IAM角色绑定正确 #

确保运行Spring Boot应用的IAM角色（ECS/EKS部署）或实例角色（EC2部署）已正确关联修正后的IAM策略，避免出现角色未绑定策略的情况。

内容的提问来源于stack exchange，提问作者Wonger