AWS MSK S3 Sink Connector KMS加密下分片上传报错求助

分片上传初始化失败的直接原因是S3桶强制KMS加密校验，但当前S3 Sink Connector配置缺失加密相关参数，发起的MultipartUpload请求未携带合法KMS密钥信息，被桶策略直接拦截。

1. 补充Connector加密配置 #

在现有Connector配置中追加以下KMS加密相关配置项，更新后重启任务：

{
    "name": "kc-s3-nuoAccountIDs-sink",
    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
    "topics": "test-topic",
    "tasks.max": "1",
    "s3.bucket.name": "s3-bucket-name",
    "value.converter.schemas.enable": "false",
    "storage.class": "io.confluent.connect.s3.storage.S3Storage",
    "format.class": "io.confluent.connect.s3.format.avro.AvroFormat",
    "partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
    "behavior.on.null.values": "ignore",
    "schema.compatibility": "NONE",
    "partition.duration.ms": 3600000,
    "path.format": "'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH",
    "locale": "en-US",
    "timezone": "UTC",
    "partition.field.name": "accountplatform",
    "s3.region": "eu-west-2",
    "flush.size": 100000,
    // 以下为新增加密配置
    "s3.sse": "aws:kms",
    "s3.sse.kms.key.id": "替换为桶绑定的自定义KMS密钥完整ARN",
    "s3.sse.kms.encryption.context": "可选，若桶策略要求校验加密上下文则填写对应值"
}

2. 校验IAM权限 #

• 确认EKS上运行Connector的Pod关联的IAM角色（若使用IRSA则为服务账号绑定的角色，否则为节点角色），已被授予目标KMS密钥的以下操作权限：
  • kms:GenerateDataKey
  • kms:Decrypt
  • kms:Encrypt
• 确认同一IAM角色已配置S3分片上传所需的基础权限：
  • s3:PutObject
  • s3:AbortMultipartUpload
  • s3:ListMultipartUploadParts

• 重启Connector后日志无Unable to initiate MultipartUpload报错
• S3桶内新生成的对象元数据中，服务端加密类型为aws:kms，关联密钥为配置中填写的KMS密钥ID

内容的提问来源于stack exchange，提问作者Narendra Makwana