AWS MSK Connect搭配MSSQL Debezium连接器连接失败求助

错误日志 #

连接器日志 #

[Worker-0a949760f6b805d4f] [2023-02-15 19:57:56,122] WARN [src-connector-014|task-0] [Consumer clientId=dlp.compcare.ccdemo-schemahistory, groupId=dlp.compcare.ccdemo-schemahistory] Bootstrap broker b-3.stuff.morestuff.c7.kafka.us-east-1.amazonaws.com:9098 (id: -2 rack: null) disconnected (org.apache.kafka.clients.NetworkClient:1079)

持续出现上述警告后，触发以下错误：

org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

MSK集群日志 #

[2023-02-15 20:08:21,627] INFO [SocketServer listenerType=ZK_BROKER, nodeId=3] Failed authentication with /172.32.34.126 (SSL handshake failed) (org.apache.kafka.common.network.Selector)

已确认的配置与状态 #

• EC2客户端通过IAM认证可正常连接MSK集群，执行生产/消费消息操作无异常
• 连接器启动时成功创建__amazon_msk_connect_status_*和__amazon_msk_connect_offsets_*主题
• 日志中出现的IP172.32.34.126已确认是连接器绑定的弹性网络接口IP
• 测试阶段已开放安全组全流量
• 连接器使用的IAM角色已配置msk*、msk-connect*、kafka*、s3*权限
• MSSQL RDS的CDC功能已验证正常
• 此前配置PostgreSQL RDS + Debezium连接器无同类问题

现有配置详情 #

MSK集群配置 #

auto.create.topics.enable=true
delete.topic.enable=true

MSK Connect Worker配置 #

key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
config.providers.secretManager.class=com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProvider
config.providers=secretManager
config.providers.secretManager.param.aws.region=us-east-1
request.timeout.ms=90000
errors.log.enable=true
errors.log.include.messages=true

Debezium MSSQL连接器配置 #

connector.class=io.debezium.connector.sqlserver.SqlServerConnector
tasks.max=1
database.history.consumer.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
schema.include.list=dbo
database.history.producer.sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
database.history.consumer.sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
database.history.consumer.security.protocol=SASL_SSL
database.instance=MSSQLSERVER
topic.prefix=dlp.compcare.ccdemo
schema.history.internal.kafka.topic=dlp.compcare.ccdemo.history
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.storage.StringConverter
database.history.sasl.mechanism=AWS_MSK_IAM
database.encrypt=false
database.history.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
database.history.producer.sasl.mechanism=AWS_MSK_IAM
database.history.producer.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
database.user=debezium
database.names=Intermodal_CCDEMO
database.history.producer.security.protocol=SASL_SSL
database.server.name=ccdemo_1
schema.history.internal.kafka.bootstrap.servers=b-1:9098
database.port=1433
database.hostname=my-mssql-rds.rds.amazonaws.com
database.history.sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
database.password=${secretManager:dlp-compcare:dbpassword}
table.include.list=dbo.EquipmentSetup
database.history.security.protocol=SASL_SSL
database.history.consumer.sasl.mechanism=AWS_MSK_IAM

怀疑方向 #

目前怀疑故障与IAM认证流程相关，但无法定位具体问题，已尝试多种配置调整仍未解决，请求协助排查。

内容的提问来源于stack exchange，提问作者Ethan Davis