MySQL降级后Debezium Kafka Connector未知表架构报错求助

我们在使用Debezium MySQL Kafka Connector捕获MySQL表变更事件时遇到报错，推测是将MySQL从8.0降级至5.7后出现的。已尝试删除数据库历史主题、调整快照模式配置、更换Kafka Connect及MSK集群，但问题仍未解决。

报错堆栈信息 #

org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
	at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:369)
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:860)
	at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1125)
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:973)
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599)
	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.debezium.DebeziumException: Error processing binlog event
	... 7 more
Caused by: io.debezium.DebeziumException: Encountered change event for table tablename whose schema isn't known to this connector
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:654)
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:633)
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$13(MySqlStreamingChangeEventSource.java:831)
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:349)

连接器配置信息 #

{
    "name": "speed-account-table-v3",
    "config": {
        "connector.class": "io.debezium.connector.mysql.MySqlConnector",
        "snapshot.locking.mode": "none",
        "topic.creation.default.partitions": "1",
        "tasks.max": "1",
        "database.history.consumer.sasl.jaas.config": "jass config",
        "database.history.kafka.topic": "speed-history.speed-account-table-new",
        "bootstrap.servers": "cluster name",
        "database.history.consumer.security.protocol": "SASL_SSL",
        "tombstones.on.delete": "true",
        "snapshot.new.tables": "parallel",
        "topic.creation.default.replication.factor": "2",
        "database.history.skip.unparseable.ddl": "true",
        "value.converter": "io.confluent.connect.avro.AvroConverter",
        "key.converter": "org.apache.kafka.connect.json.JsonConverter",
        "database.allowPublicKeyRetrieval": "true",
        "database.history.producer.sasl.mechanism": "SCRAM-SHA-512",
        "database.user": "username",
        "database.server.id": "server id",
        "database.history.producer.security.protocol": "SASL_SSL",
        "database.history.kafka.bootstrap.servers": "cluster name",
        "database.server.name": "speed-account-v3",
        "database.port": "portnumber",
        "key.converter.schemas.enable": "false",
        "value.converter.schema.registry.url": "xxxx",
        "database.hostname": "xxxxxx",
        "database.password": "xxxxx",
        "value.converter.schemas.enable": "false",
        "name": "speed-account-table-v3",
        "table.include.list": "speed.tbl_account",
        "database.history.consumer.sasl.mechanism": "SCRAM-SHA-512",
        "snapshot.mode": "initial",
        "database.include.list": "speed"
    }
}

可行解决方案 #

• 彻底清理历史元数据：除删除数据库历史主题外，需删除Kafka Connect offset主题（默认connect-offsets）中对应此连接器的记录。先用kafka-console-consumer.sh查看offset主题内容，定位到包含server.id和连接器名称的条目，再用kafka-delete-records.sh工具删除该记录。
• 验证MySQL权限与表完整性：确认Debezium使用的数据库账号拥有SELECT、RELOAD、SHOW DATABASES、REPLICATION SLAVE、REPLICATION CLIENT权限；检查speed.tbl_account表在MySQL 5.7中的结构是否完整，无损坏。
• 调整连接器配置参数：
  • 临时添加include.schema.changes=true，让连接器捕获DDL事件重新同步表结构，问题解决后可移除
  • 将snapshot.mode改为schema_only_recovery，强制连接器直接从数据库读取最新表架构，不依赖历史主题数据
  • 确保database.history.skip.unparseable.ddl保持true，避免MySQL 8.0遗留的DDL语句导致解析失败
• 检查MySQL binlog配置：确认MySQL 5.7的binlog格式为ROW，binlog_row_image设置为FULL；若降级后binlog存在8.0特定事件，可备份后执行RESET MASTER重置binlog。
• 匹配版本兼容性：确保使用的Debezium Connector版本与MySQL 5.7兼容，推荐使用Debezium 2.x及以上版本，避免版本不匹配引发的解析问题。

内容的提问来源于stack exchange，提问作者Vandit Shah