Kafka Connect无法识别MySQL连接器，请求问题排查方向

问题描述 #

我使用以下Dockerfile构建包含MySQL连接器的Kafka Connect镜像：

FROM eclipse-temurin
RUN mkdir -p /opt/kafka/bin /opt/kafka/libs /opt/kafka/config /opt/kafka/logs
COPY ./kafka/bin /opt/kafka/bin/
COPY ./kafka/libs /opt/kafka/libs/
COPY ./kafka/config/connect-distributed.properties /opt/kafka/config/
COPY ./kafka/config/connect-log4j.properties /opt/kafka/config/
COPY ./jdbc-source-bulk.json /opt/kafka
RUN mkdir -p /opt/kafka/connectors
COPY ./mysql-connector-j-8.3.0/* /opt/kafka/connectors/
WORKDIR /opt/kafka
EXPOSE 8083
ENTRYPOINT ["./bin/connect-distributed.sh","config/connect-distributed.properties"]

启动容器后得到如下日志：

[2024-04-21 20:31:45,252] INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.AbstractConnectCli:120)
[2024-04-21 20:31:45,304] INFO Loading plugin from: /opt/kafka/connectors/mysql-connector-j-8.3.0.jar (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,529] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/mysql-connector-j-8.3.0.jar} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,537] INFO Loading plugin from: /opt/kafka/connectors/build (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,546] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/build/} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,546] INFO Loading plugin from: /opt/kafka/connectors/demo (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,554] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/demo/} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,554] INFO Loading plugin from: /opt/kafka/connectors/generated (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,562] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/generated/} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,563] INFO Loading plugin from: /opt/kafka/connectors/legacy (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,570] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/legacy/} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,570] INFO Loading plugin from: /opt/kafka/connectors/main (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,577] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/main/} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,578] INFO Loading plugin from: /opt/kafka/connectors/test (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,585] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/test/} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,585] INFO Loading plugin from: classpath (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,600] INFO Registered loader: jdk.internal.loader.ClassLoaders$AppClassLoader@c387f44 (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,601] INFO Scanning plugins with ServiceLoaderScanner took 297 ms (org.apache.kafka.connect.runtime.isolation.PluginScanner:70)
[2024-04-21 20:31:45,602] INFO Loading plugin from: /opt/kafka/connectors/mysql-connector-j-8.3.0.jar (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,913] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/mysql-connector-j-8.3.0.jar} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,913] INFO Loading plugin from: /opt/kafka/connectors/build (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)

执行以下curl命令：

curl -X PUT -H "Content-Type: application/json" \
-d @jdbc-source-bulk.json \
localhost:8083/connectors/mysql-source-bulk/config

返回错误结果：

{"error_code":500,"message":"Failed to find any class that implements Connector and which name matches io.confluent.connect.jdbc.JdbcSourceConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='3.7.0', encodedVersion=3.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='3.7.0', encodedVersion=3.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='3.7.0', encodedVersion=3.7.0, type=source, typeName='source', location='classpath'}"}

我尝试加载连接器但Kafka Connect无法识别，请问哪里操作有误？

问题原因与解决方法 #

• 核心问题：你仅复制了MySQL JDBC驱动包，但缺少Confluent JDBC Connector本身。io.confluent.connect.jdbc.JdbcSourceConnector是Confluent提供的Kafka Connect JDBC连接器类，并非MySQL驱动自带。
• 错误点分析：
  1. Dockerfile中仅复制了mysql-connector-j-8.3.0的内容，这只是数据库驱动，不包含Kafka Connect所需的JDBC连接器插件逻辑。
  2. 日志显示Kafka Connect扫描了/opt/kafka/connectors下的多个目录，但这些目录中没有包含JdbcSourceConnector类的JAR包，因此无法识别该连接器。
• 解决步骤：
  1. 获取Confluent JDBC Connector：下载Confluent JDBC Connector的压缩包（例如版本10.7.7），解压后得到包含连接器JAR和依赖的目录。
  2. 修改Dockerfile：将解压后的JDBC Connector目录复制到/opt/kafka/connectors下，示例修改如下：

     # 新增复制JDBC Connector的步骤
     COPY ./confluentinc-kafka-connect-jdbc-10.7.7 /opt/kafka/connectors/jdbc-connector/
     

  3. 确认插件路径配置：检查connect-distributed.properties中的plugin.path是否包含/opt/kafka/connectors，默认配置通常已包含，若未包含则添加：

     plugin.path=/opt/kafka/connectors,/opt/kafka/libs
     

  4. 重新构建并验证：重新构建镜像，启动容器后查看日志，确认JDBC连接器被成功加载，再执行curl命令创建连接器。

内容的提问来源于stack exchange，提问作者Abdelrahman Emara