如何执行JDBC Kafka Connector Jar包？已克隆项目却不知运行方法

Hey there! Let me walk you through exactly how to build and run the JDBC Kafka Connector from your cloned kafka-connect-jdbc repository. I’ll break this down into simple, actionable steps so you can get up and running quickly.

Step 1: Build the Connector JARs #

Since you’re working from source, the first thing you need to do is compile and package the connector. The project uses Maven, so make sure you have Maven installed on your machine first.

1. Navigate to the root directory of your cloned repo:

   cd kafka-connect-jdbc
   

2. Run the Maven build command to generate the packaged connector (skip tests to speed things up):

   mvn clean package -DskipTests
   

3. Once the build finishes, you’ll find the packaged connector in the target directory. Look for a folder named kafka-connect-jdbc-<your-version>-package (replace <your-version> with the actual version from the repo). This folder contains:
   • The core connector JAR
   • All required dependencies (except database-specific JDBC drivers—more on that next)

Step 2: Add Database-Specific JDBC Drivers #

The JDBC Connector doesn’t include database drivers (like MySQL, PostgreSQL, Oracle) out of the box. You’ll need to download the driver JAR for your target database and place it inside the packaged connector folder you just built. For example:

• If you’re connecting to MySQL, download the mysql-connector-java-<version>.jar and drop it into kafka-connect-jdbc-<version>-package/lib

Step 3: Run the Connector (Two Modes) #

Kafka Connect runs in two modes: Standalone (great for testing/development) and Distributed (for production). Here’s how to set up both:

Option A: Standalone Mode (Testing) #

1. Copy the entire kafka-connect-jdbc-<version>-package folder into your Kafka installation’s plugins directory. If the plugins folder doesn’t exist, create it first:

   cp -r target/kafka-connect-jdbc-<version>-package $KAFKA_HOME/plugins/
   

2. Create a connector configuration file (e.g., jdbc-source-config.properties) with your database and Kafka details. Here’s a sample for a MySQL source connector:

   name=jdbc-mysql-source
   connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
   tasks.max=1
   connection.url=jdbc:mysql://localhost:3306/your_database?user=your_user&password=your_password
   table.whitelist=your_table
   mode=incrementing
   incrementing.column.name=id
   topic.prefix=mysql-
   

3. Start Kafka Connect in standalone mode, passing both the global Connect config and your connector config:

   $KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone.properties jdbc-source-config.properties
   

   • The connect-standalone.properties file is part of your Kafka installation—it sets global settings like bootstrap servers, key/value converters, etc.

Option B: Distributed Mode (Production) #

1. Copy the packaged connector folder to the plugins directory on all Kafka Connect nodes in your cluster.
2. Create a JSON configuration file (e.g., jdbc-source-config.json) with your connector settings:

   {
     "name": "jdbc-mysql-source",
     "config": {
       "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
       "tasks.max": "1",
       "connection.url": "jdbc:mysql://localhost:3306/your_database?user=your_user&password=your_password",
       "table.whitelist": "your_table",
       "mode": "incrementing",
       "incrementing.column.name": "id",
       "topic.prefix": "mysql-"
     }
   }
   

3. Submit the configuration to the Kafka Connect REST API (replace <connect-host> with your Connect node’s hostname/IP):

   curl -X POST -H "Content-Type: application/json" --data @jdbc-source-config.json http://<connect-host>:8083/connectors
   

4. Verify the connector is running by checking the Connect API:

   curl http://<connect-host>:8083/connectors/jdbc-mysql-source/status
   

Quick Troubleshooting Tips #

• If you see "ClassNotFoundException" for your database driver, double-check that the driver JAR is in the connector’s lib folder.
• Make sure your Kafka cluster is running before starting Connect.
• Verify the connection.url in your config is correct (check host, port, credentials, and database name).

内容的提问来源于stack exchange，提问作者adbar