基于Confluent镜像的Kafka Connect如何添加JDBC Sink配置？

Great question—let’s break this down clearly based on your current setup and future Helm deployment goal.

First: Don’t Package Connector Configs in Your Docker Image #

Your Dockerfile is already doing the right thing by bundling the JDBC Connector plugin and MySQL driver during the build stage. These are static dependencies that don’t change often, so they belong in the image.

However, your sink connector configuration (like database connection details, topic mappings, etc.) is runtime configuration—it will likely change (e.g., different environments, topic names, DB credentials) and shouldn’t be baked into the image. Rebuilding the image every time you tweak a config is inefficient and goes against container best practices.

Where to Put the Config Instead #

You have a few solid options for applying the sink config when running your Kafka Connect container, depending on your workflow:

1. Use the Kafka Connect REST API (Most Common for Distributed Mode) #

Confluent’s cp-kafka-connect-base runs in distributed mode by default, which is ideal for production. Once your container is up, you can submit your connector config via a POST request to the Connect REST endpoint:

curl -X POST -H "Content-Type: application/json" --data '{
  "name": "mysql-jdbc-sink",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "connection.url": "jdbc:mysql://your-mysql-host:3306/target_db?user=db_user&password=db_pass",
    "topics": "your-source-kafka-topic",
    "auto.create": "true",
    "insert.mode": "upsert",
    "pk.fields": "id",
    "pk.mode": "record_key",
    "tasks.max": "1"
  }
}' http://your-connect-host:8083/connectors

Tip: Store sensitive values (like DB passwords) in environment variables or secrets, not plaintext in the config. For the REST API, you can inject these values dynamically when sending the request.

2. Mount Config Files for Standalone Mode (Less Common for Production) #

If you were running Connect in standalone mode (not recommended for production), you could mount a properties file (e.g., mysql-sink.properties) into the container and pass it to the connect-standalone.sh command. But since you’re using Confluent’s distributed image, stick with the REST API approach.

Preparing for Helm Deployment #

When you’re ready to deploy via Helm, here’s how to integrate your connector config:

• Store configs in a ConfigMap: Define your connector’s JSON config in a Kubernetes ConfigMap.
• Use an init container or Kubernetes Job: Create a small job that runs after your Kafka Connect pod starts, which uses curl (or a lightweight HTTP client) to submit the config from the ConfigMap to the Connect REST endpoint.
• Leverage Confluent’s Official Helm Chart: If you use Confluent’s official Kafka Connect chart, you can define connectors directly in the values.yaml file under the connect.connectors section—this handles the config submission automatically for you.
• Secrets for Sensitive Data: Store DB credentials in a Kubernetes Secret, then reference them in your connector config (either via environment variables injected into the Connect pod, or by pulling the secret values into your init job’s curl command).

Quick Recap #

• Build stage: Keep your Docker image focused on dependencies (JDBC plugin, MySQL driver) only.
• Runtime/deployment: Apply connector configs via the REST API, or use Helm + ConfigMaps/Jobs to automate this in Kubernetes.

内容的提问来源于stack exchange，提问作者Molenpad