如何持久化保存通过HTTP创建的Kafka Connect配置文件？

To stop connector configurations from vanishing after restarting Kafka Connect, you need to configure the worker to use persistent storage for configs, offsets, and statuses instead of in-memory or local file storage. Here's the step-by-step fix:

Update Worker Configuration Properties #

Edit your Kafka Connect worker file (usually connect-distributed.properties for production setups) and add/set these critical properties:

1. Config Storage Topic #

This topic stores all connector and task configurations, and must use log compaction to retain the latest version of each config:

config.storage.topic=connect-configs
config.storage.replication.factor=3  # Match your Kafka cluster's replication requirements (use 1 for single-node dev)

2. Offset Storage Topic #

Tracks connector offsets so they resume from the correct position after a restart:

offset.storage.topic=connect-offsets
offset.storage.replication.factor=3
offset.storage.partitions=25  # Increase for high-throughput clusters

3. Status Storage Topic #

Stores the current state (running, failed, paused) of connectors and tasks:

status.storage.topic=connect-status
status.storage.replication.factor=3
status.storage.partitions=5

Key Best Practices #

• Use Distributed Mode: Standalone mode relies on local files which are not persistent across restarts (especially in containerized environments). Distributed mode is mandatory for reliable persistence.
• Topic Compaction: For connect-configs and connect-offsets, ensure the topics have log.cleanup.policy=compact enabled. If you let Connect auto-create these topics, this policy is applied by default.
• Pre-Create Topics (Production): Manually create these topics before starting Connect to enforce proper replication and partitioning:

  kafka-topics.sh --create --topic connect-configs --bootstrap-server <kafka-broker>:9092 --replication-factor 3 --partitions 1 --config cleanup.policy=compact
  kafka-topics.sh --create --topic connect-offsets --bootstrap-server <kafka-broker>:9092 --replication-factor 3 --partitions 25 --config cleanup.policy=compact
  kafka-topics.sh --create --topic connect-status --bootstrap-server <kafka-broker>:9092 --replication-factor 3 --partitions 5 --config cleanup.policy=compact
  

• Restart Connect: After updating the worker properties, restart the Kafka Connect service. Any connectors created via HTTP will now be saved to the connect-configs topic and survive restarts.

Verify Persistence #

To confirm your configs are stored, run this command to check the connect-configs topic:

kafka-console-consumer.sh --topic connect-configs --bootstrap-server <kafka-broker>:9092 --from-beginning --property print.key=true

You’ll see your connector’s config as a key-value pair. After restarting Connect, the connector will automatically reload from this topic.

内容的提问来源于stack exchange，提问作者Tim