为现有Ambari管理集群添加独立KafkaConnect Docker实例

Hey there! Great question—you don’t need to rely on images that bundle Kafka or ZooKeeper with Connect. Let’s break down how to find and configure the right image for your setup:

1. Start with Official, Purpose-Built Kafka Connect Images #

The best options are images designed specifically for Kafka Connect, which don’t include Kafka brokers or ZooKeeper by default:

• Confluent’s CP-Kafka-Connect: This is the most widely used maintained image for Kafka Connect. It’s part of the Confluent Platform, optimized for Connect workflows, and comes with common connectors pre-installed (you can add more easily). The image tag is confluentinc/cp-kafka-connect.
• Apache Kafka Official Image: While the main apache/kafka image includes the full Kafka distribution, you can configure it to run only the Connect component (no broker/ZooKeeper startup).

2. Configure the Image to Point to Your Ambari Cluster #

Once you’ve picked an image, you just need to set environment variables to connect to your existing Kafka and ZooKeeper services. Here’s a typical example using the Confluent image:

docker run -d --name kafka-connect \
  -e BOOTSTRAP_SERVERS="ambari-kafka-broker-1:9092,ambari-kafka-broker-2:9092" \
  -e ZOOKEEPER_CONNECT="ambari-zk-1:2181,ambari-zk-2:2181" \
  -e CONNECT_GROUP_ID="ambari-connect-cluster" \
  -e CONNECT_CONFIG_STORAGE_TOPIC="connect-configs" \
  -e CONNECT_OFFSET_STORAGE_TOPIC="connect-offsets" \
  -e CONNECT_STATUS_STORAGE_TOPIC="connect-statuses" \
  -e CONNECT_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  confluentinc/cp-kafka-connect:latest

Key notes on the variables:

• BOOTSTRAP_SERVERS: Use the broker addresses from your Ambari Kafka service (you can find these in Ambari’s Kafka configs under listeners or advertised.listeners).
• ZOOKEEPER_CONNECT: Only required if your Connect setup relies on ZK (some older connector implementations or legacy configs might need this; newer versions can use Kafka for storage).
• The *_STORAGE_TOPIC variables: These topics store Connect’s configuration, offsets, and status. Ensure your Ambari Kafka cluster allows Connect to create these topics (or create them manually via kafka-topics.sh first if you prefer).

3. Handle Ambari Cluster Security (If Applicable) #

If your Ambari-managed Kafka/ZooKeeper uses authentication (like SASL/PLAIN, Kerberos) or authorization (ACLs), you’ll need to pass those configs to the container:

• For Kerberos: Mount your krb5.conf and JAAS config file into the container, then set the KAFKA_OPTS environment variable to point to the JAAS file.
• For SASL: Add environment variables like CONNECT_SASL_JAAS_CONFIG and CONNECT_SECURITY_PROTOCOL to match your Ambari Kafka’s security settings.

4. Add Custom Connectors (If Needed) #

If you need connectors that aren’t pre-installed in the image, you can mount a directory of connector JARs into the image’s plugin directory:

• For Confluent images, the default plugin path is /usr/share/java. Mount your local connector folder like this:

  -v /path/to/your/connectors:/usr/share/java/custom-connectors
  

  Then add CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/java/custom-connectors" to your environment variables.

How to Verify an Image Is Right for You #

Before pulling an image, check its documentation or Dockerfile to confirm:

• It doesn’t default to starting a Kafka broker or ZooKeeper server.
• It supports configuring external Kafka/ZooKeeper via environment variables or config files.

内容的提问来源于stack exchange，提问作者Phil Baines