Docker部署ClickHouse与Kafka集成连接失败问题求助

Hey there! Let's work through your Kafka + ClickHouse Docker integration issues—this is a super common pitfall with container networking, so let's break it down step by step.

The Core Problem: localhost Doesn't Mean What You Think in Docker #

When you run services in separate Docker containers, localhost inside one container refers only to that container itself. So when ClickHouse tries to connect to localhost:9092, it's looking for a Kafka instance running inside its own container (which doesn't exist), hence the "connection refused" error. Same goes for ZooKeeper—localhost:2181 points to ClickHouse's own container, not your ZooKeeper container.

Fix 1: Correct KAFKA_ADVERTISED_LISTENERS Configuration #

Kafka needs to advertise two sets of listeners: one for communication between containers (internal) and one for tools running on your host machine (external). Here's how to set this up (example for Docker Compose):

environment:
  KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
  KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_HOST://0.0.0.0:29092
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

• PLAINTEXT://kafka:9092: This is the internal listener—kafka is the service name of your Kafka container (use whatever name you assigned in your Docker setup). Other containers in the same network can resolve this name to Kafka's container IP.
• PLAINTEXT_HOST://localhost:29092: This is for your host machine's Kafka tools (which is why you could see data arriving there earlier).

Fix 2: Update ClickHouse's Kafka Bootstrap Servers #

In your ClickHouse Kafka table engine configuration, replace localhost:9092 with the internal listener address kafka:9092. Here's an example table creation statement:

CREATE TABLE kafka_source (
    id UInt64,
    payload String
) ENGINE = Kafka
SETTINGS
    kafka_broker_list = 'kafka:9092',
    kafka_topic_list = 'your_api_topic',
    kafka_group_name = 'clickhouse_consumer_group',
    kafka_format = 'JSONEachRow';

This tells ClickHouse to connect to the Kafka container directly via the Docker network's internal DNS.

Fix 3: Resolve ZooKeeper Connection Failures #

Same logic applies to ZooKeeper—update Kafka's KAFKA_ZOOKEEPER_CONNECT setting to use your ZooKeeper container's service name instead of localhost. For example:

KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

Make sure your ZooKeeper container is named zookeeper (or adjust the name to match your setup).

Critical Check: Ensure All Containers Share the Same Docker Network #

Docker automatically creates a shared network for services in a Docker Compose file, but if you're running containers individually, you need to explicitly put them in the same network. Run this command to create a network (if you don't have one):

docker network create kafka-clickhouse-net

Then add --network kafka-clickhouse-net to each docker run command for Kafka, ZooKeeper, and ClickHouse. This ensures container names are resolvable via DNS.

Quick Test to Validate Connectivity #

To confirm ClickHouse can reach Kafka, exec into the ClickHouse container and run:

docker exec -it <clickhouse-container-name> nc -zv kafka 9092

If you see "succeeded!", the network connection is working—any remaining issues will be with table configuration or Kafka topic permissions.

Final Notes #

• The DNS error in ClickHouse logs is directly tied to using localhost instead of the container service name—fixing the bootstrap servers will resolve this.
• Your host machine's Kafka tools will still use localhost:29092 to connect, which is why you could see data arriving earlier.

内容的提问来源于stack exchange，提问作者Harshita