Docker Compose中容器无法连接Kafka Broker，报Error 111

问题现象 #

• 同一Docker Compose环境中，my_container容器无法连接Kafka Broker，报错：

2023-02-22 14:24:00 ERROR:kafka.conn:Connect attempt to <BrokerConnection node_id=bootstrap-0 host=0.0.0.0:29092 [IPv4 ('0.0.0.0', 29092)]> returned error 111. Disconnecting.

• 此前通过配置advertised listeners解决过同类问题，但当前失效
• 尝试修改容器名称、添加links、配置depends_on均无效果
• docker ps显示端口映射正常：

CONTAINER ID   IMAGE                             COMMAND                  CREATED          STATUS          PORTS                                              NAMES
b1892844f99a   confluentinc/cp-kafka:7.3.0       "/etc/confluent/dock…"   26 seconds ago   Up 24 seconds   0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp   broker

环境配置 #

Docker Compose 配置 #

zookeeper:
    image: confluentinc/cp-zookeeper:7.3.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

broker:
    image: confluentinc/cp-kafka:7.3.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TOOLS_LOG4J_LOGLEVEL: ERROR

my_container:
  image: my_container
  container_name: my_container
  ports:
    - "8000:8000"

Python生产者代码（kafka-python） #

producer = KafkaProducer(bootstrap_servers="0.0.0.0:29092", 
                        value_serializer=lambda user: json.dumps(event).encode('utf-8'),
                        api_version=(0,11,5),
                        retries = 3,
                        batch_size = 161879)

解决方案 #

1. 修正Kafka监听器配置 #

当前KAFKA_ADVERTISED_LISTENERS中，PLAINTEXT_HOST使用localhost无法被Docker内部容器正确解析。需添加Docker内部通信专用的监听器，同时保留外部访问配置：

# 修改broker的environment字段
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,DOCKER_INTERNAL:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092,DOCKER_INTERNAL://broker:29092

2. 调整Python生产者的Bootstrap地址 #

Docker Compose内部容器通过服务名通信，my_container需使用broker:9092（对应PLAINTEXT监听器）作为连接地址：

producer = KafkaProducer(bootstrap_servers="broker:9092", 
                        value_serializer=lambda user: json.dumps(event).encode('utf-8'),
                        api_version=(0,11,5),
                        retries = 3,
                        batch_size = 161879)

3. 确保容器网络互通 #

Docker Compose默认创建专属网络，所有服务默认加入该网络。若my_container有自定义网络配置，需确保与broker在同一网络：

# 在my_container配置中添加
networks:
  - default

4. 清理旧容器与重启服务 #

Kafka监听器配置会持久化到Zookeeper，修改后需彻底清理旧容器与数据：

# 停止并删除所有容器
docker-compose down
# 删除Zookeeper和Kafka的持久化数据（若有挂载卷）
docker volume rm $(docker volume ls -q | grep -E 'zookeeper|kafka')
# 重新启动服务
docker-compose up -d

验证方法 #

1. 进入my_container测试与broker的连通性：

docker exec -it my_container ping broker
# 测试端口连通
docker exec -it my_container telnet broker 9092

2. 运行Python生产者代码，检查是否能成功发送消息。

内容的提问来源于stack exchange，提问作者LauraAlice