Docker部署Kafka报错：无法连接ID为1的节点求助

问题背景 #

尝试向本地Docker部署的Kafka发送数据，但生产者始终无法建立连接，以下是相关配置、代码及报错信息：

基础设施配置（docker-compose.yml） #

version: "3.8"
    
services:
    zookeeper:
        image: bitnami/zookeeper:latest
        ports:
            - 2181:2181
        environment:
            - ALLOW_ANONYMOUS_LOGIN=yes

    kafka:
        image: bitnami/kafka:latest
        ports:
            - 9092:9092
            - 9093:9093
        environment:
            - KAFKA_BROKER_ID=1
            - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
            - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
            - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
            - ALLOW_PLAINTEXT_LISTENER=yes
            - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT
            - KAFKA_CFG_LISTENERS=CLIENT://:9092
            - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092
            - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
        depends_on:
            - zookeeper

生产者代码 #

import asyncio

from aiokafka import AIOKafkaProducer


async def send_to_kafka():
    producer = AIOKafkaProducer(
        bootstrap_servers='localhost:9092',
        enable_idempotence=True)
    await producer.start()
    try:
        await producer.send_and_wait("my_topic", b"Super message")
    finally:
        await producer.stop()

asyncio.run(send_to_kafka())

遇到的错误 #

错误1：使用localhost:9092时 #

Traceback (most recent call last):
  File "tests/test_producer.py", line 16, in <module>
    asyncio.run(send_to_kafka())
  File "/usr/local/Cellar/python@3.8/3.8.20/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/Cellar/python@3.8/3.8.20/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "tests/test_producer.py", line 10, in send_to_kafka
    await producer.start()
  File "/Users/aamoskalenko/0_root_folder/02_dev/51_fetcher_new/Fetcher-Service/venv/lib/python3.8/site-packages/aiokafka/producer/producer.py", line 352, in start
    await self.client.bootstrap()
  File "/Users/aamoskalenko/0_root_folder/02_dev/51_fetcher_new/Fetcher-Service/venv/lib/python3.8/site-packages/aiokafka/client.py", line 269, in bootstrap
    self._api_version = await self.check_version()
  File "/Users/aamoskalenko/0_root_folder/02_dev/51_fetcher_new/Fetcher-Service/venv/lib/python3.8/site-packages/aiokafka/client.py", line 564, in check_version
    raise KafkaConnectionError(f"No connection to node with id {node_id}")
aiokafka.errors.KafkaConnectionError: KafkaConnectionError: No connection to node with id 1
Unclosed AIOKafkaProducer

错误2：使用kafka:9092时 #

File "/Users/aamoskalenko/0_root_folder/02_dev/51_fetcher_new/Fetcher-Service/venv/lib/python3.8/site-packages/aiokafka/client.py", line 265, in bootstrap
    raise KafkaConnectionError(f"Unable to bootstrap from {self.hosts}")
aiokafka.errors.KafkaConnectionError: KafkaConnectionError: Unable to bootstrap from [('kafka', 9092, <AddressFamily.AF_UNSPEC: 0>)]
Unclosed AIOKafkaProducer

问题原因 #

1. 配置重复覆盖：Kafka环境变量中重复定义了KAFKA_CFG_LISTENERS和KAFKA_CFG_ADVERTISED_LISTENERS，后定义的CLIENT://kafka:9092覆盖了之前的PLAINTEXT://127.0.0.1:9092，导致外部客户端无法通过localhost访问。
2. 地址无法解析：advertised_listeners设置的kafka:9092是Docker内部域名，宿主机无法解析该地址，外部生产者自然无法连接。

解决方案 #

修改docker-compose.yml中的Kafka环境变量，同时支持宿主机外部访问和容器内部通信：

version: "3.8"
    
services:
    zookeeper:
        image: bitnami/zookeeper:latest
        ports:
            - 2181:2181
        environment:
            - ALLOW_ANONYMOUS_LOGIN=yes

    kafka:
        image: bitnami/kafka:latest
        ports:
            - 9092:9092  # 宿主机访问端口
            - 9093:9093  # 容器内部通信端口
        environment:
            - KAFKA_BROKER_ID=1
            - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
            - ALLOW_PLAINTEXT_LISTENER=yes
            # 定义两个监听器：外部访问+内部通信
            - KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9092,INTERNAL://0.0.0.0:9093
            # 映射监听器对应的协议
            - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,INTERNAL:PLAINTEXT
            # 声明外部可访问地址和内部容器通信地址
            - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092,INTERNAL://kafka:9093
            # 内部broker通信使用INTERNAL监听器
            - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL
        depends_on:
            - zookeeper

重启Kafka服务：

docker-compose down && docker-compose up -d

生产者代码保持不变（使用localhost:9092），再次运行即可正常发送消息。

补充说明 #

• PLAINTEXT://127.0.0.1:9092：供宿主机上的客户端（如Python生产者）连接使用
• INTERNAL://kafka:9093：供Docker内部其他服务（如消费者容器）连接Kafka使用
• 确保9092端口未被其他进程占用

内容的提问来源于stack exchange，提问作者mascai