Docker部署Python Kafka消费者遭遇NoBrokersAvailable错误求助

我把用Kafka通信的项目改成了Docker容器化部署，消费端用Python实现，但一直出现NoBrokersAvailable错误。试过两款不同的Kafka镜像和多种配置组合，消费者还是连不上Kafka Broker。

控制台日志 #

Traceback (most recent call last):
 File "main.py", line 15, in <module>
  kafka_consumer = kafka_client.prepare_kafka_consumer()
 File "/-----/client/kafka_client.py", line 17, in prepare_kafka_consumer
  consumer = KafkaConsumer(self.topic,
 File "/usr/local/lib/python3.8/site-packages/kafka/consumer/group.py", line 356, in __init__
  self._client = KafkaClient(metrics=self._metrics, **self.config)
 File "/usr/local/lib/python3.8/site-packages/kafka/client_async.py", line 244, in __init__
  self.config['api_version'] = self.check_version(timeout=check_timeout)
 File "/usr/local/lib/python3.8/site-packages/kafka/client_async.py", line 900, in check_version
  raise Errors.NoBrokersAvailable()
 kafka.errors.NoBrokersAvailable: NoBrokersAvailable

Docker Compose配置 #

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:3.2.0
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    restart: always

  kafka:
    image: confluentinc/cp-kafka:3.2.0
    hostname: kafka
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:19091,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    restart: always

  app:
    build:
        context: .
        dockerfile: DockerFile
    ports:
      - "8080:8080"
    depends_on:
      - kafka

Config.py配置 #

class Settings:
    KAFKA_BROKERS = ['kafka:19091']
    KAFKA_TOPIC = "topic"
    MAIL_USERNAME = "mail"
    MAIL_PASSWORD = "pass"


settings = Settings()

问题分析与解决方案 #

核心问题：Kafka未正确监听内部端口 #

当前Kafka配置只设置了KAFKA_ADVERTISED_LISTENERS，但缺少KAFKA_LISTENERS参数。这会导致Kafka实际上没有监听配置的19091内部端口，容器内的Python应用自然无法连接。

修复步骤： #

1. 补充Kafka监听配置
   在kafka服务的environment中添加：

   KAFKA_LISTENERS: LISTENER_DOCKER_INTERNAL://0.0.0.0:19091,LISTENER_DOCKER_EXTERNAL://0.0.0.0:9093
   

   这个配置让Kafka在容器内监听19091（内部通信）和9093（外部主机通信）端口，和KAFKA_ADVERTISED_LISTENERS的配置对应。

2. 处理容器启动顺序问题
   depends_on仅保证kafka容器在app之前启动，但不保证Kafka服务完全就绪。可以在app的启动脚本中添加等待逻辑，比如用nc命令检查kafka的19091端口是否可用：

   # 等待Kafka就绪
   while ! nc -z kafka 19091; do
     sleep 1
   done
   # 启动Python应用
   python main.py
   

3. 验证版本兼容性
   cp-kafka:3.2.0对应的Kafka版本是0.11.0.x，确保Python的kafka-client版本与之兼容。推荐使用kafka-python==2.0.2或相近版本，避免因版本不匹配导致连接问题。

验证修复 #

修改配置后，重新启动容器：

docker-compose down && docker-compose up --build

此时Python消费者应该能正常连接到Kafka Broker。

内容的提问来源于stack exchange，提问作者Berkay Adyemir