如何配置Docker Compose部署的Apache Kafka支持宿主机外部访问

配置修改说明 #

Kafka容器外部访问的核心是配置两套监听规则，一套用于容器内部集群通信，一套用于宿主机外部访问，需要在docker-compose中补充端口映射和环境变量配置。

完整修改后的docker-compose.yml #

services: 
  zookeeper-1:
    container_name: zookeeper-1
    image: zookeeper:2.7.0
    build:
      context: ./zookeeper
    volumes:
      - ./config/zookeeper-1/zookeeper.properties:/kafka/config/zookeeper.properties
    ports:
      - "2181:2181"

  kafka-1:
    container_name: kafka-1
    image: kafka:2.7.0
    build: 
        context: .
    volumes:
      - ./config/kafka-1/server.properties:/kafka/config/server.properties
      - ./data/kafka-1/:/tmp/kafka-logs/
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    depends_on:
      - zookeeper-1

  kafka-2:
    container_name: kafka-2
    image: kafka:2.7.0
    build: 
        context: .
    volumes:
      - ./config/kafka-2/server.properties:/kafka/config/server.properties
      - ./data/kafka-2/:/tmp/kafka-logs/
    ports:
      - "9093:9092"
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    depends_on:
      - zookeeper-1

  kafka-3:
    container_name: kafka-3
    image: kafka:2.7.0
    build: 
        context: .
    volumes:
      - ./config/kafka-3/server.properties:/kafka/config/server.properties
      - ./data/kafka-3/:/tmp/kafka-logs/
    ports:
      - "9094:9092"
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    depends_on:
      - zookeeper-1

可选配置方式 #

如果你不想通过环境变量配置，也可以直接修改你挂载的./config/kafka-x/server.properties文件，加入以下配置即可，和环境变量效果完全一致：

broker.id=1 # 每个broker对应修改为2、3
zookeeper.connect=zookeeper-1:2181
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://127.0.0.1:9092 # 每个broker对应修改为9093、9094
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
inter.broker.listener.name=PLAINTEXT

验证测试 #

配置完成后重启容器，使用以下kafka-python代码验证外部访问是否正常：

生产者测试

from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers=['127.0.0.1:9092','127.0.0.1:9093','127.0.0.1:9094'])
# 发送测试消息
producer.send('test_topic', value=b'hello from docker external')
producer.flush()

消费者测试

from kafka import KafkaConsumer
consumer = KafkaConsumer(
    'test_topic',
    bootstrap_servers=['127.0.0.1:9092','127.0.0.1:9093','127.0.0.1:9094'],
    auto_offset_reset='earliest'
)
for message in consumer:
    print(f"收到消息：{message.value.decode('utf-8')}")

内容的提问来源于stack exchange，提问作者TennisTechBoy