Docker环境下Kafka与Python应用DNS解析失败问题求助

遇到以下错误：

WARNING kafka.conn:conn.py:1527 DNS lookup failed for kafka:9092, exception was [Errno -3] Temporary failure in name resolution. Is your advertised.listeners (called advertised.host.name before Kafka 9) correct and resolvable?
ERROR kafka.conn:conn.py:315 DNS lookup failed for kafka:9092 (AddressFamily.AF_UNSPEC)

以及：

2022-12-07T20:06:57+0000|/usr/local/lib/python3.9/site-packages/kafka/conn.py:418|ERROR|Connect attempt to <BrokerConnection node_id=bootstrap-0 host=kafka:9092 [IPv4 ('171.15.0.1', 9092)]> returned error 111. Disconnecting.

Docker Compose 配置 #

zookeeper:
  image: confluentinc/cp-zookeeper:latest
  networks:
    - app_network
  environment:
    ZOOKEEPER_CLIENT_PORT: 2181
    ZOOKEEPER_TICK_TIME: 2000
  ports:
    - 22181:2181

kafka:
  image: confluentinc/cp-kafka:latest
  container_name: kafka
  depends_on:
    - zookeeper
  ports:
    - 29092:29092
  environment:
    KAFKA_BROKER_ID: 1
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
    KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  networks:
    - app_network

pythonapp:
  build: ../../
  container_name: myapp
  networks:
    - app_network
  depends_on:
    - kafka

networks:
  app_network:
    driver: bridge

Python 消费者代码 #

def kafka_consume(self):
    consumer = KafkaConsumer(
        self.topic_name,
        auto_offset_reset='earliest',
        bootstrap_servers=['kafka:9092'],
        api_version=(0, 10),
        consumer_timeout_ms=1000
    )
    consumer.subscribe([self.topic_name])

已知 Kafka 容器内 ping 功能正常，需解决连接失败问题。

1. 验证容器网络连通性 #

先确认 Python 应用容器能正常解析 kafka 主机名：

• 进入 Python 容器：docker exec -it myapp bash
• 执行 ping kafka，如果无法解析，说明网络连接异常。检查 pythonapp 的 networks 配置是否正确（当前配置无误，但可能存在容器启动顺序问题）。

2. 修复 Kafka 监听配置 #

确保 Kafka 正确监听 9092 端口：

• 进入 Kafka 容器：docker exec -it kafka bash
• 执行 netstat -tulpn | grep 9092，确认端口已被监听。
• 若未监听，在 Kafka 的 environment 中添加显式监听配置：

  KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_HOST://0.0.0.0:29092
  

  让 Kafka 监听所有网卡，避免仅绑定 localhost。

3. 处理容器启动顺序问题 #

depends_on 仅保证容器启动顺序，不保证 Kafka 服务完全就绪。Python 应用启动时 Kafka 可能还在初始化，导致连接失败。可在代码中添加重试逻辑：

from kafka.errors import KafkaError
import time

def kafka_consume(self):
    consumer = None
    max_retries = 5
    retry_count = 0
    while retry_count < max_retries:
        try:
            consumer = KafkaConsumer(
                self.topic_name,
                auto_offset_reset='earliest',
                bootstrap_servers=['kafka:9092'],
                api_version=(0, 10),
                consumer_timeout_ms=1000
            )
            consumer.subscribe([self.topic_name])
            break
        except KafkaError as e:
            retry_count += 1
            print(f"连接失败，重试 {retry_count}/{max_retries}: {e}")
            time.sleep(3)
    if not consumer:
        raise Exception("多次重试后仍无法连接 Kafka")

4. 检查 Docker 网络状态 #

如果 ping kafka 在 Python 容器中失败，查看网络详情确认容器归属：

• 执行 docker network inspect app_network
• 确认 kafka 和 pythonapp 容器都在该网络中，且 Kafka 容器的 IP 配置正常。

内容的提问来源于stack exchange，提问作者Angel