Docker Kafka容器与Python消费者容器无法连接问题排查

问题描述 #

我有两个Docker容器：一个运行Kafka（容器名broker），另一个运行带Kafka客户端的Python程序。两个容器处于同一自定义网络my-network，但运行Python连接代码时持续出现连接拒绝错误：

%3|1725885842.079|FAIL|rdkafka#consumer-1| [thrd:localhost:172/bootstrap]: localhost:172/bootstrap: Connect to ipv4#127.0.0.1:172 failed: Connection refused (after 0ms in state CONNECT)

Docker网络配置信息 #

[
    {
        "Name": "my-network",
        "Id": "c920653a8c4d9e068887bce98dab2f5bf2f554e112a2cbd337d3a7feefc45b5b",
        "Created": "2024-09-06T14:39:28.375456878Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "02bf3754f2b27276db07762cf6237872dfa36bcd85d4e1feb7d7d6f13ff97a82": {
                "Name": "broker",
                "EndpointID": "13d97b89c9c4980c7c71b25e8aba4fc94d9a60b344720fca44fb66adc40af15b",        
                "MacAddress": "02:42:ac:14:00:03",
                "IPv4Address": "172.20.0.3/16",
                "IPv6Address": ""
            },
            "8fb0fab1c6c8f97033739a5936661e04df02de25c6a77cc9bb4c8259b21095a7": {
                "Name": "agitated_shirley",
                "EndpointID": "9c14de0df28233489e56b5354bd7621239eca948b5c4c7adabf3fccf6a98779d",        
                "MacAddress": "02:42:ac:14:00:02",
                "IPv4Address": "172.20.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Python连接代码 #

from typing import Union
from fastapi import FastAPI
from pydantic import BaseModel
from confluent_kafka import Consumer

conf = {'bootstrap.servers': 'localhost:172.20.0.3', 'group.id':'main-group','auto.offset.reset':'smallest'}

consumer = Consumer(conf)

running = True

我已尝试重新创建网络和容器，但问题依旧，请问哪里操作出错了？

问题分析与解决方案 #

1. Bootstrap地址格式完全错误 #

你的bootstrap.servers配置格式写反，正确格式应为[主机名/IP]:[端口]，而非localhost:172.20.0.3。同时，Kafka默认的bootstrap端口是9092，你写的172是子网段前缀，并非端口号。

2. 错误使用localhost访问Kafka容器 #

在Docker容器内部，localhost指向容器自身，而非同一网络中的Kafka容器。同一Docker网络内，应使用Kafka容器的名称（broker）或容器IP（172.20.0.3）来访问。

修正后的Python配置 #

方案一：使用容器名（推荐，IP可能随容器重建变化）

conf = {'bootstrap.servers': 'broker:9092', 'group.id':'main-group','auto.offset.reset':'smallest'}

方案二：使用容器IP

conf = {'bootstrap.servers': '172.20.0.3:9092', 'group.id':'main-group','auto.offset.reset':'smallest'}

额外注意事项 #

确保启动Kafka容器时，正确配置KAFKA_ADVERTISED_LISTENERS环境变量，让Kafka向客户端返回可访问的地址。例如：

docker run -d --name broker --network my-network \
  -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://broker:9092 \
  confluentinc/cp-kafka:latest

内容的提问来源于stack exchange，提问作者stefan