Docker部署Kafka Broker无法从外部/其他容器访问的问题求助

问题现象 #

在Docker中分别运行Kafka Broker和独立ZooKeeper容器，本地能正常访问ZooKeeper，但访问Broker生产消息时抛出如下错误：

[Timestamp] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 65 : {tmpc=INVALID_REPLICATION_FACTOR} (org.apache.kafka.clients.NetworkClient)
[Timestamp] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 66 : {tmpc=INVALID_REPLICATION_FACTOR} (org.apache.kafka.clients.NetworkClient)

尝试过端口映射、容器链接、自定义Docker网络，均能访问ZooKeeper，但无法连接Broker生产消息。仅当在交互式基础镜像容器中直接启动Broker时，才能正常连接，确认是网络配置问题。

环境配置 #

基础镜像 #

基于Ubuntu，仅解压Kafka并安装Java。

ZooKeeper镜像配置 #

FROM IMAGE
COPY ./zookeeper_setup.sh  zookeeper_setup.sh
ENTRYPOINT ["sh","zookeeper_setup.sh"]

zookeeper_setup.sh内容：

sed -i "s%dataDir=/tmp/zookeeper%dataDir=${LOGDIR}%g" ./config/zookeeper.properties
sh ./bin/zookeeper-server-start.sh ./config/zookeeper.properties 

Kafka镜像配置 #

FROM IMAGE # 同基础镜像
COPY ./broker_setup.sh  broker_setup.sh
ENTRYPOINT ["sh","broker_setup.sh"]

broker_setup.sh内容：

sed -i "s#log.dirs=/tmp/kafka-logs#log.dirs=${LOGDIR}#g" ./config/server.properties
sed -i "s#broker.id=0#broker.id=${BROKERID}#g" ./config/server.properties
sed -i "s@#listeners=PLAINTEXT://:9092@listeners=INTERNAL://:2${KAFKAPORT},EXTERNAL://:${KAFKAPORT}@g" ./config/server.properties
sed -i "s/zookeeper.connect=localhost:2181/zookeeper.connect=${ZOOKEEPERCONNECT}/g" ./config/server.properties
echo -e "\nlistener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT" >> ./config/server.properties
echo -e "\ninter.broker.listener.name=INTERNAL" >> ./config/server.properties
sed -i "s~#advertised.listeners=PLAINTEXT://your.host.name:9092~advertised.listeners=INTERNAL://localhost:2${KAFKAPORT},EXTERNAL://localhost:${KAFKAPORT}~g" ./config/server.properties
sh ./bin/kafka-server-start.sh ./config/server.properties

容器启动命令 #

方式1：容器链接

docker run -d -e LOGDIR=/kafka/logs/zookeeper/ -p 2281:2181 -v [OutsideDir]:/kafka/logs/zookeeper/ --name zookeeper-client -t Image

docker run -d --link zookeeper-client:zookeeper --name broker0 -e LOGDIR=/kafka/logs/kafka-logs-1/ -e BROKERID=0 -e KAFKAPORT=9093 -e ZOOKEEPERCONNECT=zookeeper:2181 -e ALLOW_PLAINTEXT_LISTENER=yes -p 8093:9093 -p 28093:29093 -v [LogDir_2]:/kafka/logs/kafka-logs-1/ -t Image

方式2：自定义Docker网络

docker network create app-network

docker run -d --network app-network --network-alias app-net-zoo -e LOGDIR=/kafka/logs/zookeeper/ -p 2281:2181 -v [logDir]:/kafka/logs/zookeeper/ --name zookeeper-client -t image

docker run --network app-network --network-alias app-net-broker --name broker0 -e LOGDIR=/kafka/logs/kafka-logs-1/ -e BROKERID=0 -e KAFKAPORT=9093 -e ZOOKEEPERCONNECT=app-net-zoo:2181 -e ALLOW_PLAINTEXT_LISTENER=yes -p 8093:9093 -p 28093:29093 -v [logDir_2]:/kafka/logs/kafka-logs-1/ -t image

docker run --network app-network --network-alias app-net-it -it Image_BASE

交互式容器测试结果：

• 成功：kafka-topics.sh --create --topic Name_T --zookeeper app-net-zoo:2181
• 失败：kafka-console-producer.sh --broker-list app-net-broker:9093 --topic tempC

问题根源 #

核心问题在于Kafka的advertised.listeners配置错误：

• 当Broker运行在Docker容器中时，advertised.listeners中配置的localhost是容器内部的localhost，而非Docker网络中的容器别名或主机IP。
• 客户端（同网络内的其他容器或本地）请求Broker元数据时，Broker返回的是localhost:9093和localhost:29093，客户端会尝试连接这个地址，但在客户端的网络环境中，localhost指向自身而非Broker容器，导致无法建立连接。
• INVALID_REPLICATION_FACTOR是元数据获取失败导致的次生错误，并非主题复制因子配置问题。

解决方案 #

修改Kafka的advertised.listeners配置，根据不同场景设置正确的地址：

1. 同Docker网络内的客户端访问 #

将advertised.listeners中的localhost替换为Broker容器的网络别名或容器名称。在broker_setup.sh中修改对应行：

# 新增环境变量ADVERTISED_HOST，启动容器时传入容器别名
echo -e "\nadvertised.listeners=INTERNAL://${ADVERTISED_HOST}:2${KAFKAPORT},EXTERNAL://${ADVERTISED_HOST}:${KAFKAPORT}" >> ./config/server.properties

启动Broker容器时传入：-e ADVERTISED_HOST=app-net-broker

2. 本地主机访问 #

如果需要本地主机访问，EXTERNAL监听器的地址应设置为主机的IP或localhost，同时匹配端口映射：

echo -e "\nadvertised.listeners=INTERNAL://${ADVERTISED_HOST}:2${KAFKAPORT},EXTERNAL://localhost:8093" >> ./config/server.properties

这里8093对应启动命令中的-p 8093:9093端口映射。

完整修改后的broker_setup.sh示例 #

sed -i "s#log.dirs=/tmp/kafka-logs#log.dirs=${LOGDIR}#g" ./config/server.properties
sed -i "s#broker.id=0#broker.id=${BROKERID}#g" ./config/server.properties
sed -i "s@#listeners=PLAINTEXT://:9092@listeners=INTERNAL://:2${KAFKAPORT},EXTERNAL://:${KAFKAPORT}@g" ./config/server.properties
sed -i "s/zookeeper.connect=localhost:2181/zookeeper.connect=${ZOOKEEPERCONNECT}/g" ./config/server.properties
echo -e "\nlistener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT" >> ./config/server.properties
echo -e "\ninter.broker.listener.name=INTERNAL" >> ./config/server.properties
# 场景1：同Docker网络内访问
echo -e "\nadvertised.listeners=INTERNAL://${ADVERTISED_HOST}:2${KAFKAPORT},EXTERNAL://${ADVERTISED_HOST}:${KAFKAPORT}" >> ./config/server.properties
# 场景2：本地主机访问（注释场景1，启用此配置）
# echo -e "\nadvertised.listeners=INTERNAL://${ADVERTISED_HOST}:2${KAFKAPORT},EXTERNAL://localhost:8093" >> ./config/server.properties
sh ./bin/kafka-server-start.sh ./config/server.properties

验证 #

修改配置后重新构建Kafka镜像，启动容器：

• 同网络内客户端：kafka-console-producer.sh --broker-list app-net-broker:9093 --topic tempC
• 本地主机：kafka-console-producer.sh --broker-list localhost:8093 --topic tempC
  此时应能正常生产消息，不再出现元数据获取错误。

内容的提问来源于stack exchange，提问作者tempUser