MAC M1环境Docker中Kafka无法连接Zookeeper求助

问题描述 #

我在M1 Mac上使用Docker运行Zookeeper和Kafka，启动脚本如下：

Zookeeper启动脚本 #

docker run --name zookeeper -p 2181:2181 -e ZOOKEEPER_TICK_TIME=2000 -e ZOOKEEPER_INIT_LIMIT=10 -e ZOOKEEPER_SYNC_LIMIT=5 -e ZOOKEEPER_CLIENT_PORT=2181 confluentinc/cp-zookeeper

Kafka启动脚本 #

docker run --name kafka -p 9092:9092 -e KAFKA_ZOOKEEPER_CONNECT=0.0.0.0:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://0.0.0.0:9092  confluentinc/cp-kafka

Zookeeper已成功运行，但Kafka无法连接Zookeeper，报错信息如下：

[2022-11-18 21:15:48,461] INFO Opening socket connection to server 0.0.0.0/0.0.0.0:2181. (org.apache.zookeeper.ClientCnxn)
[2022-11-18 21:15:48,462] INFO SASL config status: Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-11-18 21:15:48,462] WARN Session 0x0 for sever 0.0.0.0/0.0.0.0:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)
[2022-11-18 21:15:49,593] INFO Opening socket connection to server 0.0.0.0/0.0.0.0:2181. (org.apache.zookeeper.ClientCnxn)
[2022-11-18 21:15:49,594] INFO SASL config status: Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-11-18 21:15:49,595] WARN Session 0x0 for sever 0.0.0.0/0.0.0.0:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)
[2022-11-18 21:15:50,722] INFO Opening socket connection to server 0.0.0.0/0.0.0.0:2181. (org.apache.zookeeper.ClientCnxn)
[2022-11-18 21:15:50,722] INFO SASL config status: Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-11-18 21:15:50,723] WARN Session 0x0 for sever 0.0.0.0/0.0.0.0:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)

尝试telnet 0.0.0.0 2181时出现以下错误：

Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
Connection closed by foreign host.

核心问题 #

在Docker容器环境中，0.0.0.0指向容器自身的网络接口，Kafka容器用这个地址无法访问宿主机上的Zookeeper容器。需要使用Docker容器间的网络通信方式来连接。

修改配置步骤 #

方法一：直接使用容器名连接（简单快捷） #

修改Kafka启动脚本中的KAFKA_ZOOKEEPER_CONNECT参数为zookeeper:2181，同时调整KAFKA_ADVERTISED_LISTENERS为PLAINTEXT://localhost:9092（方便宿主机应用连接Kafka）：

docker run --name kafka -p 9092:9092 -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 confluentinc/cp-kafka

方法二：创建自定义Docker网络（更稳定） #

如果容器名访问存在问题，可以创建自定义网络让两个容器处于同一网络环境：

1. 创建自定义网络：

   docker network create kafka-net
   

2. 使用该网络启动Zookeeper：

   docker run --name zookeeper --network kafka-net -p 2181:2181 -e ZOOKEEPER_TICK_TIME=2000 -e ZOOKEEPER_INIT_LIMIT=10 -e ZOOKEEPER_SYNC_LIMIT=5 -e ZOOKEEPER_CLIENT_PORT=2181 confluentinc/cp-zookeeper
   

3. 使用同一网络启动Kafka，并连接Zookeeper：

   docker run --name kafka --network kafka-net -p 9092:9092 -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 confluentinc/cp-kafka
   

验证连接 #

启动成功后，可以进入Kafka容器，用Zookeeper客户端验证连接是否正常：

docker exec -it kafka zkCli.sh -server zookeeper:2181

若能进入Zookeeper交互界面，说明连接成功。

内容的提问来源于stack exchange，提问作者Arijit Das