Kafka事务初始化失败：Coordinator not available问题求助

在本地macOS Monterey机器上通过Docker部署Kafka和Zookeeper实例，此前运行正常。近期尝试实现Kafka事务时，持续出现以下错误：

KafkaError{code=COORDINATOR_NOT_AVAILABLE,val=15,str="Failed to initialize Producer ID: Broker: Coordinator not available"}

多次重启服务（包括删除本地数据卷）均无法解决该问题。

可复现的Python最小示例 #

#!/usr/bin/python3

import confluent_kafka
import sys

topic = "test-topic"
config = {
    "bootstrap.servers": "localhost:9092",
    "enable.idempotence": True,
    "transactional.id": "test-0",
}

producer = confluent_kafka.Producer(config)
producer.init_transactions(300.0)

使用的Docker Compose配置 #

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181
    volumes:
      - ./data/zookeeper/data:/var/lib/zookeeper/data
      - ./data/zookeeper/log:/var/lib/zookeeper/log

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
      - 29092:29092
    hostname: kafka
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./data/kafka:/var/lib/kafka/data

超时设置为2.0秒时的报错输出 #

clemens@morticia kafka % python3.9 transaction.py
%4|1661869211.606|GETPID|rdkafka#producer-1| [thrd:main]: Failed to acquire transactional PID from broker TxnCoordinator/1: Broker: Coordinator not available: retrying
%4|1661869212.115|GETPID|rdkafka#producer-1| [thrd:main]: Failed to acquire transactional PID from broker TxnCoordinator/1: Broker: Coordinator not available: retrying
%4|1661869212.629|GETPID|rdkafka#producer-1| [thrd:main]: Failed to acquire transactional PID from broker TxnCoordinator/1: Broker: Coordinator not available: retrying
%4|1661869213.141|GETPID|rdkafka#producer-1| [thrd:main]: Failed to acquire transactional PID from broker TxnCoordinator/1: Broker: Coordinator not available: retrying
Traceback (most recent call last):
  File "/Users/clemens/docker/kafka/transaction.py", line 14, in <module>
producer.init_transactions(2.0)
cimpl.KafkaException: KafkaError{code=COORDINATOR_NOT_AVAILABLE,val=15,str="Failed to initialize Producer ID: Broker: Coordinator not available"}

临时解决方法 #

改用confluentinc/cp-server镜像替换confluentinc/cp-kafka，并参考官方cp-all-in-one的docker-compose.yml添加Schema Registry后，问题暂时解决，但仍希望明确该错误的根本原因。

内容的提问来源于stack exchange，提问作者clemens