Minikube部署Debezium遇Kafka报错：Error creating broker listeners

我是Debezium新手，用Docker Compose按照官方教程能正常运行，但在Minikube（后续计划部署到Kubernetes）里搭建时遇到报错，推测是Kafka配置存在缺失，但无法确定具体问题，恳请提供帮助。

报错信息 #

WARNING: Using default NODE_ID=1, which is valid only for non-clustered installations.
Starting in ZooKeeper mode using NODE_ID=1.
Using ZOOKEEPER_CONNECT=zookeeper:2181
Using configuration config/server.properties.
Using KAFKA_LISTENERS=PLAINTEXT://10.244.0.45:tcp://10.109.182.14:9092 and KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://10.244.0.45:9092
2024-09-18 01:49:59,213 - INFO  [main:Log4jControllerRegistration$@31] - Registered kafka:type=kafka.Log4jController MBean
2024-09-18 01:49:59,698 - INFO  [main:X509Util@78] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2024-09-18 01:49:59,704 - ERROR [main:MarkerIgnoringBase@159] - Exiting Kafka due to fatal exception
java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT://10.244.0.45:tcp://10.109.182.14:9092': No security protocol defined for listener PLAINTEXT://10.244.0.45:TCP
    at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:243)
    at kafka.server.KafkaConfig.listeners(KafkaConfig.scala:2177)
    at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1845)
    at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1639)
    at kafka.Kafka$.buildServer(Kafka.scala:71)
    at kafka.Kafka$.main(Kafka.scala:90)
    at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.IllegalArgumentException: No security protocol defined for listener PLAINTEXT://10.244.0.45:TCP
    at kafka.cluster.EndPoint$.$anonfun$createEndPoint$2(EndPoint.scala:49)
    at scala.collection.immutable.Map$Map4.getOrElse(Map.scala:548)
    at kafka.cluster.EndPoint$.securityProtocol$1(EndPoint.scala:49)
    at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:57)
    at kafka.utils.CoreUtils$.$anonfun$listenerListToEndPoints$10(CoreUtils.scala:240)
    at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
    at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
    at scala.collection.mutable.ArraySeq.map(ArraySeq.scala:35)
    at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:240)
    ... 6 more

当前使用的Kubernetes配置 #

apiVersion: v1
kind: Service
metadata:
  name: zookeeper
spec:
  ports:
    - name: client
      port: 2181
      targetPort: 2181
    - name: follower
      port: 2888
      targetPort: 2888
    - name: election
      port: 3888
      targetPort: 3888
  selector:
    app: zookeeper

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zookeeper
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
        - name: zookeeper
          image: quay.io/debezium/zookeeper:2.7.2.Final
          ports:
            - containerPort: 2181
            - containerPort: 2888
            - containerPort: 3888
---
apiVersion: v1
kind: Service
metadata:
  name: kafka
spec:
  ports:
    - port: 9092
      targetPort: 9092
  selector:
    app: kafka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
        - name: kafka
          image: quay.io/debezium/kafka:2.7.2.Final
          ports:
            - containerPort: 9092
          env:
            - name: ZOOKEEPER_CONNECT
              value: zookeeper:2181
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  ports:
    - port: 5432
      targetPort: 5432
  selector:
    app: postgres
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: quay.io/debezium/example-postgres:2.7.2.Final
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_PASSWORD
              value: postgres
---
apiVersion: v1
kind: Service
metadata:
  name: connect
spec:
  ports:
    - port: 8083
      targetPort: 8083
  selector:
    app: connect
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: connect
spec:
  replicas: 1
  selector:
    matchLabels:
      app: connect
  template:
    metadata:
      labels:
        app: connect
    spec:
      containers:
        - name: connect
          image: quay.io/debezium/connect:2.7.2.Final
          ports:
            - containerPort: 8083
          env:
            - name: BOOTSTRAP_SERVERS
              value: kafka:9092
            - name: GROUP_ID
              value: "1"
            - name: CONFIG_STORAGE_TOPIC
              value: my_connect_configs
            - name: OFFSET_STORAGE_TOPIC
              value: my_connect_offsets
            - name: STATUS_STORAGE_TOPIC
              value: my_connect_statuses

从报错日志可以明确看到，Kafka的KAFKA_LISTENERS环境变量被错误拼接成了PLAINTEXT://10.244.0.45:tcp://10.109.182.14:9092，这是因为Debezium Kafka镜像默认的自动网络信息获取逻辑在Kubernetes环境中失效，导致监听地址格式混乱，Kafka无法解析正确的安全协议和端口参数。

解决方法是手动指定Kafka的监听配置，覆盖镜像的自动生成逻辑，修改Kafka Deployment的环境变量部分：

- name: KAFKA_LISTENERS
  value: PLAINTEXT://0.0.0.0:9092
- name: KAFKA_ADVERTISED_LISTENERS
  value: PLAINTEXT://kafka:9092
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
  value: PLAINTEXT:PLAINTEXT

修改后的完整Kafka Deployment配置如下：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
        - name: kafka
          image: quay.io/debezium/kafka:2.7.2.Final
          ports:
            - containerPort: 9092
          env:
            - name: ZOOKEEPER_CONNECT
              value: zookeeper:2181
            - name: KAFKA_LISTENERS
              value: PLAINTEXT://0.0.0.0:9092
            - name: KAFKA_ADVERTISED_LISTENERS
              value: PLAINTEXT://kafka:9092
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: PLAINTEXT:PLAINTEXT

配置说明 #

• KAFKA_LISTENERS：设置Kafka监听容器的所有网卡9092端口，确保容器内部和集群内服务能正常连接。
• KAFKA_ADVERTISED_LISTENERS：指定Kafka对外公告的地址为Kubernetes Service域名kafka:9092，集群内其他服务（如Connect）可通过该域名访问Kafka。
• KAFKA_LISTENER_SECURITY_PROTOCOL_MAP：明确绑定PLAINTEXT监听的安全协议为PLAINTEXT，避免Kafka解析地址时出现协议识别错误。

更新配置后重新部署Kafka，即可正常启动服务。

内容的提问来源于stack exchange，提问作者sammy