Docker环境中Kafka Broker反复重启问题排查求助

问题现象 #

• 开发环境中基于confluentinc/cp-server:7.0.1镜像的Kafka Broker，日常运行正常，但每隔几天会陷入每10分钟一次的无限重启循环，期间所有服务均无法连接
• 唯一临时解决方法：执行docker compose down && docker compose up

重启前的关键日志 #

连接警告 #

WARN [Producer clientId=_confluent_balancer_api_state-producer-1]
Connection to node 1 (broker/172.18.0.5:29092) could not be established.
Broker may not be available. (org.apache.kafka.clients.NetworkClient)
WARN [Producer clientId=ConfluentBalancerSampleStoreProducer]
Connection to node 1 (broker/172.18.0.5:29092) could not be established.
Broker may not be available. (org.apache.kafka.clients.NetworkClient)

Zookeeper连接错误 #

EndOfStreamException: Unable to read additional data from server sessionid 0x1000003dd170035, likely server has closed socket
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)

主题相关错误 #

ERROR Skip updating configuration of topic _confluent_balancer_partition_samples due to exception. (com.linkedin.kafka.cruisecontrol.SbkTopicUtils)
java.lang.RuntimeException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
The topic '_confluent_balancer_partition_samples' does not exist.

Docker Compose配置（Broker部分） #

broker:
    image: confluentinc/cp-server:7.0.1
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - 127.0.0.1:9092:9092
      - 127.0.0.1:9101:9101
    restart: always
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://127.0.0.1:9092
      # KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      KAFKA_LOG4J_LOGGERS: org.apache.zookeeper=WARN,org.apache.kafka=WARN,kafka=WARN,kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN
      KAFKA_LOG4J_ROOT_LOGLEVEL: WARN
      KAFKA_TOOLS_LOG4J_LOGLEVEL: WARN
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'false'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'
      CONFLUENT_REPORTERS_TELEMETRY_AUTO_ENABLE: 'false'

已尝试的排查 #

• 修改KAFKA_ADVERTISED_LISTENERS和KAFKA_LISTENERS配置，无效果

求助问题 #

本人对Kafka/Docker较为陌生，不清楚大部分配置含义，请问还有哪些排查方向？

内容的提问来源于stack exchange，提问作者Leo Jiang