无法向Docker容器中的wurstmeister/kafka发送消息问题排查

问题现象 #

• 本地生产者向localhost:9093（误用PLAINTEXT协议）发送消息超时，Kafka容器无日志反应
• 向localhost:9092（误用SSL协议）发送消息时，Kafka容器出现握手错误
• ZooKeeper与Kafka均通过Docker Compose部署在容器中

相关配置 #

Docker Compose 配置 #

version: '3'

services:
    zookeeper:
        image: local_zookeeper
        build:
            context: .
            dockerfile: Dockerfile_zookeeper
        container_name: zookeeper
        volumes: 
            - zookeeperVolume:/etc/zookeeper/secrets
            - zookeeperVolume:/var/lib/zookeeper
            - zookeeperVolume:/opt/zookeeper-3.4.13
            - zookeeperVolume:/var/lib/zookeeper/data
            - zookeeperVolume:/var/lib/zookeeper/log
        ports:
            - "2181:2181"
            
    kafka:
        image: local_kafka
        build:
            context: .
            dockerfile: Dockerfile_kafka
        container_name: kafka    
        volumes: 
            - kafkaVolume:/etc/kafka/secrets
            - kafkaVolume:/kafka

        depends_on:
        - zookeeper
        ports:
        - "9092:29092"
        - "9093:29093"
          
volumes:
    kafkaVolume:
    zookeeperVolume:

ZooKeeper Dockerfile #

FROM confluentinc/cp-zookeeper:latest

ENV ZOOKEEPER_CLIENT_PORT 2181
ENV ZOOKEEPER_TICK_TIME 2000

Kafka Dockerfile #

FROM wurstmeister/kafka

ENV KAFKA_BROKER_ID 1
ENV KAFKA_ADVERTISED_LISTENERS INTERNAL_PLAIN://kafka:9092,EXTERNAL_PLAIN://localhost:29092, INTERNAL_SSL://kafka:9093,EXTERNAL_SSL://localhost:29093
ENV KAFKA_LISTENERS EXTERNAL_PLAIN://:29092,INTERNAL_PLAIN://:9092,EXTERNAL_SSL://:29093,INTERNAL_SSL://:9093
ENV KAFKA_LISTENER_SECURITY_PROTOCOL_MAP INTERNAL_SSL:SSL,EXTERNAL_SSL:SSL,INTERNAL_PLAIN:PLAINTEXT,EXTERNAL_PLAIN:PLAINTEXT
ENV KAFKA_INTER_BROKER_LISTENER_NAME INTERNAL_PLAIN
ENV KAFKA_ZOOKEEPER_CONNECT zookeeper:2181
ENV KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR 1
ENV KAFKA_SOCKET_REQUEST_MAX_BYTES 1195725856
ENV KAFKA_HEAP_OPTS "-Xmx12g -Xms1g"
ENV KAFKA_AUTO_CREATE_TOPICS_ENABLE true
ENV KAFKA_SSL_CLIENT_AUTH 'required'
ENV KAFKA_SSL_KEYSTORE_LOCATION etc/kafka/secrets/certs/keystore.jks
ENV KAFKA_SSL_KEYSTORE_PASSWORD pass
ENV KAFKA_SSL_KEY_PASSWORD pass
ENV KAFKA_SSL_TRUSTSTORE_LOCATION etc/kafka/secrets/certs/truststore.jks
ENV KAFKA_SSL_TRUSTSTORE_PASSWORD pass
ENV KAFKA_SSL_ENABLED_PROTOCOLS TLSv1.2
ENV KAFKA_CREATE_TOPICS DUMMY-TOPIC:1:1
ENV KAFKA_REQUEST_TIMEOUT_MS 5000

COPY ["kafka/keystore.jks", "etc/kafka/secrets/certs/keystore.jks"]
COPY ["kafka/truststore.jks", "etc/kafka/secrets/certs/truststore.jks"]

已配置的端口映射 #

• ZooKeeper: 2181:2181
• Kafka: 9092:29092、9093:29093

解决方案 #

1. 修正协议与端口的匹配关系 #

当前端口映射和监听器的协议对应完全搞反：

• 本地localhost:9092映射容器内29092，对应EXTERNAL_PLAIN（PLAINTEXT协议），需用PLAINTEXT方式连接
• 本地localhost:9093映射容器内29093，对应EXTERNAL_SSL（SSL协议），需用SSL方式连接

2. 修复Kafka监听器配置语法错误 #

KAFKA_ADVERTISED_LISTENERS中INTERNAL_SSL://kafka:9093前多了一个空格，会导致监听器解析异常，修正为：

ENV KAFKA_ADVERTISED_LISTENERS INTERNAL_PLAIN://kafka:9092,EXTERNAL_PLAIN://localhost:29092,INTERNAL_SSL://kafka:9093,EXTERNAL_SSL://localhost:29093

3. 验证SSL证书有效性 #

• 进入Kafka容器执行ls etc/kafka/secrets/certs/，确认keystore.jks和truststore.jks存在
• 由于配置了KAFKA_SSL_CLIENT_AUTH 'required'，本地生产者必须携带与Kafka truststore匹配的客户端证书；若无需强制客户端认证，可改为'none'或'requested'
• 检查本地生产者使用的SSL协议版本是否为TLSv1.2，与Kafka配置一致

4. 排查网络连通性 #

• 查看ZooKeeper日志docker logs zookeeper，确认服务正常启动
• 进入Kafka容器执行nc -zv zookeeper 2181，验证与ZooKeeper的连通性
• 本地执行telnet localhost 9092和telnet localhost 9093，确认端口处于监听状态

5. 开启详细日志定位SSL问题 #

在Kafka Dockerfile中添加日志配置，获取握手错误细节：

ENV KAFKA_LOG4J_LOGGERS "kafka.network.Processor=DEBUG,org.apache.kafka.common.network.SslTransportLayer=DEBUG"

重启容器后查看docker logs kafka，根据日志信息排查证书不匹配、协议不兼容等问题

内容的提问来源于stack exchange，提问作者Luke