在Docker Compose中配置Kafka Broker通过SSL及443端口访问

1. 生成Kafka SSL证书 #

生产环境建议使用CA签发证书，以下为自签证书示例：

• 创建证书目录kafka-ssl，为每个Broker生成密钥库（替换为对应Broker的DNS）：

keytool -genkey -alias broker1 -keyalg RSA -keystore kafka-ssl/broker1.keystore.jks -validity 3650 -storepass changeit -keypass changeit -dname "CN=broker1.example.com, OU=Dev, O=Company, L=City, ST=State, C=CN"

• 导出Broker证书：

keytool -export -alias broker1 -keystore kafka-ssl/broker1.keystore.jks -file kafka-ssl/broker1.crt -storepass changeit

• 创建共享信任库并导入所有Broker证书：

keytool -import -alias broker1 -file kafka-ssl/broker1.crt -keystore kafka-ssl/kafka.truststore.jks -storepass changeit -noprompt

重复上述步骤为每个Broker生成对应密钥库和证书，统一导入信任库。

2. 调整Docker Compose Kafka Broker配置 #

修改每个Broker的环境变量，启用SSL监听并配置对外公布地址：

services:
  broker1:
    image: confluentinc/cp-kafka:latest
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zoo1:2181
      # 内部集群通信用PLAINTEXT，外部SSL监听端口9093（仅Docker内部访问）
      KAFKA_LISTENERS: "PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093"
      # 对外公布Nginx代理的DNS+443端口，协议为SSL
      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://broker1:9092,SSL://broker1.example.com:443"
      # SSL密钥库/信任库配置
      KAFKA_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/broker1.keystore.jks
      KAFKA_SSL_KEYSTORE_PASSWORD: changeit
      KAFKA_SSL_KEY_PASSWORD: changeit
      KAFKA_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/kafka.truststore.jks
      KAFKA_SSL_TRUSTSTORE_PASSWORD: changeit
      KAFKA_SSL_CLIENT_AUTH: required
      KAFKA_DEFAULT_REPLICATION_FACTOR: 3
      KAFKA_MIN_INSYNC_REPLICAS: 2
    volumes:
      - ./kafka-ssl:/etc/kafka/secrets
    # 仅暴露内部端口给集群，不对外映射9092-9097
    expose:
      - "9092"
      - "9093"

  # 复制上述配置修改broker2、broker3，对应调整BROKER_ID、密钥库文件名及ADVERTISED_LISTENERS的DNS

  zoo1:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

3. 配置Nginx反向代理 #

为每个Broker的DNS配置Nginx，将443端口的SSL流量转发到对应Broker的9093端口：

server {
    listen 443 ssl;
    server_name broker1.example.com;

    # Nginx使用的SSL证书（公网场景建议用CA签发证书）
    ssl_certificate /path/to/broker1.crt;
    ssl_certificate_key /path/to/broker1.key;

    location / {
        # 转发到Docker内部Broker的SSL端口
        proxy_pass https://broker1:9093;
        # 启用TCP代理适配Kafka协议
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }
}

# 重复配置broker2.example.com、broker3.example.com，对应转发到broker2:9093、broker3:9093

确保Nginx与Kafka Broker处于同一Docker网络，可通过容器名直接访问Broker。

4. 客户端连接配置 #

客户端需配置SSL信任库，以Java客户端为例：

bootstrap.servers=broker1.example.com:443,broker2.example.com:443,broker3.example.com:443
security.protocol=SSL
ssl.truststore.location=/path/to/kafka.truststore.jks
ssl.truststore.password=changeit

常见问题排查 #

• 出现no broker available错误：
  • 验证Nginx与Broker的连通性：在Docker网络内执行curl -v https://broker1:9093
  • 检查KAFKA_ADVERTISED_LISTENERS是否与Nginx代理地址完全匹配
  • 确认Broker密钥库的CN字段与ADVERTISED_LISTENERS中的DNS一致
  • 确保Nginx配置为TCP代理模式，而非普通HTTP代理

内容的提问来源于stack exchange，提问作者Vishal Reddy