Apache Kafka远程生产者无法写入主题问题求助

问题现象 #

在与Broker、ZooKeeper服务器分离的设备上，执行Kafka控制台生产者命令尝试写入集群主题时，反复出现连接失败警告，无认证相关日志输出。

执行命令：

bin/kafka-console-producer.sh --broker-list raspberrypi:9092 --topic my-topic --producer.config config/producer.properties

报错信息：

[2023-04-19 12:55:38,819] WARN [Producer clientId=console-producer] Connection to node -1 (raspberrypi/192.168.1.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-04-19 12:55:38,819] WARN [Producer clientId=console-producer] Bootstrap broker raspberrypi:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

关键信息：在Broker所在机器上执行相同命令可正常运行；已开放防火墙9092、9093端口；Broker和ZooKeeper日志无相关报错记录。

配置信息 #

Broker端server.properties #

broker.id=0
confluent.http.server.listeners=

listeners=SASL_SSL://raspberrypi:9092,SASL_SSL1://192.168.1.104:9093
advertised.listeners=SASL_SSL://raspberrypi:9092,SASL_SSL1://192.168.1.104:9093

listener.security.protocol.map=SASL_SSL:SASL_SSL,SASL_SSL1:SASL_SSL

zookeeper.connect=raspberrypi:2182

log.dirs=/tmp/data/broker-0
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
num.partitions=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0

# Properties for SSL Zookeeper Security between Zookeeper and Broker
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.ssl.client.enable=true
zookeeper.ssl.protocol=TLSv1.2

zookeeper.ssl.truststore.location=/home/pi/kafka_2.13-3.4.0/ssl/kafka.broker.truststore.jks
zookeeper.ssl.truststore.password=exjobb123
zookeeper.ssl.keystore.location=/home/pi/kafka_2.13-3.4.0/ssl/kafka.broker.keystore.jks
zookeeper.ssl.keystore.password=exjobb123

zookeeper.set.acl=true

# Properties for SSL Kafka Security between Broker and its clients
ssl.truststore.location=/home/pi/kafka_2.13-3.4.0/ssl/kafka.broker.truststore.jks
ssl.truststore.password=exjobb123
ssl.keystore.location=/home/pi/kafka_2.13-3.4.0/ssl/kafka.broker.keystore.jks
ssl.keystore.password=exjobb123
ssl.key.password=exjobb123
security.inter.broker.protocol=SASL_SSL
ssl.client.auth=required
ssl.protocol=TLSv1.2

# Properties for SASL between a broker and its client
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
listener.name.sasl_ssl.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="kafka-admin" password="exjobb123";
listener.name.sasl_ssl1.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="kafka-admin" password="exjobb123";
super.users=User:kafka-admin

# Properties for Authorization
authorizer.class.name=kafka.security.authorizer.AclAuthorizer

生产者端producer.properties #

bootstrap.servers=raspberrypi:9092
compression.type=none
security.protocol=SASL_SSL
ssl.protocol=TLSv1.2
ssl.truststore.location=/home/exjobb/ssl/kafka.producer.truststore.jks
ssl.truststore.password=exjobb123
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="sasl-producer" password="exjobb123";

排查与修复建议 #

1. DNS解析验证

   • 在生产者设备执行ping raspberrypi，确认能解析到192.168.1.104；若解析失败，修改/etc/hosts添加映射：192.168.1.104 raspberrypi

2. SSL证书信任检查

   • 验证生产者信任库是否包含Broker证书的CA根证书，执行命令：

     keytool -list -v -keystore /home/exjobb/ssl/kafka.producer.truststore.jks
     

   • 若缺失CA证书，重新导入：

     keytool -importcert -alias broker-ca -file /path/to/ca-cert.pem -keystore /home/exjobb/ssl/kafka.producer.truststore.jks
     

3. SASL权限与凭证验证

   • 确认sasl-producer用户已创建SCRAM凭证：

     bin/kafka-configs.sh --zookeeper raspberrypi:2182 --alter --add-config 'SCRAM-SHA-512=[password=exjobb123]' --entity-type users --entity-name sasl-producer
     

   • 授予用户主题写入权限：

     bin/kafka-acls.sh --authorizer-properties zookeeper.connect=raspberrypi:2182 --add --allow-principal User:sasl-producer --operation Write --topic my-topic
     

4. Listener绑定与端口监听检查

   • 确认Broker实际监听9092端口：

     netstat -tulpn | grep 9092
     

   • 若Broker仅绑定特定网卡，修改listeners配置为SASL_SSL://0.0.0.0:9092,SASL_SSL1://192.168.1.104:9093，确保监听所有网卡。

5. 开启详细日志定位问题

   • 生产者端启动时添加Debug日志参数：

     bin/kafka-console-producer.sh --broker-list raspberrypi:9092 --topic my-topic --producer.config config/producer.properties --producer-property log4j.logger.org.apache.kafka.clients.NetworkClient=DEBUG --producer-property log4j.logger.org.apache.kafka.common.security=DEBUG
     

   • Broker端修改log4j.properties添加：

     log4j.logger.org.apache.kafka.common.network=DEBUG
     log4j.logger.org.apache.kafka.common.security=DEBUG
     

     重启Broker后查看日志，获取具体连接失败原因。

内容的提问来源于stack exchange，提问作者snigln