Filebeat Kafka客户端与AWS MSK SSL握手失败排查求助

背景 #

我们尝试使用Filebeat（镜像版本：docker.elastic.co/beats/filebeat:8.5.1）向AWS MSK（预配置版）发送日志，采用基于Root CA和Intermediate CA的mTLS认证方案。相关CA由Vault管理，其中中间CA部署在AWS PCA并分配给MSK集群，用于为MSK Broker颁发证书。

我们使用常规Kafka客户端（携带必要证书）可成功完成mTLS认证，但Filebeat的Kafka输出始终无法完成SSL握手，且所有用于握手的证书均有效。

Filebeat配置（filebeat.yaml） #

---
filebeat.shutdown_timeout: 0
fields_under_root: false
logging.level: debug
.
.
.

output.kafka:
  hosts: 'XXXXMSK_BOOTSTRAP_HOSTSXXXX'
  ssl.enabled: true
  ssl.verification_mode: 'certificate'
  ssl.certificate: '/path/to/obained-cert.crt'
  ssl.key: '/path/to/obained-key.pki.key'
  ssl.authorities: ['/path/to/root/int/ca/combined-file/msk_ca_chain.pem']
  topic: 'XXXXKAFKA_TOPICXXXX'
  codec.format:
    string: '{"timestamp": "%{[@timestamp]}", "message": %{[message]}, "host": %{[host]}}'
  close_inactive: 10m
  required_acks: 1
  partition.round_robin:
    reachable_only: false
  keep-alive: 30000ms

证书文件说明 #

• obtained-cert.crt：客户端证书
• obtained-key.pki.key：客户端私钥
• msk_ca_chain.pem：包含中间CA与Root CA的证书链

错误日志 #

Filebeat日志 #

{"log.level":"error","@timestamp":"2023-01-06T10:59:48.701Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":337},"message":"Kafka (topic=XXXXKAFKA_TOPICXXXX): kafka: client has run out of available brokers to talk to (Is your cluster reachable?)","service.name":"filebeat","ecs.version":"1.6.0"}

AWS CloudWatch Broker日志 #

[2023-01-06 12:48:07,716] INFO [SocketServer listenerType=ZK_BROKER, nodeId=3] Failed authentication with /INTERNAL_IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2023-01-06 12:48:08,004] INFO [SocketServer listenerType=ZK_BROKER, nodeId=2] Failed authentication with /INTERNAL_IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2023-01-06 12:48:08,016] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)

我们已开启Filebeat调试日志，但未获取到SSL握手失败的具体原因，现寻求以下帮助：

1. 如何在Filebeat Kafka客户端或AWS MSK Broker侧获取更详细的调试日志以定位失败原因？
2. Filebeat配置可能存在的问题排查方向？

内容的提问来源于stack exchange，提问作者kk.