Banzai Logging Operator v4.2.2连接Elasticsearch v8.9.0失败求助

问题背景 #

在Kubernetes v1.25集群中，通过Banzai Logging Operator v4.2.2部署Fluent Bit与Fluentd后，无法连接Elasticsearch v8.9.0，出现以下错误：

The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the server is running an unsupported product.

2023-08-16 11:30:37 +0000 [error]: fluent/log.rb:372:error: unexpected error error_class=Elastic::Transport::Transport::Error error="EOFError (EOFError)"
2023-08-16T11:30:37.803235525Z   2023-08-16 11:30:37 +0000 [error]: fluent/supervisor.rb:1055:main_process: /usr/lib/ruby/gems/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:324:in `rescue in perform_request' 
2023-08-16T11:30:37.803240778Z   2023-08-16 11:30:37 +0000 [error]: fluent/supervisor.rb:1055:main_process: /usr/lib/ruby/gems/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:285:in `perform_request'
2023-08-16T11:30:37.803244112Z   2023-08-16 11:30:37 +0000 [error]: fluent/supervisor.rb:1055:main_process: /usr/lib/ruby/gems/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/http/faraday.rb:36:in `perform_request' 
2023-08-16T11:30:37.803670295Z   2023-08-16 11:30:37 +0000 [error]: fluent/supervisor.rb:1055:main_process: /usr/lib/ruby/gems/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/client.rb:176:in `perform_request'
2023-08-16T11:30:37.803685465Z   2023-08-16 11:30:37 +0000 [error]: fluent/supervisor.rb:1055:main_process: /usr/lib/ruby/gems/3.1.0/gems/elasticsearch-8.6.0/lib/elasticsearch.rb:71:in `method_missing' 
2023-08-16T11:30:37.803689566Z   2023-08-16 11:30:37 +0000 [error]: fluent/supervisor.rb:1055:main_process: /usr/lib/ruby/gems/3.1.0/gems/elasticsearch-api-8.6.0/lib/elasticsearch/api/actions/info.rb:41:in `info'
2023-08-16 11:30:37 +0000 [error]: fluent/supervisor.rb:1055:main_process: /usr/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:498:in `detect_es_major_version' 
2023-08-16T11:30:37.803696370Z   2023-08-16 11:30:37 +0000 [error]: fluent/supervisor.rb:1055:main_process: /usr/lib/ruby/gems/3.1.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:489:in `block in handle_last_seen_es_major_version'

原因分析 #

1. 插件版本不兼容：当前使用的fluent-plugin-elasticsearch v5.3.0对Elasticsearch 8.x的支持不完善，版本检测逻辑无法识别ES 8.x的响应格式，触发验证错误。
2. TLS配置问题：ES 8.x默认强制启用HTTPS，若Fluentd未正确配置CA证书或TLS参数，会导致TLS握手失败，出现EOFError。
3. 认证方式不匹配：ES 8.x默认禁用用户名密码认证（需手动开启），推荐使用API密钥，旧版插件对新认证方式支持不足。

解决方案 #

1. 升级fluent-plugin-elasticsearch插件 #

将插件升级至v5.12.0及以上版本（推荐v5.16.0稳定版），该版本完全支持ES 8.x。通过Banzai Logging Operator配置升级：

apiVersion: logging.banzaicloud.io/v1beta1
kind: Fluentd
spec:
  plugins:
    - name: fluent-plugin-elasticsearch
      version: 5.16.0

2. 正确配置TLS与认证 #

根据ES 8.x的认证方式，配置Fluentd输出：

• 用户名密码认证（需先在ES中开启）：

apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
  name: elasticsearch-output
spec:
  elasticsearch:
    host: elasticsearch-master
    port: 9200
    scheme: https
    ssl_verify: true
    ssl_ca_path: /etc/fluentd/certs/ca.crt
    user: "${ELASTIC_USER}"
    password: "${ELASTIC_PASSWORD}"
    es_version: 8
    buffer:
      tags: "kubernetes"
      timekey: 1m
      timekey_wait: 30s
      timekey_use_utc: true

• API密钥认证（ES 8.x推荐）：

apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
  name: elasticsearch-output
spec:
  elasticsearch:
    host: elasticsearch-master
    port: 9200
    scheme: https
    ssl_verify: true
    ssl_ca_path: /etc/fluentd/certs/ca.crt
    api_key: "${ELASTIC_API_KEY}"
    es_version: 8
    buffer:
      tags: "kubernetes"
      timekey: 1m
      timekey_wait: 30s
      timekey_use_utc: true

注意：需将ES的CA证书通过Kubernetes Secret挂载到Fluentd Pod的/etc/fluentd/certs/目录。

3. 强制指定ES版本（临时 workaround） #

若无法立即升级插件，可强制指定ES版本跳过自动检测，避免版本检测时的错误：

# 在Output配置中添加
es_version: 8

4. 验证网络与TLS连通性 #

在Fluentd Pod中执行命令测试ES连接：

curl -v https://elasticsearch-master:9200 -u elastic:your_password --cacert /etc/fluentd/certs/ca.crt

若返回ES节点信息，则网络与TLS配置正常；若报错，需检查NetworkPolicy、防火墙或证书配置。

内容的提问来源于stack exchange，提问作者AniketGole