K8s环境下Logstash连接Elasticsearch失败的部署问题求助

在Kubernetes集群中已成功部署Elasticsearch和Kibana，尝试通过Logstash+Filebeat收集Pod日志并传入Elasticsearch，但Logstash Pod部署后崩溃。

相关配置文件 #

Elasticsearch 配置（elasticsearch.yaml） #

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 7.14.1 # 使用你需要的版本
  http:
    service:
      spec:
        type: LoadBalancer # 配置外部IP
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false

Logstash 配置（logstash.yaml） #

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-configmap
data:
  logstash.yml: |
    http.host: "0.0.0.0"
    path.config: /usr/share/logstash/pipeline
  logstash.conf: |
    input {
      beats {
        port => 5044
      }
    }
    output {
      elasticsearch {
        hosts => "https://External ip:9200"
        index =>"testlogs"
        user => "elastic"
        password => "xxxx"
        cacert => "/etc/"
        ssl_certificate_verification => false
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logstash
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
      - name: logstash
        image: docker.elastic.co/logstash/logstash:7.14.1
        ports:
        - containerPort: 5044
        volumeMounts:
        - name: config-volume
          mountPath: /usr/share/logstash/config
        - name: logstash-pipeline-volume
          mountPath: /usr/share/logstash/pipeline
      volumes:
      - name: config-volume
        configMap:
          name: logstash-configmap
          items:
          - key: logstash.yml
            path: logstash.yml
      - name: logstash-pipeline-volume
        configMap:
          name: logstash-configmap
          items:
          - key: logstash.conf
            path: logstash.conf
---
kind: Service
apiVersion: v1
metadata:
  name: logstash-service
spec:
  selector:
    app: logstash
  ports:
  - protocol: TCP
    port: 5044
    targetPort: 5044

报错信息 #

[ERROR] 2022-10-21 08:54:16.937 [[main]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"main", :exception=>#<Errno::EISDIR: Is a directory - channel: org.jruby.util.io.ChannelFD@2f88e4ba /etc/>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.1-java/lib/manticore/client.rb:642:in block in setup_trust_store'", "org/jruby/RubyIO.java:1158:in open'", "org/jruby/RubyKernel.java:317:in open'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/open-uri.rb:37:in open'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.1-java/lib/manticore/client.rb:641:in setup_trust_store'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.1-java/lib/manticore/client.rb:629:in ssl_socket_factory_from_options'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.1-java/lib/manticore/client.rb:396:in pool_builder'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.1-java/lib/manticore/client.rb:404:in pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.1-java/lib/manticore/client.rb:207:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:26:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:319:in build_adapter'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:323:in build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:62:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:34:in build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/outputs/elasticsearch.rb:275:in register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in block in register_plugins'", "org/jruby/RubyArray.java:1820:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:585:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x5b2f9d8f@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[INFO ] 2022-10-21 08:54:16.940 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[ERROR] 2022-10-21 08:54:16.957 [Converge PipelineAction::Create] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}

疑问 #

怀疑是CA证书路径配置错误，想请教正确的证书路径配置方式，以及在K8s集群中哪里可以找到该CA证书？

1. 错误根源 #

从报错信息Errno::EISDIR: Is a directory - channel: org.jruby.util.io.ChannelFD@2f88e4ba /etc/可以明确：你把cacert配置成了目录/etc/，但该参数需要指向具体的证书文件路径，而非目录。

另外你已设置ssl_certificate_verification => false，理论上可跳过证书验证，但Logstash在配置了cacert时仍会尝试读取该路径，因此要么删除cacert配置，要么配置正确的证书文件路径。

2. 获取Elasticsearch的CA证书 #

使用Elastic Operator部署Elasticsearch时，CA证书会自动生成并存储在Secret中，名称格式为<elasticsearch-name>-es-ca，你的Elasticsearch实例名为elasticsearch，所以Secret名称是elasticsearch-es-ca。

查看并导出CA证书： #

# 查看Secret内容
kubectl get secret elasticsearch-es-ca -o yaml

Secret中的tls.crt字段是CA证书的base64编码内容，可解码后保存为文件，或直接通过Volume挂载到Logstash Pod中。

3. 修正Logstash配置 #

方式一：跳过证书验证（适合测试环境） #

直接删除cacert => "/etc/"这一行，因已设置ssl_certificate_verification => false，无需指定CA证书：

output {
  elasticsearch {
    hosts => "https://External ip:9200"
    index =>"testlogs"
    user => "elastic"
    password => "xxxx"
    ssl_certificate_verification => false
  }
}

方式二：挂载CA证书并配置正确路径（生产环境推荐） #

步骤1：修改Logstash Deployment，添加CA证书的Volume挂载

在Deployment的volumes部分添加：

- name: es-ca-cert
  secret:
    secretName: elasticsearch-es-ca
    items:
    - key: tls.crt
      path: ca.crt

在容器的volumeMounts部分添加：

- name: es-ca-cert
  mountPath: /usr/share/logstash/certs
  readOnly: true

步骤2：修改Logstash配置中的cacert路径

将cacert指向挂载的证书文件路径：

output {
  elasticsearch {
    hosts => "https://External ip:9200"
    index =>"testlogs"
    user => "elastic"
    password => "xxxx"
    cacert => "/usr/share/logstash/certs/ca.crt"
    # 生产环境建议开启证书验证
    ssl_certificate_verification => true
  }
}

4. 重新部署Logstash #

更新ConfigMap和Deployment：

kubectl apply -f logstash.yaml

内容的提问来源于stack exchange，提问作者psp