K8s ECK环境下Logstash Pod与Elasticsearch连接异常求助

问题场景 #

Elasticsearch（v2.9）与Logstash Pod运行在同一elastic-system命名空间中，但Logstash的所有HTTP请求均被Elasticsearch拒绝。

相关日志 #

Logstash Pod日志 #

[2023-11-03T10:57:39,092][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@quickstart-es-http:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://quickstart-es-http:9200/][Manticore::ClientProtocolException] quickstart-es-http:9200 failed to respond"}

Elasticsearch Pod日志 #

{"@timestamp":"2023-11-03T11:04:55.719Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.10.234.134:9200, remoteAddress=/10.10.234.158:60508}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[quickstart-es-default-0][transport_worker][T#1]","log.logger":"org.elasticsearch.http.netty4.Netty4HttpServerTransport","elasticsearch.cluster.uuid":"cLtrkFiZSXGk9Y7Vkh1dIA","elasticsearch.node.id":"ixrbei6lTcKEJhIxmhemJw","elasticsearch.node.name":"quickstart-es-default-0","elasticsearch.cluster.name":"quickstart"}

现有配置文件 #

Persistent Volume与Logstash YAML配置 #

apiVersion: v1
kind: PersistentVolume
metadata:
  name: logstash-data
  labels:
    type: local
spec:
  storageClassName: ls-scn
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/logstash/data"
---
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: quickstart
spec:
  count: 1
  elasticsearchRefs:
    - name: quickstart
      clusterName: qs
  version: 8.10.2
  podTemplate:
    spec:
      containers:
        - name: logstash
          env:
            - name: ECK_CACRT
              value: 81FCA623ED460EA0832CB35AD73D9A87B2E9B323ACFF07B985451E815CABF3D2
  pipelines:
    - pipeline.id: main
      config.string: |
        input {
          beats {
            port => 5044
          }
        }
        output {
          elasticsearch {
            hosts => [ "quickstart-es-http" ]
            user => "elastic"
            password => "4PyI3513SR21cSHs7lj6GyN9"
            ca_trusted_fingerprint => "${ECK_CACRT}"
          }
        }
  services:
    - name: beats
      service:
        spec:
          type: NodePort
          ports:
            - port: 5044
              name: "filebeat"
              protocol: TCP
              targetPort: 5044
              nodePort: 32000

  volumeClaimTemplates:
    - metadata:
        name: logstash-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: ls-scn

问题根源 #

从Elasticsearch日志可直接定位问题：Elasticsearch的9200端口仅接受HTTPS请求，但Logstash当前发送的是明文HTTP请求，导致连接被ES主动关闭。

修复方案 #

方案1：手动修改Logstash输出协议 #

在Logstash的output.elasticsearch配置中，将hosts明确指定为HTTPS地址：

output {
  elasticsearch {
    hosts => [ "https://quickstart-es-http:9200" ]
    user => "elastic"
    password => "4PyI3513SR21cSHs7lj6GyN9"
    ca_trusted_fingerprint => "${ECK_CACRT}"
  }
}

ECK部署的Elasticsearch默认开启TLS加密，9200端口为HTTPS端口，必须使用HTTPS协议访问。

方案2：利用ECK自动关联简化配置 #

由于你的Logstash已配置elasticsearchRefs，ECK会自动向Logstash Pod注入连接所需的环境变量，无需手动配置hosts、账号和证书信息。可将output部分简化为：

output {
  elasticsearch {
    cloud_id => "${ELASTIC_CLOUD_ID}"
    cloud_auth => "${ELASTIC_CLOUD_AUTH}"
    ssl_certificate_authorities => "${ELASTICSEARCH_SSL_CERTIFICATE_AUTHORITIES}"
  }
}

此方式更可靠，避免手动配置出错。

验证修复 #

更新Logstash资源后，查看Pod日志，确认连接警告消失，数据正常写入Elasticsearch即可。

内容的提问来源于stack exchange，提问作者siwm