Minikube中Logstash与Elasticsearch连接失败问题求助

在Minikube环境的Kubernetes中，Logstash与Elasticsearch连接失败问题已持续多日，以下是相关日志、配置及操作步骤，恳请协助排查解决：

Logstash Pod日志 #

[2023-11-20T10:16:01,269][WARN ][logstash.licensechecker.licensereader] 尝试恢复与已失效ES实例的连接，但出现错误 {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch无法访问: [http://elasticsearch:9200/][Manticore::ClientProtocolException] elasticsearch:9200无响应"}
[2023-11-20T10:16:01,303][ERROR][logstash.licensechecker.licensereader] 无法从许可证服务器获取许可证信息 {:message=>"无可用连接"}
[2023-11-20T10:16:05,280][WARN ][logstash.outputs.elasticsearch][main] 尝试恢复与已失效ES实例的连接，但出现错误 {:url=>"https://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"连接Elasticsearch URL 'https://elasticsearch:9200/'时收到响应码'401'"}

Elasticsearch日志 #

{"@timestamp":"2023-11-20T10:11:36.006Z", "log.level": "INFO", "message":"JVM参数 [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-11862069038782619862, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1902m, -Xmx1902m, -XX:MaxDirectMemorySize=997195776, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-57dc5fc6f7-42zzd","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2023-11-20T10:16:31.425Z", "log.level": "WARN", "message":"在HTTPS通道上收到明文HTTP流量，关闭连接 Netty4HttpChannel{localAddress=/10.244.0.15:9200, remoteAddress=/10.244.0.16:47910}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-57dc5fc6f7-42zzd][transport_worker][T#1]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"f_RDyR5xRwyVLO9IugQkZw","elasticsearch.node.id":"ZUGoJReVReixqtOnRe6LOg","elasticsearch.node.name":"elasticsearch-57dc5fc6f7-42zzd","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2023-11-20T10:16:36.319Z", "log.level": "INFO", "message":"[elastic]用户的认证被[reserved]域终止 - 无法认证用户[elastic]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-57dc5fc6f7-42zzd][system_critical_read][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","elasticsearch.cluster.uuid":"f_RDyR5xRwyVLO9IugQkZw","elasticsearch.node.id":"ZUGoJReVReixqtOnRe6LOg","elasticsearch.node.name":"elasticsearch-57dc5fc6f7-42zzd","elasticsearch.cluster.name":"docker-cluster"}

Kubernetes配置 #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: logging
spec:
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
        ports:
        - containerPort: 9200      

---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: logging
spec:
  ports:
    - port: 9200
      protocol: TCP
      targetPort: 9200
  selector:
    app: elasticsearch

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-config
  namespace: logging
data:
  logstash.conf: |
    input {
      beats {
        port => 5044
      }
    }

    output {
      elasticsearch {
        hosts => [ "https://elasticsearch:9200" ]
        ssl => true
        ssl_certificate_verification => false
        user => "elastic"
        password => "mypass"
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash
  namespace: logging
spec:
  selector:
    matchLabels:
      app: logstash
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
        - name: logstash
          image: docker.elastic.co/logstash/logstash:8.8.0
          ports:
            - containerPort: 5044
          volumeMounts:
            - name: pipeline-volume
              mountPath: /usr/share/logstash/pipeline/logstash.conf
              subPath: logstash.conf

      volumes:
        - name: pipeline-volume
          configMap:
            name: logstash-config
---
apiVersion: v1
kind: Service
metadata:
  name: logstash
  namespace: logging
spec:
  ports:
    - port: 5044
      protocol: TCP
      targetPort: 5044
  selector:
    app: logstash

Elastic用户密码重置步骤 #

oc exec -it elasticsearch-57dc5fc6f7-42zzd -- bash

elasticsearch@elasticsearch-57dc5fc6f7-42zzd:~$ bin/elasticsearch-reset-password -u elastic -a

WARNING: Owner of file [/usr/share/elasticsearch/config/users] used to be [root], but now is [elasticsearch]

WARNING: Owner of file [/usr/share/elasticsearch/config/users_roles] used to be [root], but now is [elasticsearch]

This tool will reset the password of the [elastic] user to an autogenerated value.

The password will be printed in the console.

Please confirm that you would like to continue [y/N] y

Password for the [elastic] user successfully reset.

New value: mypass

排查与修复建议 #

• 统一Elastic Stack版本：当前Elasticsearch为8.11.0，Logstash为8.8.0，版本差异可能引发兼容性问题，将Logstash镜像版本改为docker.elastic.co/logstash/logstash:8.11.0，保持全栈版本一致。
• 修正Logstash全链路连接协议：Logstash的许可证检查器默认使用HTTP连接ES，但ES 8.x默认启用HTTPS，导致协议不匹配。需在Logstash配置中添加监控配置，确保所有ES连接均使用HTTPS：

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => [ "https://elasticsearch:9200" ]
    ssl => true
    ssl_certificate_verification => false
    user => "elastic"
    password => "mypass"
  }
}

xpack.monitoring.elasticsearch.hosts: ["https://elasticsearch:9200"]
xpack.monitoring.elasticsearch.ssl: true
xpack.monitoring.elasticsearch.ssl.certificate_verification: false
xpack.monitoring.elasticsearch.username: "elastic"
xpack.monitoring.elasticsearch.password: "mypass"

• 配置Elasticsearch单节点模式：Minikube环境中ES以单节点运行，需添加环境变量确保集群发现正常，同时调整JVM参数适配Minikube资源：

containers:
- name: elasticsearch
  image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
  ports:
  - containerPort: 9200
  env:
  - name: discovery.type
    value: "single-node"
  - name: ES_JAVA_OPTS
    value: "-Xms512m -Xmx512m"

• 本地验证elastic用户认证：进入ES Pod，执行以下命令验证用户有效性：

# 列出所有用户
bin/elasticsearch-users list
# 本地测试认证
curl -u elastic:mypass https://localhost:9200 --insecure

若认证失败，需重新重置密码并确认密码与Logstash配置一致。

• 检查ES安全域配置：确认ES配置文件config/elasticsearch.yml中xpack.security.authc.realms.reserved.reserved.enabled为true（默认启用），确保reserved域可正常处理elastic用户认证。

内容的提问来源于stack exchange，提问作者Belbo