Kibana无法获取Elasticsearch版本信息报错排查求助

我使用Elastic官方Helm Chart在Kubernetes中部署Elasticsearch和Kibana，Kibana容器出现以下错误，导致Ingress返回503错误且容器始终无法就绪：

[2022-11-08T12:30:53.321+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. socket hang up - Local: 10.112.130.148:42748, Remote: 10.96.237.95:9200

IP地址10.96.237.95是有效的Elasticsearch服务地址，端口也正确。从Kibana容器内部curl Elasticsearch可成功获得响应。

Chart版本：7.17.3

Elasticsearch Chart配置值 #

clusterName: "elasticsearch"
nodeGroup: "master"

createCert: false

roles:
  master: "true"
  data: "true"
  ingest: "true"
  ml: "true"
  transform: "true"
  remote_cluster_client: "true"

protocol: https

replicas: 2

sysctlVmMaxMapCount: 262144

readinessProbe:
   failureThreshold: 3
   initialDelaySeconds: 90
   periodSeconds: 10
   successThreshold: 1
   timeoutSeconds: 10

imageTag: "7.17.3"

extraEnvs:
- name: ELASTIC_PASSWORD
  valueFrom:
    secretKeyRef:
      name: elasticsearch-creds
      key: password
- name: ELASTIC_USERNAME
  valueFrom:
    secretKeyRef:
      name: elasticsearch-creds
      key: username

clusterHealthCheckParams: "wait_for_status=green&timeout=20s"

antiAffinity: "soft"

resources:
  requests:
    cpu: "100m"
    memory: "1Gi"
  limits:
    cpu: "1000m"
    memory: "1Gi"

esJavaOpts: "-Xms512m -Xmx512m"

volumeClaimTemplate:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 30Gi

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.client_authentication: required
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12


secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

Kibana Chart配置值 #

elasticSearchHosts: "https://elasticsearch-master:9200"

extraEnvs:
  - name: ELASTICSEARCH_USERNAME
    valueFrom:
      secretKeyRef:
        name: elasticsearch-creds
        key: username
  - name: ELASTICSEARCH_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elasticsearch-creds
        key: password
  - name: KIBANA_ENCRYPTION_KEY
    valueFrom:
      secretKeyRef:
        name: encryption-key  
        key: encryption_key

kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: true
      key: /usr/share/kibana/config/certs/elastic-certificate.pem
      certificate: /usr/share/kibana/config/certs/elastic-certificate.pem
    xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate
    
protocol: https

secretMounts:
  - name: elastic-certificate-pem
    secretName: elastic-certificate-pem
    path: /usr/share/kibana/config/certs

imageTag: "7.17.3"

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-issuer
    kubernetes.io/ingress.allow-http: 'false'
  paths:
    - path: /
      pathType: Prefix
      backend:
        serviceName: kibana
        servicePort: 5601
  hosts:
    - host: mydomain.com
      paths:
        - path: /
          pathType: Prefix
          backend:
            serviceName: kibana
            servicePort: 5601
  tls:
    - hosts:
        - mydomain.com
      secretName: mydomain.com

更新信息 #

尝试更换镜像版本为8.4.1，问题依旧。另外Logstash可成功向该Elasticsearch实例推送日志，因此怀疑问题出在Kibana。

针对Kibana无法连接Elasticsearch但curl正常的问题，可从以下方向排查修复：

1. 修复证书权限与匹配性 #

• 确保挂载到Kibana的PEM证书是从Elasticsearch使用的elastic-certificates.p12中完整导出的，包含证书链
• 检查Kibana容器内证书文件权限，默认Kibana进程以UID 1000运行，需保证证书可读取：

  kubectl exec -it <kibana-pod-name> -- ls -l /usr/share/kibana/config/certs/
  

  权限不足时，在Kibana的secretMounts中添加权限配置：

  secretMounts:
    - name: elastic-certificate-pem
      secretName: elastic-certificate-pem
      path: /usr/share/kibana/config/certs
      defaultMode: 0644
  

2. 调整Kibana就绪探针 #

默认探针可能因超时或初始化时间不足误判容器状态，自定义探针配置：

readinessProbe:
  httpGet:
    path: /api/status
    port: 5601
    scheme: HTTPS
  initialDelaySeconds: 120
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 6

3. 补全Kibana的Elasticsearch SSL配置 #

若Elasticsearch HTTP层启用了客户端认证，Kibana需配置客户端证书，在kibana.yml中添加：

elasticsearch.ssl.key: /usr/share/kibana/config/certs/elastic-certificate.pem
elasticsearch.ssl.certificate: /usr/share/kibana/config/certs/elastic-certificate.pem

4. 验证Elasticsearch HTTP SSL要求 #

通过curl测试Elasticsearch是否需要客户端证书：

# 不带证书请求
kubectl exec -it <kibana-pod-name> -- curl -k https://elasticsearch-master:9200
# 带证书请求
kubectl exec -it <kibana-pod-name> -- curl --cert /usr/share/kibana/config/certs/elastic-certificate.pem --key /usr/share/kibana/config/certs/elastic-certificate.pem -k https://elasticsearch-master:9200

若不带证书无法访问，说明Elasticsearch要求客户端认证，必须补全Kibana的SSL证书配置。

5. 排查环境变量冲突 #

Elastic Helm Chart中环境变量会覆盖kibana.yml配置，检查是否存在ELASTICSEARCH_SSL_VERIFICATIONMODE等变量干扰连接逻辑。

内容的提问来源于stack exchange，提问作者A735