ELK从Docker Compose迁移至K8s后Kibana服务就绪失败排查

现象 #

• 原Docker Compose部署的ELK集群运行正常，迁移至Kubernetes后所有Pod状态正常，但浏览器访问Kibana时显示**"Kibana server is not ready"**
• 请求http://20.4.187.154:5601/internal/interactive_setup/status返回503 (Service Unavailable)
• Chrome控制台存在CSP错误（已确认与当前问题无关）

排查信息 #

在Kibana Pod内，使用elastic和kibana_system用户通过curl可正常访问Elasticsearch集群：

kibana@tone-kibana:~/logs$ curl -u elastic:somepassword -XGET tone-elasticsearch:9200/_cluster/health?pretty
{
  "cluster_name" : "docker-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 10,
  "active_shards" : 10,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 5,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 66.66666666666666
}

配置文件 #

Kibana配置（kibana.yaml） #

server.name: tone-kibana
server.host: 0.0.0.0

elasticsearch.hosts: [ "${ELASTICSEARCH_HOSTS}" ]
monitoring.ui.container.elasticsearch.enabled: true
server.basePath: /app/kibana

# X-Pack security credentials

elasticsearch.username: "kibana_system"
elasticsearch.password: "${KIBANA_PASSWORD}"
elasticsearch.ssl.certificateAuthorities: "/usr/share/kibana/config/certs/ca/ca.crt"

Elasticsearch配置（elasticsearch.yml） #

---
## Default Elasticsearch configuration from Elasticsearch base image.
## https://github.com/elastic/elasticsearch/blob/master/distribution/docker/src/docker/config/elasticsearch.yml
#
cluster.name: "docker-cluster"
network.host: 0.0.0.0
node.name: "${SVC_ELASTIC}"

## X-Pack settings
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-xpack.html
#
xpack.license.self_generated.type: basic 
xpack.monitoring.collection.enabled: false

## Security
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.key: "/usr/share/elasticsearch/config/certs/${SVC_ELASTIC}/${SVC_ELASTIC}.key"
xpack.security.transport.ssl.certificate: "/usr/share/elasticsearch/config/certs/${SVC_ELASTIC}/${SVC_ELASTIC}.crt"
xpack.security.transport.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/ca/ca.crt"
xpack.security.authc:
  anonymous:
    username: anonymous_user
    roles: metrics
    authz_exception: true

# backup repo
path.repo: ["/backup"]

关键错误日志 #

Kibana日志显示请求使用了匿名用户而非配置的kibana_system，导致权限不足：

[2023-07-28T08:19:11.395+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: action [cluster:monitor/nodes/info] is unauthorized for user [anonymous_user] with effective roles [metrics], this action is granted by the cluster privileges [monitor,manage,all]

原因分析 #

Elasticsearch启用了匿名用户认证模块（xpack.security.authc.anonymous配置），但Kibana的初始化请求未正确携带配置的kibana_system用户凭据，导致请求被自动切换为匿名用户。而anonymous_user关联的metrics角色不具备cluster:monitor/nodes/info权限，无法完成Kibana启动所需的集群节点信息查询，最终导致Kibana服务无法就绪。

解决方法 #

1. 验证Kibana环境变量注入
   检查Kubernetes中Kibana的环境变量ELASTICSEARCH_HOSTS和KIBANA_PASSWORD是否正确注入Pod，确保KIBANA_PASSWORD与Elasticsearch中kibana_system用户的密码一致。可在Pod内执行echo ${KIBANA_PASSWORD}确认。

2. 手动验证凭据权限
   在Kibana Pod内执行以下命令，确认kibana_system用户具备节点信息查询权限：

   curl -u kibana_system:${KIBANA_PASSWORD} tone-elasticsearch:9200/_nodes/info?pretty
   

   如果返回正常节点信息，说明凭据有效，问题出在Kibana配置加载环节；如果返回权限错误，需要检查kibana_system用户的角色是否被修改（默认kibana_system角色已包含所需权限）。

3. 调整Elasticsearch匿名用户配置

   • 方案一：禁用匿名访问（推荐）
     注释或删除Elasticsearch.yml中的xpack.security.authc.anonymous配置块，重启Elasticsearch Pod，强制Kibana使用配置的kibana_system用户连接。
   • 方案二：给匿名用户添加权限
     如果需要保留匿名访问，给metrics角色添加monitor集群权限：

     curl -u elastic:somepassword -XPUT tone-elasticsearch:9200/_security/role/metrics -H "Content-Type: application/json" -d '{
       "cluster": ["monitor"],
       "indices": [
         {
           "names": ["*"],
           "privileges": ["read", "view_index_metadata"]
         }
       ]
     }'
     

4. 检查SSL配置
   确认Kibana的elasticsearch.ssl配置是否与Elasticsearch的HTTP SSL设置匹配，如果Elasticsearch启用了HTTP SSL，Kibana可能需要额外配置客户端证书（当前配置仅指定了CA证书）。

5. 重启Kibana
   完成上述调整后，重启Kibana Pod，观察日志是否显示使用kibana_system用户成功连接Elasticsearch，且无权限错误。

内容的提问来源于stack exchange，提问作者h00jraq