启用Xpack的Elasticsearch集群与Logstash连接失败求助

一、先解决Logstash启动失败的核心问题 #

日志中明确报错：Found a file at /etc/logstash/logstash.keystore, but it is not a valid Logstash keystore.，这是导致Logstash启动崩溃的直接原因。

修复步骤： #

1. 删除无效的keystore文件

rm /etc/logstash/logstash.keystore

2. 若需要存储敏感信息（如ES密码），重新创建合法的Logstash keystore

# 创建keystore，设置访问密码
/usr/share/logstash/bin/logstash-keystore create

# 将ES用户密码存入keystore（避免配置文件明文暴露）
/usr/share/logstash/bin/logstash-keystore add output.elasticsearch.password
# 执行后输入对应的ES用户密码即可

二、修正Logstash到Elasticsearch的连接配置 #

启动问题解决后，还需要调整ES输出配置，确保能正常连接启用Xpack的ES集群：

1. 证书使用错误修正 #

当前配置中使用的elastic-certificates.p12是ES节点间transport层的通信证书，Logstash通过HTTP接口连接ES需要使用HTTP CA证书（http_ca.crt）：

• 从任意ES节点复制/etc/elasticsearch/certs/http_ca.crt到Logstash节点的/etc/logstash/certs/目录（先创建certs目录）
• 修改输出配置，移除truststore项，添加cacert配置：

elasticsearch {
  hosts => ["https://172.16.20.66:9200","https://172.16.20.60:9200","https://172.16.20.105:9200","https://172.16.100.28:9200"]
  index => "jsonfile-daemonset-applog-%{+YYYY.MM.dd}"
  cacert => "/etc/logstash/certs/http_ca.crt"
  user => "logstash_writer" # 后续替换为专用用户
  password => "${output.elasticsearch.password}" # 从keystore读取密码
}

2. 用户权限优化 #

logstash_system用户默认用于Logstash与ES的内部监控通信，不建议用来写入业务数据，建议创建专用用户：
在任意ES节点执行：

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -XPOST https://localhost:9200/_security/user/logstash_writer -H "Content-Type: application/json" -d '{
  "password" : "your_secure_password",
  "roles" : ["index_admin"] # 可根据需求创建更细粒度的自定义角色，赋予索引创建、写入权限
}'

然后将Logstash配置中的user字段替换为logstash_writer。

3. 配置精简优化 #

合并重复的ES输出配置，减少冗余：

output {
  stdout { codec => rubydebug }
  
  if [type] in ["jsonfile-daemonset-applog", "jsonfile-daemonset-syslog"] {
    elasticsearch {
      hosts => ["https://172.16.20.66:9200","https://172.16.20.60:9200","https://172.16.20.105:9200","https://172.16.100.28:9200"]
      index => "jsonfile-daemonset-%{type}-%{+YYYY.MM.dd}"
      cacert => "/etc/logstash/certs/http_ca.crt"
      user => "logstash_writer"
      password => "${output.elasticsearch.password}"
    }
  }
}

三、验证与启动 #

1. 测试配置文件语法正确性：

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logsatsh-daemonset-jsonfile-kafka-to-es.conf --config.test_and_exit

2. 启动Logstash：

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logsatsh-daemonset-jsonfile-kafka-to-es.conf --path.settings=/etc/logstash

内容的提问来源于stack exchange，提问作者Wei Yu