Logstash导入JSON至Elasticsearch失败：索引未更新问题排查

我有如下file.json文件内容：

{"event":"send_message","id":"9ee9295e-1c11-41be-8e8c-219688f0c770","type":"link","channel_number":"5735565091","steps":[{"step1":"verify_queue","timestamp1":1664829652,"step2":"send_to_route","timestamp2":1664829680,"step3":"wait_for_confirmation","timestamp3":1664829690,"step4":"remove_from_queue","timestamp4":1664829718}],"number_of_tries":1,"status":true}
{"event":"send_message","id":"daa0acda-3924-42ca-b8d1-0aa384eb9a27","type":"text","channel_number":"5591594463","steps":[{"step1":"verify_queue","timestamp1":1664829653,"step2":"send_to_route","timestamp2":1664829713,"step3":"wait_for_confirmation","timestamp3":1664829741,"step4":"remove_from_queue","timestamp4":1664829769}],"number_of_tries":6,"status":true}

尝试通过Logstash将数据上传至已创建的Elasticsearch索引，但索引未更新。我的Logstash配置如下：

input {
  file {
    type => "json"
    path => "/home/user/logs/file.json"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"
  }
}

filter {
  json {
    source => "message"
  }
}

output {
  elasticsearch{
        hosts => ["elasticsearch:9200"]
        user => elastic
        password => changeme
        index => "testando"
        document_type => "json"
    }
}

我使用的是elk-docker，请问哪里配置出错了？

你的配置存在三个关键问题：

• 重复JSON解析导致处理失败
  输入模块的file插件已经指定了codec => "json"，这会自动将每行JSON内容解析为Logstash事件，此时事件中不存在message字段。而filter模块里的json { source => "message" }会因为找不到message字段抛出错误，导致事件无法正常输出到Elasticsearch。
  解决：删除filter模块的json解析配置。

• 过时的document_type参数
  Elasticsearch 7.x及以上版本已经移除了文档类型（document_type）的概念，Logstash的elasticsearch输出插件中保留这个参数会导致连接或写入失败。
  解决：删除document_type => "json"这一行。

• Docker环境下的文件路径挂载问题
  你在配置中指定的路径/home/user/logs/file.json是宿主机路径，但Logstash容器内部无法直接访问宿主机文件，需要确保该目录已经通过Docker挂载到容器内部的对应路径（比如/usr/share/logstash/logs/file.json），并且容器有读取权限。
  解决：启动Logstash容器时添加挂载参数，例如：

  docker run -v /home/user/logs:/usr/share/logstash/logs ...
  

  同时修改Logstash配置中的path为容器内路径：path => "/usr/share/logstash/logs/file.json"

修正后的Logstash配置 #

input {
  file {
    type => "json"
    path => "/usr/share/logstash/logs/file.json" # 改为容器内挂载路径
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"
  }
}

# 移除多余的json filter

output {
  elasticsearch{
        hosts => ["elasticsearch:9200"]
        user => elastic
        password => changeme
        index => "testando"
    }
}

额外排查步骤 #

1. 查看Logstash容器日志，确认是否有连接错误、权限错误或解析错误：

   docker logs <logstash-container-name>
   

2. 验证Elasticsearch连接：在Logstash容器内部执行curl命令测试ES连通性：

   curl -u elastic:changeme http://elasticsearch:9200
   

3. 确认testando索引已创建，且elastic用户有写入权限。

内容的提问来源于stack exchange，提问作者Arthur Breguez