如何从Elasticsearch索引中获取最新数据？

尝试从Elasticsearch索引earthquake-v2中获取最新数据，但当前Logstash配置执行后会打印索引中的所有文档。尝试通过DateTime字段筛选最新记录，始终返回全部文档。

当前Logstash配置：

input {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "earthquake-v2"
    query => '{
     "size": 1,
     "sort": [
       {
         "DateTime": {
           "order": "asc"
         }
       }
     ],
     "query": {
       "match_all": {}
     }
   }'
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

当前输出示例（返回全部文档）：

> {
> "Gap" => "59",
> "parsedDateTime" => "2015-12-31T22:26:08.000Z",
> "Magnitude" => "4.30",
> "Distance" => "2",
> "Longitude" => "72.6379",
> "MagType" => "Mb",
> "@version" => "1",
> "Latitude" => "40.8745",
> "Depth" => "40.30",
> "DateTime" => "2016-01-01 03:26:08",
> "Source" => "us",
> "RMS" => "1.21",
> "@timestamp" => 2024-05-20T07:40:44.169218400Z
> }
> {
> "Gap" => "219",
> "parsedDateTime" => "2015-12-31T22:15:43.000Z",
> "Magnitude" => "1.09",
> "Distance" => "17",
> "Longitude" => "-119.6086",
> "MagType" => "ML",
> "NbStations" => "9",
> "@version" => "1",
> "Latitude" => "41.8857",
> "Depth" => "4.99",
> "DateTime" => "2016-01-01 03:15:43",
> "Source" => "NN",
> "RMS" => "0.10",
> "@timestamp" => 2024-05-20T07:40:44.169218400Z
> }
> {
> "Gap" => "115",
> "parsedDateTime" => "2015-12-31T21:06:45.000Z",
> "Magnitude" => "2.60",
> "Distance" => "0",
> "Longitude" => "-155.0622",
> "MagType" => "ML",
> "@version" => "1",
> "Latitude" => "19.5748",
> "Depth" => "14.40",
> "DateTime" => "2016-01-01 02:06:45",
> "Source" => "hv",
> "RMS" => "0.44",
> "@timestamp" => 2024-05-20T07:40:44.166006100Z
> }

需求：仅获取索引中DateTime字段最新的一条记录，后续用于检查本地数据文件，若有更多记录则创建新索引。

问题出在Logstash的Elasticsearch输入插件默认会分页遍历整个索引，即使你在query里指定了size:1，插件仍会忽略这个参数并拉取所有文档。要解决这个问题，需要添加scroll和全局size参数控制行为，同时修正排序方向。

修改后的配置如下：

input {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "earthquake-v2"
    # 禁用scroll，仅执行单次查询
    scroll => "0s"
    # 全局限制返回的文档总数
    size => 1
    query => '{
     "sort": [
       {
         "DateTime": {
           "order": "desc"
         }
       }
     ],
     "query": {
       "match_all": {}
     }
   }'
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

关键修改说明：

• scroll => "0s"：告诉插件不要使用scroll API遍历全量数据，仅执行一次查询。
• 全局size => 1：直接限制插件返回的文档数量为1。
• 排序方向改为desc：排序后第一条即为DateTime最新的记录。

额外注意：需确认DateTime字段在Elasticsearch中是日期类型，如果是字符串类型，排序可能不符合时间逻辑。从输出看parsedDateTime是标准ISO日期格式，若DateTime为字符串类型，建议改用parsedDateTime字段排序，确保结果正确。

内容的提问来源于stack exchange，提问作者Danyal Danish