Elasticsearch Logstash过滤器补全Hostname时遇401授权错误求助

场景说明 #

需要从.fleet-agents索引补全*-testalert类主索引中缺失的hostname字段，通过Logstash的Elasticsearch过滤器实现时，手动测试管道触发401未授权错误。

错误日志 #

[ERROR] 2023-04-24 10:02:58.784 [[main]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"main", :exception=>#<Elasticsearch::Transport::Transport::Errors::Unauthorized: [401] >, :backtrace=>["/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/transport/base.rb:341:in `perform_request'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/transport/http/manticore.rb:91:in `perform_request'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/client.rb:197:in `perform_request'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-7.17.1/lib/elasticsearch.rb:41:in `method_missing'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/elasticsearch-api-7.17.1/lib/elasticsearch/api/actions/ping.rb:38:in `ping'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-filter-elasticsearch-3.12.0/lib/logstash/filters/elasticsearch.rb:330:in `test_connection!'", "/appserver/logstash-8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-filter-elasticsearch-3.12.0/lib/logstash/filters/elasticsearch.rb:118:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:233:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:232:in `register_plugins'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:599:in `maybe_setup_out_plugins'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:245:in `start_workers'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:190:in `run'", "/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:142:in `block in start'"], "pipeline.sources"=>["/data/logstash/pipelines/sendEmailAlerts_updated.conf"], :thread=>"#<Thread:0x1d8d4a8@/appserver/logstash-8.4.3/logstash-core/lib/logstash/java_pipeline.rb:130 run>"}
[INFO ] 2023-04-24 10:02:58.785 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[ERROR] 2023-04-24 10:02:58.793 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

备注与配置文件 #

• 其他Logstash管道运行正常，排除全局配置问题
• 对应管道的配置文件如下：

input {
        elasticsearch
        {
        hosts => "localhost:9200"
        user => "reader"
        password => "*******************"
        index => "*-testalert"
        query => '{ "query": {
                             "bool": {
                                      "must": [{"terms": { "kibana.alert.severity": [ "high", "critical"] }} ],
                                      "filter": [ {"range":{"@timestamp": { "gte": "now-2d"}}}]}
                             }
                  }'
        schedule => "/5 * * * *"
        size => 500
        scroll => "5m"
        docinfo => true
        docinfo_target => "[@metadata][doc]"
        codec => "json"
        }
    }

filter {
if [host][hostname] != ""
{
mutate {
        add_field => {
            "alertHostName" => "%{[host][hostname]}"
            "alertReason" => "%{kibana.alert.reason}"
            "alertSeverity" => "%{kibana.alert.severity}"
            "alertTime" => "%{kibana.alert.original_time}"

        }
    }
}
if [host][hostname] == ""
{
elasticsearch {
              hosts => "localhost:9200"
              index => ".fleet-agents"
              query => "{[local_metadata][host][id]}:%{[host][id]}"
              fields => {
                       "[local_metadata][host][id]" => "host_name"
                      }
              }

mutate {
        add_field => {
            "alertHostName" => "%{[host_name]}"
            "alertReason" => "%{kibana.alert.reason}"
            "alertSeverity" => "%{kibana.alert.severity}"
            "alertTime" => "%{kibana.alert.original_time}"

        }
    }
}
}

output {
stdout {
 codec => "json"
}
}

问题原因与修复方案 #

核心原因 #

过滤器中的Elasticsearch模块未配置认证信息：input段的Elasticsearch输入已配置user和password，但filter段的Elasticsearch过滤器是独立连接，不会自动继承input的认证参数，导致访问.fleet-agents索引时未携带凭证，触发401错误。

修复步骤 #

在filter段的elasticsearch配置块中添加与input一致的认证参数：

elasticsearch {
              hosts => "localhost:9200"
              user => "reader"
              password => "*******************"
              index => ".fleet-agents"
              query => "{[local_metadata][host][id]}:%{[host][id]}"
              fields => {
                       "[local_metadata][host][id]" => "host_name"
                      }
              }

额外优化 #

1. 修正查询语法：当前查询格式不符合Lucene语法，建议改为：

   query => "local_metadata.host.id:%{[host][id]}"
   

   或使用完整DSL查询：

   query => '{ "query": { "match": { "local_metadata.host.id": "%{[host][id]}" } } }'
   

2. 修正字段映射：当前配置映射的是id字段，若要获取hostname，需调整为：

   fields => {
       "[local_metadata][host][hostname]" => "host_name"
   }
   

内容的提问来源于stack exchange，提问作者NiveditaK