You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

如何验证Filebeat是否将日志发送至Elasticsearch Ingest Pipeline

avatar
算力阿峰
2026-5-8
问题

我们直接将日志从Filebeat发送至Elasticsearch，未经过Logstash处理。日志的不同字段中包含需要解析的JSON内容。我已创建名为application_pipeline的Ingest Pipeline，在开发者控制台测试后输出符合预期，并在filebeat.yml中添加配置项pipeline: application_pipeline，指定Filebeat将日志发送至该管道。但在索引管理中仅能看到原始文档，请问如何检查Filebeat是否确实将日志发送至该管道？

日志示例 

{"level":"info","message":"Webhook DeletePrice-{\"_headers\":{\"x-forwarded-proto\":[\"https\"],\"x-requested-with\":[\"\"],\"x-client-ip\":[\"93.84.120.32\"],\"user-agent\":[\"1C+Enterprise\\/8.3\"],\"accept\":[\"application\\/json\"],\"host\":[\"host.com\"],\"content-length\":[\"\"],\"content-type\":[\"\"]},\"company_id\":\"10248103\",\"service_id\":\"102.01.02S\",\"service_type\":\"clientApi\"}","service":"servicename","project":"someproject.com","event_id":"255A854BED569B8D4C21B5DE6D8E109C","payload":[],"date_server":"2020-07-24T11:45:48+00:00","date_unix":1595591148.966919}
{"level":"error","message":"NO service integration","service":"servicename","project":"someproject.com","event_id":"D3986456E5A42AF8574230C29D1D474D","payload":{"exception":{"class":"\\Ship\\Exceptions\\IntegrationException","message":"NO service integration","code":0,"file":"/var/www/builds/someproject.com/build.lab.service-public-api.2020_07_22_12_17_45/app/Containers/Price/UI/API/Controllers/Controller.php:406"}},"date_server":"2020-07-24T08:40:34+00:00","date_unix":1595580034.975073}
{"level":"info","message":"No photo in priceId-3696930","service":"service-private-api","project":"someproject.com","event_id":"FBEDA2C9600BFE11523592114B32BAEB","payload":[],"date_server":"2020-07-24T12:16:40+00:00","date_unix":1595593000.97212}
{"level":"error","message":"C404HttpException: 404 Страница не найдена in /var/www/builds/build.lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/modules/personal/controllers/RobotsController.php:65\nStack trace:\n#0 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(4226): RobotsController->actionIndex()\n#1 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3739): CInlineAction->runWithParams(Array)\n#2 /var/www/builds/build.lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3724): CController->runAction(Object(CInlineAction))\n#3 /var/www/builds/build.lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3714): CController->runActionWithFilters(Object(CInlineAction), Array)\n#4 /var/www/builds/build.lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1799): CController->run('index')\n#5 /var/www/builds/build.lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1719): CWebApplication->runController('personal/robots...')\n#6 /var/www/builds/build.lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1236): CWebApplication->processRequest()\n#7 /var/www/builds/build.lab.classified-platform.2020_07_29_12_13_54/htdocs/index.php(22): CApplication->run()\n#8 {main}\nREQUEST_URI=/robots.txt\n---","service":"artox-lab/classified-platform","project":"someproject.com","event_id":"91a10782a3566a74d5abefa9589c926c","payload":"exception.C404HttpException.404","date_server":"2020-07-29T14:25:34+03:00","date_unix":1596021934.218448}

Ingest Pipeline示例 

PUT _ingest/pipeline/application_pipeline
{ 
  "description" : "Pipeline for parsing application.log for services",
  "processors" : [
    {
      "grok" : {
        "field" : "message",
        "patterns" : [ "%{JSON:json_message_payload}" ],
        "pattern_definitions" : {
          "JSON" : "{.*$"
        },
        "ignore_failure" : true,
        "ignore_missing" : true
      }
    },
    {
      "remove" : {
        "field" : "json_message_payload",
        "ignore_failure" : true
      }
    }
  ]
}

Pipeline测试输出 

{ 
  "_index" : "application_index",
  "_type" : "_doc",
  "_id" : "6",
  "_version" : 1,
  "_seq_no" : 3,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "date_server" : "2020-07-29T15:16:17+03:00",
    "level" : "error",
    "project" : "103by",
    "message" : """ C404HttpException: 404 Страница не найдена in /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/modules/personal/components/PersonalController.php:140 Stack trace: #0 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3737): PersonalController->beforeAction(Object(ShowGalleryPhotoAction)) #1 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3724): CController->runAction(Object(ShowGalleryPhotoAction)) #2 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3714): CController->runActionWithFilters(Object(ShowGalleryPhotoAction), Array) #3 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1799): CController->run('showGalleryPhot...') #4 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1719): CWebApplication->runController('personal/galler...') #5 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1236): CWebApplication->processRequest() #6 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/index.php(22): CApplication->run() #7 {main} REQUEST_URI=/gallery/23609/1439643/ HTTP_REFERER=http://rnpcomr.103.by/gallery/23609/1439643/ --- """,
    "date_unix" : 1.596024977817727E9,
    "event_id" : "b75c7a1ef2f8780986931b038d2f8599",
    "payload" : "exception.C404HttpException.404",
    "service" : "artox-lab/classified-platform"
  }
}

Filebeat相关配置 

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["elk.artoxlab.com:9200"]
  pipeline: application_pipeline
检查方法及排查步骤

下面这些方法可以帮你验证Pipeline是否被正确调用，以及定位问题所在：

1. 查看文档的Ingest元数据

当文档经过Pipeline处理后，Elasticsearch会在文档的元数据中记录使用的Pipeline名称。你可以通过查询单个文档的完整信息来确认：

GET /<你的索引名称>/_doc/<目标文档ID>

如果文档确实经过了指定Pipeline，返回结果中会包含_ingest字段，示例如下：

{
  "_index" : "your-index",
  "_type" : "_doc",
  "_id" : "123",
  "_version" : 1,
  "_ingest": {
    "pipeline": "application_pipeline"
  },
  "_source" : { ... }
}

如果没有这个字段，说明文档完全没经过Pipeline处理。

2. 查看Pipeline的执行统计数据

Elasticsearch会跟踪每个Pipeline的调用情况，你可以通过以下命令查看application_pipeline的运行统计：

GET _ingest/pipeline/application_pipeline/_stats

返回结果中的count（总调用次数）、success（成功次数）、failed（失败次数）等指标能直接反映Pipeline是否被触发。如果count为0，说明Pipeline从未被调用；如果failed有数值，说明处理过程中存在错误，需要进一步排查。

3. 开启Filebeat调试日志排查请求

修改Filebeat配置文件，开启Elasticsearch输出模块的调试日志：

logging.level: debug
logging.selectors: ["output.elasticsearch"]

重启Filebeat后，查看日志文件（默认路径一般为/var/log/filebeat/filebeat.log），搜索pipeline关键词，确认Filebeat在发送请求时是否携带了pipeline: application_pipeline参数。如果日志中完全看不到相关内容，说明配置可能未生效或写错了位置。

4. 验证Filebeat配置的正确性

用Filebeat自带的命令检查配置文件是否有语法错误：

filebeat test config -c filebeat.yml

如果配置存在问题，这个命令会直接抛出错误提示。另外，也可以查看Filebeat的启动日志，确认配置是否被正确加载，有没有关于Pipeline的警告或错误信息。

5. 手动模拟请求测试Pipeline

用curl命令模拟Filebeat发送日志的请求，手动指定Pipeline参数，验证Pipeline本身是否能正常处理日志：

curl -X POST "http://elk.artoxlab.com:9200/_bulk?pipeline=application_pipeline" -H "Content-Type: application/json" -d '
{"index":{}}
{"level":"info","message":"Webhook DeletePrice-{\"_headers\":{\"x-forwarded-proto\":[\"https\"]},\"company_id\":\"10248103\"}","service":"servicename","project":"someproject.com","event_id":"test-id","date_server":"2020-07-24T11:45:48+00:00"}
'

之后查看索引中的文档，如果手动请求能正常处理但Filebeat发送的不行，问题大概率出在Filebeat的配置或运行状态上。

额外优化：给Pipeline添加处理标记

从你的Pipeline配置来看，grok处理器设置了ignore_failure: true，匹配失败时不会报错，文档会原样存入索引。你可以给Pipeline添加一个标记字段，直观区分是否经过处理：

PUT _ingest/pipeline/application_pipeline
{ 
  "description" : "Pipeline for parsing application.log for services",
  "processors" : [
    {
      "set": {
        "field": "pipeline_processed",
        "value": "true"
      }
    },
    {
      "grok" : {
        "field" : "message",
        "patterns" : [ "%{JSON:json_message_payload}" ],
        "pattern_definitions" : {
          "JSON" : "{.*$"
        },
        "ignore_failure" : true,
        "ignore_missing" : true
      }
    },
    {
      "remove" : {
        "field" : "json_message_payload",
        "ignore_failure" : true
      }
    }
  ]
}

之后新存入的文档如果带有pipeline_processed: true字段，就说明确实经过了该Pipeline处理。

内容的提问来源于stack exchange，提问作者Axel_lurcher

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
火山引擎 最新活动
全面开放
Doubao-Seedance-2.5全新模型升级

模型再升级,30秒超长叙事, 模态参考扩容

HOT
方舟Codingplan

模型自由，工具不限，最新支持 Deepseek-V4 系列、GLM-5.3 系列

HOT
方舟Agent plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列

HOT
新用户特惠专场

大模型19元起，Al应用9.9元畅享，新人首购爆款尽享优惠

最近更新时间：2026.05.08 17:57:44