如何使用OpenSearch Ingest Pipeline拆分多文档日志

你的场景是：OpenSearch集群接收的日志中，log字段是包含多个JSON对象的字符串数组，需要将这类日志拆分为独立的单条文档，且希望避免外部日志处理器带来的额外故障点和成本。可以通过以下Ingest Pipeline配置实现：

完整Pipeline配置 #

{
  "description": "拆分log字段中的多文档数组",
  "processors": [
    {
      "json": {
        "field": "log",
        "target_field": "log_array",
        "ignore_failure": false
      }
    },
    {
      "split": {
        "field": "log_array",
        "target_field": "single_doc",
        "ignore_failure": false
      }
    },
    {
      "script": {
        "source": """
          ctx.putAll(ctx.single_doc);
          ctx.remove('log');
          ctx.remove('log_array');
          ctx.remove('single_doc');
        """,
        "ignore_failure": false
      }
    },
    {
      "set": {
        "field": "@timestamp",
        "value": "{{time}}",
        "ignore_failure": true
      }
    }
  ]
}

各处理器说明 #

• json处理器：将log字段的字符串格式JSON数组解析为OpenSearch可识别的数组类型，存入log_array字段，为后续拆分做准备。
• split处理器：基于log_array数组拆分文档，数组中的每个元素会生成一个独立的新文档，拆分后每个文档的single_doc字段对应原数组中的单个JSON对象。
• script处理器：把single_doc中的所有键值对合并到文档根节点，同时清理掉log、log_array、single_doc这些临时字段，让最终文档结构更简洁。
• set处理器（可选）：将原日志的time字段值赋值给OpenSearch标准的@timestamp字段，方便后续进行时间维度的查询、聚合分析。

配置与测试步骤 #

1. 创建Pipeline：通过OpenSearch API提交上述配置，示例命令：

curl -X PUT "http://<your-opensearch-host>:<port>/_ingest/pipeline/split-multi-docs" -H "Content-Type: application/json" -d @pipeline-config.json

2. 测试Pipeline：使用_simulate端点验证效果，示例请求：

POST _ingest/pipeline/split-multi-docs/_simulate
{
  "docs": [
    {
      "_source": {
        "log": """[{"_id": "a1f4770b6e78dddddddddf220", "access_method": "Client", "app": "aaaaaaaaaaaaaaaaaaaaaaaaa Accounts", "appcategory": "Application Suite",  "bypass_traffic": "yes", "category": "Application Suite", "cci": 84,  "dstport": 443, "netskope_pop": "US-BOS1", "organization_unit": "", "os_family": "Windows", "os_version": "Windows NT 12.0", "other_categories": ["aa Allow List", "Technology", "Login Screens", "Application Suite"], "page": "aa.xx.com"},{"_id": "a1f4770b6e783fc56871f220", "access_method": "Client", "app": "bbbbbbbbbbbbbbbbbbbbbbbbbbbb Accounts", "appcategory": "Application Suite",  "bypass_traffic": "yes", "category": "Application Suite", "cci": 84,  "dstport": 443, "netskope_pop": "US-BOS1", "organization_unit": "", "os_family": "Windows", "os_version": "Windows NT 1.0", "other_categories": ["aa Allow List", "Technology", "Login Screens", "Application Suite"], "page": "aa.vv.com"}]""",
        "time": "2025-07-01T23:39:54.427976"
      }
    }
  ]
}

执行后会返回两个独立文档，每个文档包含对应业务字段及@timestamp。

内容的提问来源于stack exchange，提问作者Aditya Pednekar