如何在Datadog Grok处理管道中从JSON列表提取字段？

If you're looking to pull out specific fields from that JSON array in Datadog's processing pipeline, you have two solid approaches—using Datadog's built-in processors (the easiest and most efficient way) or combining Grok with JSON parsing if your data is embedded in unstructured logs. Let's break both down step by step:

Approach 1: Use Datadog's Native Processors (Recommended) #

This is the best path for pure JSON data since it leverages Datadog's optimized parsers:

1. Parse the JSON Array

   • Head to your Datadog pipeline and add a JSON Parser processor.
   • Set the Source to message (assuming your raw JSON is in the default message field).
   • Choose a Target name like service_list—this will turn the JSON array into a structured array of objects under that field.
   • Save the processor.

2. Split the Array into Individual Events

   • Next, add a Split processor.
   • Set the Source to service_list (the array we just parsed). This will split each object in the array into its own separate log event.
   • Now each event will have all your desired fields (like Names, ActiveState, ASG) as top-level structured fields, ready to use in dashboards or monitors.

3. Optional: Tweak Fields

   • If you need to rename fields or adjust their types, use the Rename or Attribute Remapper processors to fine-tune things.

Approach 2: Grok + JSON Parsing (For Embedded JSON) #

If your JSON is wrapped in unstructured log text (e.g., [SERVICE_LOG] [{"ExecMainStartTimestamp": "...", ...}]), use Grok to first extract the JSON payload, then parse it:

1. Extract JSON with Grok

   • Add a Grok Parser processor and use this pattern to capture the entire JSON array into a field:

     %{GREEDYDATA:json_payload}
     

     Adjust the pattern if your log has a specific prefix/suffix (e.g., \[SERVICE_LOG\] %{GREEDYDATA:json_payload}).

2. Parse the Extracted JSON

   • Add the JSON Parser processor again, this time setting Source to json_payload and Target to service_list.
   • Follow up with the Split processor as in Approach 1 to break the array into individual events.

Example Final Event #

After processing, each event will look like this (with all fields neatly structured):

{
  "ExecMainStartTimestamp": "Wed 2021-10-27 11:31:36 UTC",
  "hostname": "i-XX",
  "_time": "2021-10-27 12:20:01",
  "ASG": "prod1",
  "ExecMainPID": "1447",
  "Names": "nginx.service",
  "LoadState": "loaded",
  "ActiveState": "active",
  "UnitFileState": "enabled"
}

This setup ensures you can easily query, visualize, or alert on any of these fields in Datadog.

内容的提问来源于stack exchange，提问作者Dipal Parmar