Azure Kubernetes集群中Cilium日志解析：Kusto自定义函数可行性咨询

Absolutely! You can absolutely build a custom function in Kusto (the query language behind Azure Log Analytics) to parse this specific Cilium log format. This is a common scenario when dealing with unstructured logs that follow a key=value pattern (with quoted values for spaces), and Kusto’s flexibility makes this totally doable.

Step 1: Create the Custom Parsing Function #

Here’s a custom function tailored to your Cilium log format. It handles both quoted values (like msg="Identity of endpoint changed") and unquoted values (like level=info):

.create-or-alter function parse_cilium_logs(log_line: string)
{
    // Regex pattern to match key=value pairs, accounting for quoted and unquoted values
    let cilium_log_pattern = @'(?<key>\w+)=((?<value_quoted>"[^"]+")|(?<value_unquoted>[^\s]+))\s*';
    log_line
    // Parse all key-value pairs in the log line using the regex
    | parse kind=regex multiple with * cilium_log_pattern *
    // Pick the non-empty value (either quoted or unquoted)
    | extend value = coalesce(tostring(value_quoted), tostring(value_unquoted))
    // Remove surrounding quotes from quoted values
    | extend value = replace_string(value, '"', '')
    // Group by the original log line to combine all key-value pairs into a dynamic bag
    | summarize bagexpansion=array_merge(pack(key, value)) by log_line
    // Expand the dynamic bag into individual columns
    | evaluate bag_unpack(bagexpansion)
    // Remove the original log line column (optional, if you don't need it anymore)
    | project-away log_line
}

Step 2: How the Function Works #

Let’s break down what each part does:

• Regex Pattern: The pattern (?<key>\w+)=((?<value_quoted>"[^"]+")|(?<value_unquoted>[^\s]+)) targets key names (letters/numbers/underscores), then matches either a quoted value (anything inside double quotes) or an unquoted value (any sequence of non-space characters).
• Multiple Parsing: Using kind=regex multiple ensures we capture all key-value pairs in a single log line, not just the first one.
• Value Cleanup: coalesce picks the valid value (quoted or unquoted), and replace_string strips the surrounding quotes from quoted values.
• Bag Expansion: We pack the key-value pairs into a dynamic object, then use bag_unpack to turn each key into its own column (so you can query level, msg, containerID directly).

Step 3: Use the Function in Your Queries #

Once the function is created, you can invoke it on your Cilium log table. For example, if your logs are stored in a table named CiliumLogs with a column LogMessage:

CiliumLogs
// Invoke our custom parsing function
| invoke parse_cilium_logs(LogMessage)
// Now you can query parsed fields directly
| where level == "info"
| project msg, containerID, datapathPolicyRevision

Adjustments for Edge Cases #

If your Cilium logs have keys with special characters (like hyphens), modify the regex pattern’s key group to include those. For example, change (?<key>\w+) to (?<key>[\w-]+) to allow hyphens in key names.

内容的提问来源于stack exchange，提问作者lmr2391