如何为Prometheus采集的mtail指标添加自定义标签？

Got it, let's break down how to add extra labels to your mtail-generated line_count metric. There are a couple of straightforward ways to do this depending on your needs:

1. Add Fixed Labels Directly in the mtail Program #

mtail lets you attach static labels when defining your metric. Just add a {key="value"} block right after the metric type and name. For example, modify your linecounter.mtail code like this:

counter line_count {source="application_logs", environment="staging"} /$/ {
  line_count++
}

Once you restart mtail, Prometheus will collect the metric with these extra fixed labels:

line_count{prog="linecounter.mtail",instance="bd0a0d119df6",source="application_logs",environment="staging"}

2. Add Global Labels via mtail Command-Line Arguments #

If you want the same set of labels applied to all metrics generated by mtail (great for things like service name, region, or hostname), use the --label flag when starting mtail. For example:

mtail --label service="user_auth" --label region="eu-west-1" -f /var/log/app.log -p 3903 linecounter.mtail

This will append the service and region labels to every metric mtail exposes, including your line_count metric.

3. Extract Dynamic Labels from Log Content #

If you need labels that change based on the log content (like request paths, user IDs, or environment values from the log itself), you can capture those values with regex and attach them as dynamic labels.

For example, if your logs look like [env:prod] 2024-05-20 12:34:56 Request processed, you can extract the env value like this:

counter line_count /^\[env:([a-z]+)\].*$/ {
  line_count[$1]++
}

Prometheus will then show metrics with dynamic labels based on the log data:

line_count{prog="linecounter.mtail",instance="bd0a0d119df6",label="prod"}

You can combine these methods too—for example, use command-line global labels plus fixed labels in your mtail program, and all will show up in the final metric.

内容的提问来源于stack exchange，提问作者SangminKim