PromQL中label:metric:function格式语法解析及使用疑问

Great question! Let's unpack what's going on here—this is a common point of confusion with Prometheus recording rules, so you're not alone.

1. This isn't native PromQL syntax—it's a recording rule-generated metric #

First off, that namespace:kube_pod_container_resource_requests_cpu_cores:sum string isn't a special PromQL query syntax. It's the name of a precomputed metric created by a recording rule in the kube-prometheus stack.

Kube-prometheus defines a set of recording rules to pre-aggregate raw metrics into more useful, queryable forms. The rule that generates this specific metric looks something like this (simplified from the repo you're working with):

- record: namespace:kube_pod_container_resource_requests_cpu_cores:sum
  expr: sum by (namespace) (kube_pod_container_resource_requests_cpu_cores)

This rule takes the raw kube_pod_container_resource_requests_cpu_cores metric (your example has labels like container, namespace, pod), aggregates it by the namespace label using sum, and saves the result as a new metric with that structured name.

2. Why container:kube_pod_container_resource_requests_cpu_cores:sum returns no results #

Simply put: kube-prometheus doesn't define a recording rule for that metric name.

Looking at the prometheus-rules.yaml file you referenced, the repo includes rules for aggregating by namespace, node, and other high-level labels—but not by container. Since no rule precomputes and stores that aggregated metric, querying it returns an empty result set.

If you wanted this metric, you could add your own recording rule:

- record: container:kube_pod_container_resource_requests_cpu_cores:sum
  expr: sum by (container) (kube_pod_container_resource_requests_cpu_cores)

After adding this rule and reloading Prometheus, you'd be able to query the container: prefixed metric successfully.

3. The naming convention: {label}:{metric}:{function} #

This structured naming is a widely adopted community convention for recording rules. It's designed to make it immediately clear what a precomputed metric represents:

• {label}: The label used to group/aggregate the original metric
• {metric}: The name of the raw source metric
• {function}: The aggregation function applied (like sum, avg, max)

It's not enforced by Prometheus itself, but it's a best practice that makes your metrics inventory much easier to navigate.

4. Where to find this in Prometheus docs #

• Recording Rules: The official Prometheus docs cover recording rules in detail, including how to define them and their use cases. Look for the "Recording and Alerting Rules" section.
• Metric Naming Best Practices: The Prometheus community best practices guide includes recommendations on naming aggregated metrics like this, to ensure consistency across environments.

5. Can you use the label:metric:function format? #

Absolutely! This is a convention, not a hard rule. You can use this format (or any other naming scheme you prefer) when creating your own recording rules. The key is to be consistent so you and your team can quickly understand what each precomputed metric does.

For example, if you wanted to aggregate by node instead, you could create:

- record: node:kube_pod_container_resource_requests_cpu_cores:avg
  expr: avg by (node) (kube_pod_container_resource_requests_cpu_cores)

内容的提问来源于stack exchange，提问作者Shubham Choudhary