使用and运算符后执行Prometheus除法查询无数据返回求助

Hey there! Let's break down why you're hitting that "no data" roadblock when dividing your two memory metrics after filtering with node_meta—even though each filtered metric works perfectly on its own.

Common Root Causes #

1. Mismatched Label Sets #

Prometheus binary operations (like /) demand exact label matches between the left and right-hand time sequences, unless you explicitly use group_left/group_right to allow partial matches. When you chain and node_meta{...} to your queries, you might be adding extra labels from node_meta to one or both metrics that don't align perfectly. For example, if node_meta includes a namespace label that one of your memory metrics lacks, the division will fail because the label sets don't match.

2. Time Sequence Misalignment #

Even if each metric has data, their timestamps might not overlap perfectly. The and operator only retains time points where both metrics have a sample. If one metric has gaps in the time range you're querying, the combined sequence might have no overlapping points left for the division to compute.

3. Misuse of and for Label Filtering #

The and operator is a logical check—it verifies that samples exist at the same time with matching labels. But if your goal is to filter memory metrics by node_name via node_meta, and isn't always the right tool. It's better to use label-based matching (like on()/ignoring()) to associate the metrics more reliably.

Step-by-Step Troubleshooting #

• Check Label Consistency
  Run these two queries to compare the label sets of your filtered metrics:

  label_names(node_memory_MemAvailable{instance="10.0.0.15:9100",job="node-exporter"} and node_meta{node_name="dc0-docker6"})
  

  label_names(node_memory_MemTotal{instance="10.0.0.15:9100",job="node-exporter"} and node_meta{node_name="dc0-docker6"})
  

  If the results differ, use label_replace to strip or add labels to align them, or add group_left()/group_right() to your division query to allow partial matching. Example:

  (node_memory_MemAvailable{instance="10.0.0.15:9100",job="node-exporter"} and node_meta{node_name="dc0-docker6"}) 
  / 
  on(instance, job) group_left() 
  (node_memory_MemTotal{instance="10.0.0.15:9100",job="node-exporter"} and node_meta{node_name="dc0-docker6"})
  

• Verify Time Range Overlap
  Use range queries to check if both metrics have samples in the same time window:

  node_memory_MemAvailable{instance="10.0.0.15:9100",job="node-exporter"} and node_meta{node_name="dc0-docker6"}[1h]
  

  node_memory_MemTotal{instance="10.0.0.15:9100",job="node-exporter"} and node_meta{node_name="dc0-docker6"}[1h]
  

  Look for gaps in either sequence—if one has missing samples, adjust your query time range or check the node-exporter's scrape configuration.

• Switch to Label-Based Matching Instead of and
  Replace and with * on(instance) group_left() to associate memory metrics with node_meta by the instance label. This avoids unexpected label mismatches. Try this query:

  (node_memory_MemAvailable{instance="10.0.0.15:9100",job="node-exporter"} * on(instance) group_left(node_name) node_meta{node_name="dc0-docker6"})
  /
  (node_memory_MemTotal{instance="10.0.0.15:9100",job="node-exporter"} * on(instance) group_left(node_name) node_meta{node_name="dc0-docker6"})
  

• Check node_meta Instance Label Format
  Ensure the instance label in node_meta{node_name="dc0-docker6"} exactly matches the one in your memory metrics. For example, if node_meta uses 10.0.0.15 (without the port) but your memory metrics use 10.0.0.15:9100, the and operator won't match. Fix this with label_replace:

  node_memory_MemAvailable{instance="10.0.0.15:9100",job="node-exporter"} 
  and 
  label_replace(node_meta{node_name="dc0-docker6"}, "instance", "$1:9100", "instance", "(.*)")
  

内容的提问来源于stack exchange，提问作者gaelperret