如何消除Grafana中显示的来自InfluxDB的已终止Kubernetes Pod

Hey there! Dealing with cluttered Pod dashboards full of terminated pods is a common pain point with this stack. Let’s break this down into two practical approaches—filtering in Grafana (the safer, more flexible option) and cleaning up the InfluxDB data (if you definitely don’t need historical terminated pod records).

Option 1: Filter Terminated Pods in Grafana (Recommended) #

This method keeps your historical data intact but hides terminated pods from your dashboards, which is great for troubleshooting later if needed.

1.1 Use Grafana Variables for Dynamic Filtering #

Create a reusable variable to let you switch between viewing running vs. terminated pods:

• Go to your dashboard settings → Variables → Add variable.
• Set the variable type to Query.
• For the data source, select your InfluxDB instance.
• Enter this query to fetch all pod phases:

  SELECT DISTINCT(phase) FROM "kubernetes_pod"
  

• Name the variable something like pod_phase, set the default value to Running, and save.
• Update your panel queries to include the variable: add AND "phase" = '$pod_phase' to the WHERE clause. For example:

  SELECT mean("cpu/usage_rate") FROM "kubernetes_pod" 
  WHERE $timeFilter AND "phase" = '$pod_phase' 
  GROUP BY "pod_name"
  

Now you can use the dropdown to toggle between showing only running pods or all pods.

1.2 Hardcode Filtering in Panel Queries #

If you don’t need dynamic switching, just modify each Pod-related panel’s query to exclude terminated phases:
Add AND "phase" = 'Running' (or exclude Failed/Succeeded explicitly) to your existing query. Example:

SELECT mean("memory/usage") FROM "kubernetes_pod" 
WHERE $timeFilter AND "phase" != 'Failed' AND "phase" != 'Succeeded' 
GROUP BY "pod_name"

1.3 Use Grafana Transforms #

For existing panels with loaded data, you can apply a transform to filter out terminated pods:

• Edit the panel → Go to the Transform tab → Add transform.
• Select Filter by value.
• Set the filter to: phase → is not → Failed (add another filter for Succeeded too).
• Apply the transform, and your panel will only show running pods.

Option 2: Clean Up Terminated Pod Data from InfluxDB #

If you’re sure you don’t need historical data for terminated pods, you can delete these records directly from the database.

2.1 Access the InfluxDB CLI #

First, connect to your InfluxDB instance’s command line (replace placeholders with your actual values):

influx -host <your-influxdb-ip> -port <influxdb-port> -database <heapster-db-name>

(Note: Heapster typically uses k8s or heapster as the default database name.)

2.2 Delete Terminated Pod Records #

• To delete all terminated pods (Failed/Succeeded phases):

  DELETE FROM "kubernetes_pod" WHERE "phase" = 'Failed' OR "phase" = 'Succeeded'
  

• To delete only old terminated pods (e.g., older than 7 days):

  DELETE FROM "kubernetes_pod" 
  WHERE ("phase" = 'Failed' OR "phase" = 'Succeeded') AND time < now() - 7d
  

2.3 Clean Up InfluxDB Series (Optional) #

After deleting data, you can remove unused series to free up space:

DROP SERIES FROM "kubernetes_pod" WHERE "phase" = 'Failed' OR "phase" = 'Succeeded'

Warning: This deletes all data for these pod phases, not just old records—use carefully.

2.4 Set Up a Retention Policy (Automate Cleanup) #

To avoid this problem in the future, create an InfluxDB retention policy to automatically delete old data:

CREATE RETENTION POLICY "30day_retention" ON "<heapster-db-name>" 
DURATION 30d REPLICATION 1 DEFAULT

This sets a 30-day retention period, so all data (including terminated pods) older than 30 days will be deleted automatically.

Quick Note #

Keep in mind that Heapster has been deprecated for a while—modern Kubernetes monitoring typically uses Metrics Server + Prometheus + Grafana. If you’re planning to upgrade, that stack has built-in better handling of pod lifecycle data. But for your current setup, the above methods should solve your problem!

内容的提问来源于stack exchange，提问作者Yonah Dissen