如何将Kubernetes中Prometheus指标导出至文件而非UI展示？

Hey, great to hear your Prometheus is up and running smoothly on Kubernetes! If you want to export all those metrics to files instead of just viewing them via the UI, there are a couple of solid approaches you can take — let me break them down for you:

Option 1: One-Time Export of Prometheus TSDB Data #

If you only need to dump metrics occasionally, you can use Prometheus's built-in tsdb dump tool directly from the running pod:

• First, find your Prometheus pod name:

  kubectl get pods -n <your-namespace> | grep prometheus
  

• Shell into the Prometheus container:

  kubectl exec -it <prometheus-pod-name> -n <your-namespace> -- /bin/sh
  

• Run the TSDB dump command to export data to a file (we'll use metrics_dump.json as an example):

  promtool tsdb dump /prometheus/data > metrics_dump.json
  

  Note: /prometheus/data is Prometheus's default TSDB storage path. If you've customized this in your config, replace it with your actual storage path.
• Copy the exported file from the pod to your local machine:

  kubectl cp <prometheus-pod-name>:/prometheus/metrics_dump.json ./metrics_dump.json -n <your-namespace>
  

Option 2: Continuous, Scheduled Export with a Sidecar Container #

For long-term, automated metric exports, a sidecar container is the way to go. This setup will regularly scrape Prometheus's /metrics endpoint and save the output to timestamped files.

Step 1: Create an Export Script #

First, make a simple shell script (export-metrics.sh) that handles the scraping and file writing:

#!/bin/sh
while true; do
  # Scrape Prometheus's local metrics endpoint and save to a timestamped file
  curl http://localhost:9090/metrics > /shared/metrics-$(date +%Y%m%d-%H%M%S).txt
  # Adjust the sleep interval to fit your needs (300 seconds = 5 minutes)
  sleep 300
done

Don't forget to make it executable: chmod +x export-metrics.sh

Step 2: Update Your Prometheus Deployment #

Modify your Prometheus Deployment YAML to add the sidecar container and a shared volume for storing exported files:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: <your-namespace>
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus:latest
        args:
          - --config.file=/etc/prometheus/prometheus.yml
          - --storage.tsdb.path=/prometheus/data
        volumeMounts:
          - name: prometheus-config
            mountPath: /etc/prometheus
          - name: metrics-storage
            mountPath: /prometheus/data
          # Optional: Mount the shared volume in the main Prometheus container if needed
          - name: shared-volume
            mountPath: /shared
      # Add the sidecar container for metric exports
      - name: metrics-exporter
        image: alpine/curl:latest
        command: ["/bin/sh", "/scripts/export-metrics.sh"]
        volumeMounts:
          - name: shared-volume
            mountPath: /shared
          - name: exporter-scripts
            mountPath: /scripts
      volumes:
        - name: prometheus-config
          configMap:
            name: prometheus-config
        - name: metrics-storage
          persistentVolumeClaim:
            claimName: prometheus-pvc
        # Shared volume to store exported metric files
        - name: shared-volume
          emptyDir: {}
        # ConfigMap to hold our export script
        - name: exporter-scripts
          configMap:
            name: exporter-scripts-config
            defaultMode: 0755

Step 3: Create the Script ConfigMap #

Push your script into a ConfigMap so the sidecar can access it:

kubectl create configmap exporter-scripts-config --from-file=export-metrics.sh -n <your-namespace>

Step 4: Apply the Updated Deployment #

kubectl apply -f prometheus-deployment.yaml -n <your-namespace>

Now the sidecar will automatically export metrics every 5 minutes (adjust the sleep value if you need a different interval). You can copy the files to your local machine using kubectl cp, or replace the emptyDir with a PersistentVolumeClaim if you need long-term storage.

Bonus: Filter Specific Metrics

If you don't need all metrics, you can filter the output with grep in the script:

curl http://localhost:9090/metrics | grep "node_cpu_seconds_total" > /shared/cpu-metrics-$(date +%Y%m%d-%H%M%S).txt

内容的提问来源于stack exchange，提问作者Mehmood ulhassan