Grafana+Prometheus无法展示Kubernetes节点历史数据问题排查

每日早晨均出现同一问题：Kubernetes节点的CPU、磁盘及内存图表仅从6:00/8:30开始展示，无法查看此前的历史数据。

我通过Terraform与Helm搭建监控环境，相关配置如下：

1. Prometheus初始配置 (prometheus-values.yaml) #

prometheus:
    serviceMonitorSelector: {}
    podMonitorSelector: {}
    additionalScrapeConfigs:
      - job_name: 'kubernetes-nodes'
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}/proxy/metrics
      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name]
            action: replace
            target_label: __metrics_path__
            replacement: /api/v1/namespaces/${1}/pods/${2}/proxy/metrics

2. Grafana监控查询语句 #

• CPU使用率：

  100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle", instance=~"$node"}[5m])) * 100)
  

• 磁盘使用率：

  sum by (instance) (node_filesystem_size_bytes{fstype=~"ext4|xfs", instance=~"$node"} - node_filesystem_free_bytes{fstype=~"ext4|xfs", instance=~"$node"})
  

• 可用内存：

  sum by (instance) (node_memory_MemAvailable_bytes{instance=~"$node"})
  

• 内存使用率：

  sum by (instance) (node_memory_MemTotal_bytes{instance=~"$node"} - node_memory_MemAvailable_bytes{instance=~"$node"})
  

3. 已尝试的修复配置（添加至prometheus-values.yaml） #

为解决问题，我在prometheus-values.yaml中添加了以下配置，但问题仍未解决：

prometheusSpec:
    retention: 7d
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi
    resources:
      requests:
        memory: "4Gi"
        cpu: "2"
      limits:
        memory: "8Gi"
        cpu: "4"

4. Terraform Helm部署配置 #

resource "helm_release" "prometheus" {
    name = "prometheus"
    repository = "https://charts.bitnami.com/bitnami"
    chart = "kube-prometheus"
    namespace = var.namespace

    set {
      name = "prometheus.service.type"
      value = "ClusterIP"
    }

    set {
      name = "prometheus.service.port"
      value = 9090
    }

    values = [
      file("${path.module}/files/prometheus-values.yaml")
    ]
}

resource "helm_release" "grafana" {
    name = "grafana"
    repository = "https://grafana.github.io/helm-charts"
    chart = "grafana"
    namespace = var.namespace

    set {
      name = "admiinUser"
      value = var.adminUser
    }

    set {
      name = "adminPassword"
      value = var.adminPassword
    }

    set {
      name    = "persistence.enabled"
      value   = true
    }

    set {
      name    = "persistence.size"
      value   = "10Gi"
    }

    set {
      name = "datasources.datasources\.yaml.apiVersion"
      value = 1
    }

    set {
      name = "datasources.datasources\.yaml.datasources[0].name"
      value = "Prometheus"
    }

    set {
      name = "datasources.datasources\.yaml.datasources[0].type"
      value = "prometheus"
    }

    set {
      name = "datasources.datasources\.yaml.datasources[0].url"
      value = "http://prometheus-operated.${var.namespace}.svc.cluster.local:9090"
    }

    set {
      name = "datasources.datasources\.yaml.datasources[0].access"
      value = "proxy"
    }

}

恳请协助排查该问题。

内容的提问来源于stack exchange，提问作者Sarah