Prometheus远程写入中缺失Kube State Metrics的问题排查

背景与配置 #

我正尝试通过Kube State Metrics结合Prometheus获取Pod的运行时长，核心需要两个指标：

• kube_pod_created（Pod启动时间）
• kube_pod_completion_time（Pod完成/停止时间）

Prometheus采集配置（config.yml） #

global:
    scrape_interval: 10m
    scrape_timeout: 10s
    evaluation_interval: 10m
scrape_configs:
    - job_name: kubernetes-nodes-cadvisor
      honor_timestamps: true
      scrape_interval: 10m
      scrape_timeout: 10s
      metrics_path: /metrics
      scheme: https
      authorization:
          type: Bearer
          credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
      follow_redirects: true
      enable_http2: true
      relabel_configs:
          - separator: ;
            regex: __meta_kubernetes_node_label_(.+)
            replacement: $1
            action: labelmap
          - separator: ;
            regex: (.*)
            target_label: __address__
            replacement: kubernetes.default.svc:443
            action: replace
          - source_labels: [__meta_kubernetes_node_name]
            separator: ;
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
            action: replace
      metric_relabel_configs:
          - source_labels: [__name__]
            regex: '(container_cpu_usage_seconds_total|container_fs_reads_bytes_total|container_fs_writes_bytes_total|container_memory_max_usage_bytes|container_network_receive_bytes_total|container_network_transmit_bytes_total)'
            action: keep
      kubernetes_sd_configs:
          - role: node
            kubeconfig_file: ''
            follow_redirects: true
            enable_http2: true
    - job_name: 'kube-state-metrics'
      scrape_interval: 10m
      static_configs:
        - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']
      metric_relabel_configs:
        - source_labels: [__name__]
          regex: '(kube_pod_labels|kube_pod_created|kube_pod_completion_time|kube_pod_container_resource_limits)'
          action: keep
remote_write:
    - url: http://example.com
      remote_timeout: 30s
      follow_redirects: true
      enable_http2: true
      oauth2:
          token_url: https://example.com
          client_id: myCoolID
          client_secret: myCoolPassword
      queue_config:
          capacity: 2500
          max_shards: 200
          min_shards: 1
          max_samples_per_send: 10
          batch_send_deadline: 5s
          min_backoff: 30ms
          max_backoff: 5s
      metadata_config:
          send: false

测试用Pod Deployment #

---
apiVersion: apps/v1
kind: Deployment
metadata:
    name: busy-box-test
spec:
    replicas: 1
    selector:
        matchLabels:
            app: busy-box-test
    template:
        metadata:
            labels:
                app: busy-box-test
        spec:
            containers:
                - command:
                      - sleep
                      - '300'
                  image: busybox
                  name: test-box

当前问题 #

远程写入源中能找到kube_pod_labels、kube_pod_created等其他指定指标，但找不到kube_pod_completion_time指标。

尝试用以下命令在集群内查询该指标，但未得到有效结果，怀疑命令路径错误：

• kubectl get --raw '/metrics' | grep kube_
• kubectl get --raw 'kube-state-metrics.kube-system.svc.cluster.local:8080'

疑问 #

1. 集群内是否有可返回Pod完成时间的端点？
2. 对于每5分钟启停一次的Pod，10分钟的采集间隔是否存在问题？
3. Kube State Metrics中保留已终止Pod历史记录的时长是多少？

集群环境：AWS EKS 1.22版本，Kube State Metrics配置已按指定内容部署。

内容的提问来源于stack exchange，提问作者Daniel Wasserlauf