如何在kube-system命名空间用Prometheus采集kubelet与metrics-server指标？

先处理当前metrics-server的采集错误 #

你遇到的lookup InternalIP on 10.96.128.2:53: no such host错误，是因为配置未指定metrics-server的完整集群内访问地址，或引用了未正确解析的变量。按以下方式修正：

1. 指定完整集群服务地址：metrics-server在kube-system命名空间下，集群内访问的完整域名是metrics-server.kube-system.svc:443
2. 跳过SSL证书验证：默认metrics-server使用自签证书，Prometheus需关闭证书校验才能正常访问
3. 配置认证：挂载Prometheus ServiceAccount的token，满足metrics-server的认证要求

修正后的metrics-server采集段：

- job_name: metrics-server
  scheme: https
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  static_configs:
    - targets: ["metrics-server.kube-system.svc:443"]

配置kubelet指标采集 #

kubelet指标推荐用自动发现机制，避免手动维护节点IP：

1. 节点自动发现：通过kubernetes_sd_configs让Prometheus自动识别所有集群节点
2. 指定kubelet端口与端点：kubelet默认在10250端口提供HTTPS服务，核心端点包括/metrics（基础节点指标）和/metrics/cadvisor（容器资源指标）
3. 认证与证书配置：同样挂载ServiceAccount token并跳过自签证书验证

kubelet采集配置段：

- job_name: kubelet-metrics
  scheme: https
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
    - role: node
  relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: "${1}:10250"
      regex: (__meta_kubernetes_node_address_InternalIP)
    - source_labels: [__meta_kubernetes_node_name]
      target_label: node_name

完整的Prometheus ConfigMap配置 #

替换你原有的ConfigMap，内容如下：

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus
  namespace: kube-system
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s  # 不建议设为1s，会给集群带来过高负载，15s为合理默认值
      evaluation_interval: 15s
    scrape_configs:
      # 采集metrics-server指标
      - job_name: metrics-server
        scheme: https
        tls_config:
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        static_configs:
          - targets: ["metrics-server.kube-system.svc:443"]
      # 采集kubelet基础指标
      - job_name: kubelet-metrics
        scheme: https
        tls_config:
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: "${1}:10250"
            regex: (__meta_kubernetes_node_address_InternalIP)
          - source_labels: [__meta_kubernetes_node_name]
            target_label: node_name
      # 采集kubelet容器资源指标（可选）
      - job_name: kubelet-cadvisor
        scheme: https
        tls_config:
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: "${1}:10250"
            regex: (__meta_kubernetes_node_address_InternalIP)
          - source_labels: [__meta_kubernetes_node_name]
            target_label: node_name
        metrics_path: /metrics/cadvisor

额外注意事项 #

• ServiceAccount权限：确保Prometheus使用的ServiceAccount绑定了cluster-monitoring类ClusterRole，至少需具备获取节点列表、访问kubelet metrics端点的权限
• 重启Pod生效：修改ConfigMap后，需重启Prometheus Pod使配置生效
• 负载控制：避免将scrape_interval设得过小，防止给集群和Prometheus自身带来过高负载

内容的提问来源于stack exchange，提问作者dmbibishkin