如何通过VictoriaAgent将K8s节点IP添加为Prometheus指标属性

问题背景 #

Kubernetes集群中运行node-exporter-prometheus-node-exporter和kube-state-metrics，通过Deployment部署的VictoriaAgent采集指标。查询kube_node_status_condition时，指标缺少节点IP属性，此前用OpenTelemetry Collector实现过该需求，现需通过VictoriaAgent配置补上该属性。

当前配置 #

ConfigMap #

apiVersion: v1
kind: ConfigMap
metadata:
  name: vmagent-config
  namespace: kube-system
data:
  vmagent.yaml: |
    global:
      scrape_interval: 10s
    scrape_configs:
      - job_name: "hke"
        scrape_interval: 30s
        metrics_path: /metrics
        relabel_configs:
        - target_label: k8s_node_ip
          source_labels: [__meta_kubernetes_node_address_InternalIP]
        - target_label: client_cluster
          replacement: "achebe-cluster"
        - target_label: client_email
          replacement: "admin@example.com"
        - source_labels: [__meta_kubernetes_node_name]
          target_label: instance
        static_configs:
          - targets:
            - "kube-state-metrics.kube-system.svc.cluster.local:8080"
          - targets:
            - "node-exporter-prometheus-node-exporter.kube-system.svc.cluster.local:9100"
      - job_name: "hke-cadvisor"
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor

Deployment #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vmagent
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vmagent
  template:
    metadata:
      labels:
        app: vmagent
    spec:
      containers:
        - name: vmagent
          image: victoriametrics/vmagent:v1.94.0
          args:
            - -promscrape.config=/etc/vmagent-config/vmagent.yaml
            - -remoteWrite.url=https://metrics.hostspacecloud.com/api/v1/write
          volumeMounts:
            - name: config-volume
              mountPath: /etc/vmagent-config
          ports:
            - containerPort: 8429
              name: web
      volumes:
        - name: config-volume
          configMap:
            name: vmagent-config

问题原因 #

当前hke Job使用static_configs直接指定Service地址，这种方式无法触发Kubernetes服务发现的元数据收集，__meta_kubernetes_node_address_InternalIP、__meta_kubernetes_node_name等节点元数据不存在，导致relabel规则无法生效，指标自然不会带上k8s_node_ip标签。

解决方案 #

通过Kubernetes节点服务发现获取节点元数据，结合relabel_configs和metric_relabel_configs将节点IP绑定到对应指标上。以下是两种可行的配置方案：

方案1：拆分Job独立处理 #

将原hke Job拆分为kube-state-metrics和node-exporter两个独立Job，分别配置标签规则：

apiVersion: v1
kind: ConfigMap
metadata:
  name: vmagent-config
  namespace: kube-system
data:
  vmagent.yaml: |
    global:
      scrape_interval: 10s
    scrape_configs:
      # 采集kube-state-metrics并绑定节点IP标签
      - job_name: "hke-kube-state-metrics"
        scrape_interval: 30s
        metrics_path: /metrics
        static_configs:
          - targets: ["kube-state-metrics.kube-system.svc.cluster.local:8080"]
        # 启用节点服务发现获取元数据
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - target_label: client_cluster
            replacement: "achebe-cluster"
          - target_label: client_email
            replacement: "admin@example.com"
        # 通过metric_relabel将节点IP匹配到带node标签的指标
        metric_relabel_configs:
          - source_labels: [node]
            target_label: __tmp_node
          - action: keep
            source_labels: [__tmp_node, __meta_kubernetes_node_name]
            regex: ^(.+);\1$
          - source_labels: [__meta_kubernetes_node_address_InternalIP]
            target_label: k8s_node_ip

      # 采集node-exporter并绑定节点IP标签
      - job_name: "hke-node-exporter"
        scrape_interval: 30s
        metrics_path: /metrics
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          # 生成每个节点的node-exporter采集地址
          - target_label: __address__
            source_labels: [__meta_kubernetes_node_address_InternalIP]
            replacement: "$1:9100"
          - target_label: k8s_node_ip
            source_labels: [__meta_kubernetes_node_address_InternalIP]
          - target_label: client_cluster
            replacement: "achebe-cluster"
          - target_label: client_email
            replacement: "admin@example.com"
          - target_label: instance
            source_labels: [__meta_kubernetes_node_name]

      # 为cadvisor指标添加节点IP标签
      - job_name: "hke-cadvisor"
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
          - target_label: k8s_node_ip
            source_labels: [__meta_kubernetes_node_address_InternalIP]

方案2：单Job混合采集 #

在原hke Job中启用节点服务发现，同时采集node-exporter和kube-state-metrics：

apiVersion: v1
kind: ConfigMap
metadata:
  name: vmagent-config
  namespace: kube-system
data:
  vmagent.yaml: |
    global:
      scrape_interval: 10s
    scrape_configs:
      - job_name: "hke"
        scrape_interval: 30s
        metrics_path: /metrics
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          # 生成node-exporter采集地址
          - target_label: __address__
            source_labels: [__meta_kubernetes_node_address_InternalIP]
            replacement: "$1:9100"
          # 添加kube-state-metrics采集地址
          - action: add
            target_label: __address__
            replacement: "kube-state-metrics.kube-system.svc.cluster.local:8080"
          - target_label: k8s_node_ip
            source_labels: [__meta_kubernetes_node_address_InternalIP]
          - target_label: client_cluster
            replacement: "achebe-cluster"
          - target_label: client_email
            replacement: "admin@example.com"
          - target_label: instance
            source_labels: [__meta_kubernetes_node_name]
        # 为kube-state-metrics的指标绑定节点IP
        metric_relabel_configs:
          - source_labels: [node]
            target_label: __tmp_node
          - action: keep
            source_labels: [__tmp_node, __meta_kubernetes_node_name]
            regex: ^(.+);\1$
          - source_labels: [__meta_kubernetes_node_address_InternalIP]
            target_label: k8s_node_ip

      - job_name: "hke-cadvisor"
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
          - target_label: k8s_node_ip
            source_labels: [__meta_kubernetes_node_address_InternalIP]

应用配置并验证 #

1. 应用修改后的ConfigMap：

   kubectl apply -f vmagent-config.yaml -n kube-system
   

2. 重启VictoriaAgent Deployment：

   kubectl rollout restart deployment vmagent -n kube-system
   

3. 等待Pod重启完成后，在VictoriaMetrics UI中查询kube_node_status_condition，即可看到指标带上了k8s_node_ip标签。

内容的提问来源于stack exchange，提问作者okechukwu achebe