通过Helm部署Prometheus后，如何为CockroachDB配置insecure_skip_verify？

通过Prometheus社区的Helm Chart部署Prometheus到Kubernetes集群后，发现CockroachDB监控目标处于下线状态，报错信息如下：

Get "https://10.244.0.48:8080/_status/vars": x509: certificate is valid for 127.0.0.1, not 10.244.0.48

CockroachDB服务带有以下注解：

prometheus.io/path: _status/vars
prometheus.io/port: '8080'
prometheus.io/scrape: 'true'

需要为CockroachDB监控目标设置insecure_skip_verify以跳过证书验证。

针对上述问题，提供三种可行的配置方式：

方法1：添加专门的CockroachDB Scrape Job #

修改Prometheus的Helm values配置文件，新增一个针对CockroachDB的scrape任务，直接配置证书跳过验证：

prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
    - job_name: 'cockroachdb-monitoring'
      kubernetes_sd_configs:
      - role: service
      # 筛选带有Prometheus监控注解的CockroachDB服务
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: 'true'
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - source_labels: [__meta_kubernetes_service_name]
        action: keep
        regex: cockroachdb # 替换为你的CockroachDB服务名称
      # 启用证书跳过验证
      tls_config:
        insecure_skip_verify: true

执行Helm升级命令应用配置：

helm upgrade prometheus prometheus-community/prometheus -f your-values.yaml -n monitoring

方法2：修改默认Service Scrape任务的Relabel规则 #

在默认的kubernetes-services scrape任务中添加relabel规则，为CockroachDB目标自动注入跳过证书验证的标识：

prometheus:
  prometheusSpec:
    scrapeConfigs:
    - job_name: 'kubernetes-services'
      kubernetes_sd_configs:
      - role: service
      relabel_configs:
      # 新增规则：匹配CockroachDB服务并设置跳过验证
      - source_labels: [__meta_kubernetes_service_name]
        regex: cockroachdb
        action: replace
        target_label: __insecure_skip_verify__
        replacement: "true"
      # 保留原有基于服务注解的scrape规则
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

执行同样的Helm升级命令使配置生效。

方法3：通过ServiceMonitor配置（推荐） #

如果集群使用ServiceMonitor管理监控目标，创建或修改CockroachDB的ServiceMonitor资源，直接配置证书跳过验证：

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cockroachdb
  namespace: monitoring # 替换为Prometheus所在命名空间
spec:
  selector:
    matchLabels:
      app: cockroachdb # 替换为CockroachDB服务的标签
  endpoints:
  - port: http # 替换为CockroachDB暴露监控指标的端口名称
    path: /_status/vars
    tlsConfig:
      insecureSkipVerify: true

同时确保Prometheus的Helm配置能发现该ServiceMonitor：

prometheus:
  prometheusSpec:
    serviceMonitorSelector:
      matchLabels:
        app: cockroachdb
    serviceMonitorNamespaceSelector:
      matchNames:
      - monitoring

应用ServiceMonitor配置：

kubectl apply -f cockroachdb-servicemonitor.yaml

再执行Helm升级命令更新Prometheus配置。

内容的提问来源于stack exchange，提问作者hobyte