VictoriaMetrics Agent对接Blackbox Exporter采集探针指标失败求助

尝试部署Prometheus Blackbox Exporter监控外部Web URL，已通过Helm完成安装：

helm install my -f ./values.yaml prometheus-community/prometheus-blackbox-exporter -n monitoring

对应的ConfigMap配置如下：

ConfigMap - my-prometheus-blackbox-exporter
apiVersion: v1
data:
  blackbox.yaml: |
    modules:
      http_2xx:
        http:
          follow_redirects: true
          preferred_ip_protocol: ip4
          tls_config:
            insecure_skip_verify: true
          valid_http_versions:
          - HTTP/1.1
          - HTTP/2.0
        prober: http
        timeout: 5s
      http_4xx:
        http:
          follow_redirects: true
          method: GET
          preferred_ip_protocol: ip4
          tls_config:
            insecure_skip_verify: true
          valid_http_versions:
          - HTTP/1.1
          - HTTP/2.0
          valid_status_codes:
          - 404
          - 405
        prober: http
        timeout: 5s
kind: ConfigMap

values.yaml仅修改了上述ConfigMap配置。

随后更新了VictoriaMetrics的additional-scrape-configs Secret以采集Blackbox的探针数据：

apiVersion: v1
kind: Secret
metadata:
  name: additional-scrape-configs
  namespace: monitoring
stringData:
  prometheus-additional.yaml: |
    - job_name: 'blackbox'
      metrics_path: /probe
      params:
        module: [http_2xx]
      static_configs:
       - targets:
          - https://myrlsek8s.my.local/grafana
      relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: my-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115

但VMAgent并未拉取到目标地址https://myrlsek8s.my.local/grafana的指标。已通过curl测试Blackbox功能正常：

curl -s "http://my-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115/probe?target=https://myrlsek8s.my.local/grafana&module=http_2xx"

尝试创建VMProbe仍未成功将探针指标导入VictoriaMetrics：

cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMProbe
metadata:
  name: probe-agent
  namespace: monitoring
spec:
  jobName: blackbox
  vmProberSpec:
     # by default scheme http, and path is /probe
     url: my-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
  module: http_2xx
  targets:
   staticConfig: 
      targets:
      - vmagent-my-vm-victoria-metrics-k8s-stack.monitoring.svc:8429/health
      - https://myrlsek8s.my.local/grafana
  interval: 2s 
EOF

注：未使用Prometheus，而是使用VictoriaMetrics Agent进行指标采集，期望探针指标能成功导入VictoriaMetrics。

排查与解决方案 #

针对additional-scrape-configs的问题排查 #

1. 确认VMAgent加载了额外采集配置

   • 查看VMAgent Pod日志，验证Secret是否被正确挂载解析：

     kubectl logs -n monitoring <vmagent-pod-name> | grep additional-scrape-configs
     

   • 若未加载，检查VMAgent部署配置是否包含--promscrape.config=/etc/prometheus/additional-scrape-configs/prometheus-additional.yaml参数，且Secret已挂载到对应路径。

2. 验证采集配置的relabel规则

   • 添加调试标签验证relabel流程，修改后的配置：

     - job_name: 'blackbox'
       metrics_path: /probe
       params:
         module: [http_2xx]
       static_configs:
        - targets:
           - https://myrlsek8s.my.local/grafana
       relabel_configs:
       - source_labels: [__address__]
         target_label: __param_target
       - source_labels: [__param_target]
         target_label: instance
       - target_label: __address__
         replacement: my-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
       # 调试用：保留所有元标签
       - action: labelmap
         regex: __meta_(.+)
     

   • 查看VMAgent的target状态，确认blackbox job是否正常：

     curl -s http://<vmagent-service>:8429/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job=="blackbox")'
     

针对VMProbe的问题排查 #

1. 修正VMProberSpec配置

   • vmProberSpec.url必须包含完整scheme，即使是http也不能省略，修改为：

     vmProberSpec:
       url: http://my-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
     

2. 检查VMProbe与VMAgent的关联

   • 查看VMAgent Pod日志，确认Operator是否注入了VMProbe配置：

     kubectl logs -n monitoring <vmagent-pod-name> | grep probe-agent
     

3. 验证目标地址的连通性

   • 在Blackbox或VMAgent Pod内测试目标地址的DNS解析与网络连通：

     kubectl exec -n monitoring <blackbox-pod-name> -- curl -I https://myrlsek8s.my.local/grafana
     

通用排查步骤 #

• 检查Blackbox Exporter日志，确认是否有探针请求记录：

  kubectl logs -n monitoring <blackbox-pod-name>
  

• 查询VictoriaMetrics中的探针指标，确认是否有数据：

  curl -s http://<victoriametrics-service>:8428/api/v1/query?query=probe_success
  

内容的提问来源于stack exchange，提问作者Abhi