Nginx指标未在Prometheus中显示的问题求助

我用MicroK8s部署了单节点Kubernetes集群，采用默认Nginx Ingress Controller与Prometheus插件，希望将Nginx指标采集至Prometheus以便后续在Grafana中监控。创建ServiceMonitor后，Prometheus虽能检测到抓取池，但无法找到关联端点，已确认Nginx指标导出正常，相关配置如下：

ServiceMonitor配置 #

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-prom-stack-nginx-ingress-microk8s
  namespace: observability
  annotations:
    meta.helm.sh/release-name: kube-prom-stack
    meta.helm.sh/release-namespace: observability
  labels:
    app.kubernetes.io/component: metrics
    app.kubernetes.io/instance: kube-prom-stack
    app.kubernetes.io/name: nginx-ingress-microk8s
    app.kubernetes.io/part-of: nginx-ingress-microk8s
    release: kube-prom-stack
spec:
  selector:
    matchLabels:
      microk8s-application: nginx-ingress-microk8s
  endpoints:
    - honorLabels: true
      enableHttp2: true
      port: health
      path: /metrics
      interval: 5s
      scheme: http
  namespaceSelector:
    matchNames:
      - ingress
      - observability
      - kube-system
      - default

生成的Prometheus Job配置 #

- job_name: serviceMonitor/observability/kube-prom-stack-nginx-ingress-microk8s/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_microk8s_application, __meta_kubernetes_service_labelpresent_microk8s_application]
    separator: ;
    regex: (nginx-ingress-microk8s);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: health
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: health
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: \"0\"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: \"\"
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - ingress
      - observability
      - kube-system
      - default

排查与解决建议 #

• 检查Service标签匹配：执行kubectl get svc -n ingress -l microk8s-application=nginx-ingress-microk8s，确认Nginx Ingress对应的Service是否带有microk8s-application: nginx-ingress-microk8s标签。无结果则需给Service添加对应标签，或修改ServiceMonitor的selector为Service实际标签。
• 确认Service端口配置：执行kubectl get svc <nginx-ingress-svc-name> -n ingress -o yaml，查看spec.ports中是否存在name: health的端口，且其targetPort指向的容器端口确实暴露/metrics路径。
• 核对Namespace范围：执行kubectl get svc -A | grep nginx-ingress，确认Nginx Ingress Service所在命名空间是否在ServiceMonitor的namespaceSelector.matchNames列表中，不在则添加该命名空间。
• 检查Prometheus的ServiceMonitor选择器：执行kubectl get prometheus kube-prom-stack-prometheus -n observability -o yaml，查看spec.serviceMonitorSelector.matchLabels是否包含release: kube-prom-stack（你的ServiceMonitor带有该标签），不匹配则调整Prometheus选择器或给ServiceMonitor添加对应标签。
• 查看Prometheus服务发现日志：执行kubectl logs -n observability <prometheus-pod-name> | grep "serviceMonitor/observability/kube-prom-stack-nginx-ingress-microk8s"，从日志中定位标签不匹配、端口不存在等具体问题。

内容的提问来源于stack exchange，提问作者FrenchTechLead