Kubernetes集群中Prometheus监控Golang WebApp Pod时2112端点异常排查求助

Let's break down what's going wrong here and walk through actionable fixes and checks to get your monitoring up and running.

1. You've Added Prometheus Annotations to the Wrong Service #

The core mistake here is that you applied the prometheus.io/* annotations to the Prometheus Service (prometheus-service) instead of the Golang WebApp Service (goapp). These annotations are meant to signal to Prometheus, "this is a service you should scrape for metrics" — so they belong on the target service you want to monitor, not the Prometheus service itself.

Fix: Update the goapp Service Annotations #

Modify your goapp Service definition to include the required monitoring annotations:

apiVersion: v1
kind: Service
metadata:
  namespace: monitoring
  name: goapp
  annotations:
    prometheus.io/scrape: 'true'  # Instruct Prometheus to scrape this service
    prometheus.io/path: '/metrics'  # Path to your app's metrics endpoint
    prometheus.io/port: '2112'  # Port where your Golang app exposes metrics
spec:
  type: NodePort
  selector:
    app: golang
  ports:
    - name: main
      protocol: TCP
      port: 80
      targetPort: 2112
      nodePort: 30001

2. Verify Prometheus is Configured for Kubernetes Service Discovery #

Even with the correct annotations, Prometheus won't find your service unless its configuration includes Kubernetes service discovery with relabel rules that honor these annotations.

Check Your Prometheus ConfigMap (prometheus-server-conf) #

Ensure your prometheus.yml has a scrape job like this to enable annotation-based service discovery:

scrape_configs:
  # Existing jobs (e.g., scraping Prometheus itself) can stay here

  - job_name: 'kubernetes-services'
    kubernetes_sd_configs:
      - role: service
    relabel_configs:
      # Keep only services marked for scraping
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      # Use the custom metrics path if specified
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      # Use the custom port if specified
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2

After updating the ConfigMap, restart your Prometheus Pod to apply changes:

kubectl rollout restart deployment prometheus-deployment -n monitoring

3. Step-by-Step Debugging Checks #

If the endpoint still shows down, run these checks to narrow down the root cause:

• Test metrics endpoint accessibility from within the cluster:
  Exec into your Prometheus Pod and curl the Golang app's metrics endpoint directly:

  kubectl exec -it <prometheus-pod-name> -n monitoring -- curl http://goapp.monitoring.svc.cluster.local:2112/metrics
  

  • A 404 response means your Golang app isn't exposing metrics at /metrics, or the port is incorrect.
  • A connection timeout points to a network issue (check Service selectors, Pod readiness status, or cluster network policies).

• Check Prometheus Targets for Error Details:
  Open the Prometheus UI (via your NodePort 30000) → Status > Targets. Look for the goapp target and check the "Error" column — it will tell you exactly why the scrape is failing (e.g., context deadline exceeded, 404 Not Found).

• Validate Service and Pod Selector Matching:
  Ensure your goapp Service's spec.selector matches the labels on your Golang Pods:

  # Check Service selector
  kubectl describe service goapp -n monitoring | grep Selector
  # Check Pod labels
  kubectl get pods -n monitoring -l app=golang --show-labels
  

Note on Your Proposed Fix #

Adding prometheus.io/port: '2112' to the Prometheus Deployment won't resolve this issue — that annotation is used to scrape metrics from the Prometheus server itself, not your Golang app. The fix needs to target the app's Service and Prometheus's service discovery configuration instead.

内容的提问来源于stack exchange，提问作者allen