如何将Kubernetes指标推送至Prometheus？跨EKS集群采集方案咨询

Hey there! I've worked through similar scenarios with external Prometheus scraping EKS metrics, so let's cover both your desired pull-based path and push-based alternatives clearly.

You don't have to run Prometheus inside the EKS cluster to scrape metrics—you just need to properly expose your metric endpoints via Ingress and configure your external Prometheus to target them. Here's how to make it work:

1. Configure Ingress to Expose Metric Services #

First, you'll need to route traffic from your Ingress Controller to the cluster's metric pods (like kube-state-metrics, node-exporter, or custom metric workloads). Create an Ingress resource with path mappings to each metric service, and add security layers to prevent unauthorized access.

Example Ingress YAML:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: eks-metrics-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: metrics-basic-auth
spec:
  tls:
  - hosts:
    - metrics.your-company.com
    secretName: metrics-tls-cert
  rules:
  - host: metrics.your-company.com
    http:
      paths:
      - path: /kube-state-metrics
        pathType: Prefix
        backend:
          service:
            name: kube-state-metrics
            port:
              number: 8080
      - path: /node-exporter
        pathType: Prefix
        backend:
          service:
            name: node-exporter
            port:
              number: 9100

• Key Notes: Use TLS for encrypted traffic and Basic Auth (via a Kubernetes Secret) to restrict access. Adjust paths and service names to match your cluster's metric services.

2. Update External Prometheus Scrape Configs #

Modify your external Prometheus's prometheus.yml to add scrape jobs targeting the exposed Ingress endpoints. Make sure to match the paths, auth, and TLS settings from your Ingress configuration.

Example scrape jobs:

scrape_configs:
  - job_name: 'eks-kube-state-metrics'
    scrape_interval: 30s
    scheme: https
    basic_auth:
      username: prom-scraper
      password: your-secure-password
    static_configs:
      - targets: ['metrics.your-company.com']
    metrics_path: '/kube-state-metrics/metrics'

  - job_name: 'eks-node-exporter'
    scrape_interval: 30s
    scheme: https
    basic_auth:
      username: prom-scraper
      password: your-secure-password
    static_configs:
      - targets: ['metrics.your-company.com']
    metrics_path: '/node-exporter/metrics'

• TLS Tip: If you're using a self-signed cert, add tls_config: insecure_skip_verify: true (avoid this in production—use a trusted CA cert instead).

3. Verify Endpoint Reachability #

• Check that your Ingress is properly addressed: kubectl get ingress eks-metrics-ingress
• Test access from the external Prometheus host: curl -u prom-scraper:your-secure-password https://metrics.your-company.com/kube-state-metrics/metrics
• Ensure EKS security groups allow incoming traffic to the Ingress Controller's service port (usually 443 for HTTPS).

If pull-based isn't ideal for your setup, there are two reliable push-based approaches:

1. Use Prometheus Pushgateway #

Pushgateway is an official component that acts as a middleman for metrics pushed by jobs. It's great for short-lived workloads, but can also work for long-running services (note: it retains metrics until manually cleared, so it's not a perfect replacement for pull-based for permanent services).

• Deploy Pushgateway in EKS (or expose an external Pushgateway if network allows)
• Configure your metric pods/sidecars to push metrics to the gateway. For example, a simple curl command to push custom metrics:

  curl -X POST http://pushgateway.your-company.com:9091/metrics/job/eks-custom-metrics --data-binary @metrics-output.txt
  

• Have your external Prometheus scrape the Pushgateway by adding this job to prometheus.yml:

  - job_name: 'eks-pushgateway'
    scrape_interval: 30s
    static_configs:
      - targets: ['pushgateway.your-company.com:9091']
  

2. Remote Write from an Internal "Relay" Prometheus #

For a more robust push-like flow (that preserves Prometheus's native time-series handling), deploy a lightweight Prometheus inside EKS that scrapes all cluster metrics, then uses the Remote Write API to send them to your external Prometheus.

• Configure the internal Prometheus's remote_write section:

  remote_write:
    - url: 'https://external-prom.your-company.com/api/v1/write'
      basic_auth:
        username: remote-write-user
        password: your-write-password
      tls_config:
        insecure_skip_verify: false
  

• This approach maintains all of Prometheus's scraping logic (like service discovery, relabeling) while forwarding metrics to your external instance.

• Security First: Never expose metric endpoints without auth and TLS—metrics often contain sensitive cluster data (node resources, pod names, etc.).
• Network Checks: Ensure your external Prometheus can reach the EKS Ingress endpoint (verify VPC peering, security groups, and DNS resolution).
• Performance Monitoring: Keep an eye on Ingress Controller throughput and Prometheus scrape success rates to catch bottlenecks early.

内容的提问来源于stack exchange，提问作者Vishal Patil