如何以2秒轮询间隔获取Kubernetes Pod实时指标并维护健康控制平面？

Great question—let’s break this down step by step since your use case (2-second interval Pod health checking to avoid routing traffic to laggy endpoints) doesn’t align with Metrics Server’s intended purpose. Let’s tackle each of your questions and land on the best fit for your needs.

1. How to Directly Use the Kubelet /metrics/resource Endpoint? #

You’re right that most examples rely on metrics.k8s.io (powered by Metrics Server), but accessing the Kubelet’s native endpoint is straightforward once you handle authentication and configuration:

• Endpoint Basics: Every Kubelet exposes /metrics/resource on port 10250 (HTTPS by default). This endpoint serves raw CPU/memory metrics for all Pods on the node, directly from the Kubelet’s internal calculations.
• Authentication: To access it, you’ll need valid Kubernetes credentials—either a ServiceAccount token with proper RBAC permissions, or a kubeconfig file with cluster admin access. For example, using curl with a ServiceAccount token:

  # Fetch the token from a mounted ServiceAccount secret
  TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
  # Query the local Kubelet's resource metrics endpoint
  curl -k -H "Authorization: Bearer $TOKEN" https://localhost:10250/metrics/resource
  

• Adjusting Collection Frequency: As you noted, the Kubelet’s default metric resolution is 60 seconds, but you can modify it with the --metric-resolution flag (e.g., --metric-resolution=15s). Critical Note: The official warning against setting values below 15s isn’t arbitrary—this is the minimum interval the Kubelet uses to compute accurate resource usage metrics. Setting it to 2s won’t yield fresher data than 15s, but it will unnecessarily increase Kubelet CPU/memory overhead.

2. Designing an Aggregation Layer for Direct Cgroup Data #

If you need strict 2-second latency (sub-15s), reading directly from /sys/fs/cgroup is a viable path. Here’s how to structure the aggregation layer:

Core Components #

• Node-Level Collector (DaemonSet): Deploy a lightweight collector as a DaemonSet so one instance runs on every node. Mount the host’s /sys/fs/cgroup directory into the collector Pod via a hostPath volume to access cgroup files.
• Pod-to-Cgroup Mapping: Kubernetes names Pod cgroups using the Pod’s UID (e.g., /sys/fs/cgroup/cpu/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod<UID>.slice). The collector can fetch the node’s Pod list via the Kubelet’s /pods endpoint or the Kubernetes API, then map each Pod to its corresponding cgroup path.
• Data Collection & Aggregation: Every 2 seconds, the collector reads metrics like cpuacct.usage (total CPU nanoseconds used) and memory.usage_in_bytes from each Pod’s cgroup directory. Compute real-time CPU usage by comparing the current cumulative value to the previous reading.
• Exposure for Control Plane: The collector can expose a local HTTP endpoint (e.g., port 8080) with the latest Pod metrics. Your control plane can either:
  • Query each node’s collector directly using a NodePort or ClusterIP service for the DaemonSet
  • Have the collector push metrics to a central low-latency cache (like Redis) for unified access

Key Considerations #

• Permissioning: Ensure the collector’s ServiceAccount has permissions to list Pods on the node, and the Pod runs with sufficient privileges to read host cgroup files (use securityContext.privileged: true or adjust host file permissions).
• Data Accuracy: cpuacct.usage is a cumulative counter—you’ll need to handle counter resets (e.g., when a Pod restarts) and compute delta values to get actual usage rates.

3. Other Feasible Solutions #

• eBPF-Based Collection: eBPF lets you hook into the kernel to collect Pod-level CPU/memory metrics with sub-second latency. Tools like BCC or Cilium have pre-built eBPF programs for this use case. The downside is higher development complexity if you need custom logic.
• Prometheus with Custom Scrape Configs: You can configure Prometheus to scrape Kubelet /metrics/resource every 15 seconds, or use a custom cgroup exporter with 2-second scrapes. However, Prometheus isn’t optimized for ultra-low-latency querying, so you’d need a separate cache layer for your control plane.

Best Recommended Solution #

Your choice depends on whether you can accept 15-second latency (aligned with Kubelet’s native metric resolution) or strictly need 2-second intervals:

1. For 15-Second Latency (Simplest & Most Stable):
   Use the Kubelet /metrics/resource endpoint with --metric-resolution=15s. Deploy a lightweight DaemonSet collector to pull metrics from each Kubelet every 15 seconds, cache them in a central store (like Redis), and expose them to your control plane. This leverages Kubernetes’ native metrics pipeline, requires minimal custom code, and avoids unnecessary resource overhead.

2. For Strict 2-Second Latency:
   Build the cgroup-based aggregation layer as outlined above. This gives you the lowest possible latency but requires more custom development and careful handling of cgroup mappings and metric calculations. If you want to avoid building from scratch, look for open-source tools like cgroup-exporter that handle node-level collection out of the box.

内容的提问来源于stack exchange，提问作者Raji