Pod高负载下Heapster停止采集CPU指标问题求助

Hey there, let’s dig into this tricky issue where your CPU metrics disappear during load testing, even though memory metrics stay intact. Based on your setup (K8s 1.8.4 on AWS via kops, Heapster 1.5.0 with InfluxDB sink, 60s collection interval), here’s a step-by-step breakdown of what to check and fix:

1. Check if Heapster is resource-starved #

Since CPU metrics drop off under load but memory doesn’t, Heapster might be struggling to process CPU data fast enough.

• Monitor Heapster’s own resource usage during load testing:

  kubectl top pod -n kube-system | grep heapster
  

  If its CPU usage is hitting the request/limit ceiling, it can’t keep up with processing incoming CPU metrics, leading to lost data or unresponsive HPA queries.
• Dig into Heapster’s logs for clues like timeouts, queue overflows, or InfluxDB write failures:

  kubectl logs -n kube-system <your-heapster-pod-name> -f
  

  Look for errors related to CPU metric processing or InfluxDB sink issues—these will point directly to where the bottleneck is.

2. Verify InfluxDB’s write capacity #

Even though metrics bounce back right after load ends, InfluxDB might be unable to handle the spike in CPU metric writes during testing:

• Check InfluxDB’s resource usage while load is applied:

  kubectl top pod -n kube-system | grep influxdb
  

  If its CPU is maxed out, it’s rejecting or delaying writes from Heapster, which causes CPU metrics to get dropped.
• Check InfluxDB logs for write errors or latency warnings:

  kubectl logs -n kube-system <your-influxdb-pod-name> -f
  

  CPU metrics often generate more frequent data points than memory, so they’re more likely to hit write bottlenecks.

3. Tune Heapster’s configuration for better load handling #

Heapster 1.5.0 has configurable parameters you can adjust to handle higher load:

• Increase parallelism: By default, Heapster uses 1 worker thread to process metrics. Bump this up with the --parallelism flag (e.g., --parallelism=2) in your Heapster Deployment to let it process multiple node metrics at once.
• Optimize InfluxDB batch writes: Adjust the sink’s batch settings to reduce the number of write requests to InfluxDB. Add flags like:

  --sink.influxdb.batch-size=1000 --sink.influxdb.batch-timeout=10s
  

  This makes Heapster collect more metrics before sending a batch, easing pressure on InfluxDB.

4. Align HPA’s query interval with Heapster’s collection interval #

Kubernetes 1.8’s HPA defaults to querying metrics every 30s, while your Heapster collects data every 60s. This mismatch can lead to HPA asking for metrics that haven’t been collected/written yet during load.

• Adjust the kube-controller-manager’s sync period: Modify the --horizontal-pod-autoscaler-sync-period flag to 60s so HPA only queries metrics once Heapster has had time to collect and write them.

5. Consider version updates (if feasible) #

Both K8s 1.8.4 and Heapster 1.5.0 are quite old, and there might be known bugs around metric handling under load that’ve been fixed in newer versions:

• Try upgrading Heapster to the latest patch in the 1.5.x line, or a newer stable release like 1.6.x (just make sure to back up your InfluxDB data first to avoid loss).

6. Quick temporary fix: Boost resource allocations #

If you need a stopgap while working on long-term fixes, increase the CPU and memory requests/limits for Heapster and InfluxDB Pods:

• For Heapster, bump CPU requests from the default 100m to 200m, and limits to 500m; raise memory requests from 200Mi to 300Mi.
• Do the same for InfluxDB if its resource usage is spiking during load.

内容的提问来源于stack exchange，提问作者whites11