基于Prometheus与Kubernetes的Spring Boot应用HPA扩缩容指标咨询

Great question! Scaling a stable Spring Boot app based on traffic growth using Prometheus and HPA is a common use case, and picking the right metrics is key to making it reliable. Let's break down the metrics you should prioritize, plus core considerations for integrating Prometheus with Kubernetes scaling.

Metrics for HPA (Horizontal Pod Scaling) #

You'll want to combine resource-based metrics (Kubernetes native) and custom Spring Boot metrics (application-specific) to cover both infrastructure and service health.

1. Resource Utilization Metrics (Kubernetes + Prometheus) #

These are foundational for ensuring your pods aren't stretched beyond their resource limits:

• CPU Usage: Use container_cpu_usage_seconds_total (calculate the rate over time to get percentage usage) or kube_pod_container_resource_usage_cpu_cores. This is the most common starting point—set a threshold like 70% CPU usage to trigger scaling.
• Memory Usage: container_memory_working_set_bytes or kube_pod_container_resource_usage_memory_bytes. Track memory utilization (e.g., 80% of allocated memory) to avoid OOM kills that could take down pods before scaling kicks in.

2. Spring Boot Application-Specific Metrics #

These metrics directly reflect how your application is handling traffic, making them ideal for traffic-driven scaling:

• HTTP Request QPS: http_server_requests_seconds_count (filter by endpoint/method if needed, e.g., http_server_requests_seconds_count{endpoint="/api/checkout"}). If your core API's QPS exceeds a threshold (like 100 requests/sec per pod), scale out to handle the load.
• Request Latency: Use http_server_requests_seconds_bucket to calculate percentiles (P95/P99) or http_server_requests_seconds_sum divided by http_server_requests_seconds_count for average latency. If P95 latency crosses 500ms, scaling can help reduce queueing and improve user experience.
• JVM Health:
  • jvm_memory_used_bytes{area="heap"}: Track heap memory usage (e.g., 75% of max heap) to catch garbage collection pressure or memory leaks that might impact performance.
  • jvm_threads_active: Monitor active threads—if your app is hitting thread limits (common in Tomcat or custom thread pools), scaling adds more thread capacity.
• Thread Pool Usage: For Tomcat apps, use tomcat_threads_current; for custom thread pools, expose metrics via Spring Actuator. If active threads reach 80% of the pool size, scaling prevents request backlogs.

Core Metrics to Monitor for Prometheus-Kubernetes Scaling Integration #

Beyond HPA-specific metrics, these ensure your scaling pipeline works reliably:

• Cluster Resource Availability: kube_node_allocatable_cpu_cores and kube_node_allocatable_memory_bytes minus current usage. You need to make sure your cluster has enough spare resources to schedule new pods—otherwise, scaled-out pods will stay in Pending state.
• Pod Readiness: kube_pod_ready (value 1 means ready). HPA waits for pods to be ready before counting them towards replica count, but monitoring this ensures your new pods are actually healthy and serving traffic.
• Metric Pipeline Latency: Keep an eye on Prometheus scrape intervals (default 15s) and HPA sync intervals (default 15s). If metrics take too long to propagate, HPA decisions will lag behind real traffic changes—adjust these settings if you need faster responsiveness.
• Error Rate: http_server_requests_seconds_count{status=~"5.."} (5xx errors). A sudden spike in errors might mean scaling isn't the fix (e.g., a broken dependency), but if errors are tied to high traffic (like timeouts), combining error rate with QPS can help trigger smarter scaling.

Quick Tips for Implementation #

• Combine multiple metrics (e.g., CPU + QPS) to avoid false positives—don't rely on a single metric alone.
• Tune HPA parameters: Set minReplicas/maxReplicas to match your capacity needs, and use behavior rules to control scaling speed (e.g., scale up quickly but scale down slowly to avoid traffic jitter).
• Build a Grafana dashboard that visualizes all these metrics together—this makes it easy to validate when scaling triggers and how well it's working.

内容的提问来源于stack exchange，提问作者Anson