K8s HPA设置minReplicas=3仍报TooFewReplicas问题排查
问题背景
创建如下HPA配置,预期关联的Deployment启动时默认副本数为3:
apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: backend-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: backend minReplicas: 3 maxReplicas: 20 metrics: - type: Resource resource: name: memory target: type: AverageValue averageValue: 500Mi - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 behavior: scaleDown: stabilizationWindowSeconds: 300 policies: - type: Pods value: 1 periodSeconds: 100 scaleUp: stabilizationWindowSeconds: 60 policies: - type: Pods value: 1 periodSeconds: 30 - type: Percent value: 10 periodSeconds: 60 selectPolicy: Max
HPA持续返回如下状态,无法理解提示原因:
ScalingLimited True TooFewReplicas the desired replica count is less than the minimum replica count
相关kubectl查询结果如下:
➜ kg hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE prod-backend-hpa-v1 Deployment/prod-backend-v1 145881770666m/500Mi, 2%/70% 3 20 3 7m16s ➜ kd hpa Name: prod-backend-hpa-v1 Namespace: prod Labels: argocd.argoproj.io/instance=backend-prod Annotations: <none> CreationTimestamp: Thu, 02 Jun 2022 19:34:30 -0500 Reference: Deployment/prod-backend-v1 Metrics: ( current / target ) resource memory on pods: 145596416 / 500Mi resource cpu on pods (as a percentage of request): 1% (3m) / 70% Min replicas: 3 Max replicas: 20 Behavior: Scale Up: Stabilization Window: 60 seconds Select Policy: Max Policies: - Type: Pods Value: 1 Period: 30 seconds - Type: Percent Value: 10 Period: 60 seconds Scale Down: Stabilization Window: 300 seconds Select Policy: Max Policies: - Type: Pods Value: 1 Period: 100 seconds Deployment pods: 3 current / 3 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ReadyForNewScale recommended size matches current size ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource ScalingLimited True TooFewReplicas the desired replica count is less than the minimum replica count Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 9m22s horizontal-pod-autoscaler New size: 3; reason: Current number of replicas below Spec.MinReplicas ➜ kgpo NAME READY STATUS RESTARTS AGE prod-backend-v1-8dd687999-54mzp 1/1 Running 0 58m prod-backend-v1-8dd687999-nn7c2 1/1 Running 0 2d17h prod-backend-v1-8dd687999-rcxsw 1/1 Running 0 2d17h ➜ kg rs NAME DESIRED CURRENT READY AGE prod-backend-v1-566b9c8856 0 0 0 2d19h prod-backend-v1-578d699c45 0 0 0 2d19h prod-backend-v1-64859b74c9 0 0 0 2d18h prod-backend-v1-6498b4b45c 0 0 0 2d19h prod-backend-v1-656cccdc4b 0 0 0 2d19h prod-backend-v1-66cc5cf44 0 0 0 2d19h prod-backend-v1-698c7ddc7d 0 0 0 2d19h prod-backend-v1-6bdbc77f5d 0 0 0 2d19h prod-backend-v1-7486c95664 0 0 0 2d19h prod-backend-v1-774cdbdcdc 0 0 0 2d19h prod-backend-v1-8dd687999 3 3 3 2d17
原因说明
这个ScalingLimited不是配置错误,是HPA的正常状态提示,触发原因是内存指标配置和实际预期不匹配:
- 给内存指标配置的target类型是
AverageValue,目标值是单Pod平均内存500Mi,HPA会按期望副本数 = 所有Pod内存占用总和 / 单Pod目标内存值计算需要的副本数。 - 从查询结果可以看到,当前单Pod实际内存占用只有约139Mi(145596416字节),3个Pod总内存约417Mi,算出来期望副本数不到1,远小于设置的
minReplicas=3。 - HPA受最小副本数限制不会继续缩容到计算值,就会返回
TooFewReplicas的提示,意思是当前计算出的期望副本数小于配置的最小副本数,伸缩被阈值限制。 - 看到的内存指标
145881770666m/500Mi是单位换算展示问题,AverageValue模式下内存以字节的千分之一(m)为单位展示,139Mi换算后正好是这个数值,远低于500Mi的目标。
处理方案
- 如果预期是按Pod内存request的使用率伸缩,把内存指标的target类型从
AverageValue改成Utilization,配置averageUtilization字段填写目标使用率百分比即可,比如填70代表平均内存达到request的70%时触发扩容。 - 如果确实需要按内存绝对值做伸缩阈值,这个提示不需要额外处理,等业务负载升高,HPA计算出的期望副本数大于等于3时,这个状态会自动消失。
内容的提问来源于stack exchange,提问作者DmitrySemenov
相关产品推荐
相关产品推荐

