HPA显示的外部指标值与API端点返回值不符

问题背景 #

我尝试通过Prometheus适配器导出的外部指标queues_idle_agent_count，结合HPA对Deployment进行扩缩容。调用外部指标API端点能获取到正确的空闲代理数量（值为7），但HPA中显示的当前指标值却为1，与实际值不符。

外部指标API验证结果 #

执行命令kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/default/queues_idle_agent_count | jq，返回结果正确：

{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metricName": "queues_idle_agent_count",
      "metricLabels": {},
      "timestamp": "2023-03-24T17:24:52Z",
      "value": "7"
    }
  ]
}

HPA状态异常表现 #

执行kubectl describe hpa得到以下输出，核心异常点为HPA显示的当前指标值为1（实际应为7）：

❯ k describe hpa                                                                                          
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name:                                                buildkite-agent
Namespace:                                           default
Labels:                                              <none>
Annotations:                                         <none>
CreationTimestamp:                                   Fri, 24 Mar 2023 13:20:02 -0400
Reference:                                           Deployment/buildkite-agent
Metrics:                                             ( current / target )
  "queues_idle_agent_count" (target average value):  1 / 2
Min replicas:                                        3
Max replicas:                                        25
Deployment pods:                                     7 current / 7 desired
Conditions:
  Type            Status  Reason               Message
  ----            ------  ------               -------
  AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendation
  ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from external metric queues_idle_agent_count(nil)
  ScalingLimited  False   DesiredWithinRange   the desired count is within the acceptable range
Events:
  Type    Reason             Age    From                       Message
  ----    ------             ----   ----                       -------
  Normal  SuccessfulRescale  2m45s  horizontal-pod-autoscaler  New size: 7; reason: All metrics below target

关键异常行：

Metrics:                                             ( current / target )
  "queues_idle_agent_count" (target average value):  1 / 2

HPA配置文件 #

当前使用的HPA配置如下（autoscaling/v2版本）：

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: buildkite-agent
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: buildkite-agent
  minReplicas: 3
  maxReplicas: 25
  metrics:
  - type: External
    external:
      metric:
        name: queues_idle_agent_count
      target:
        type: AverageValue
        averageValue: 2

排查方向与解决建议 #

1. 指标计算逻辑解析：当HPA使用AverageValue类型的外部指标时，Kubernetes会自动将外部指标的总值除以当前Pod副本数，得到单Pod平均指标值。这里实际指标值是7，当前Pod数是7，7/7=1，正好对应HPA显示的当前值——这说明HPA是按“指标总值/副本数”的逻辑计算平均值，而你可能期望直接使用指标原始值作为扩缩容依据。
2. 调整HPA目标类型：如果不需要按Pod平均分配指标，而是直接基于指标原始值进行扩缩，将target.type改为Value，并设置对应的target.value。修改后的配置示例：

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: buildkite-agent
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: buildkite-agent
  minReplicas: 3
  maxReplicas: 25
  metrics:
  - type: External
    external:
      metric:
        name: queues_idle_agent_count
      target:
        type: Value
        value: 2

3. 验证适配器配置：确认Prometheus适配器在导出指标时没有对值进行额外聚合或转换，确保指标原始值正确传递给Kubernetes API。

内容的提问来源于stack exchange，提问作者clbx