Keda基于Prometheus自动扩缩容失败：Client.Timeout超时问题求助

我尝试通过Prometheus数据实现KEDA自动扩缩容，但遇到了Client.Timeout exceeded while awaiting headers错误。

本地执行Prometheus查询正常：

http://localhost:9090/api/v1/query?query=sum%28process_cpu_seconds_total%29&time=2022-09-28T11:15:59Z

返回结果：

{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1664363399,"67.67"]}]}}

我的KEDA ScaledObject配置：

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: prometheus-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: example-deploy
  pollingInterval: 5
  cooldownPeriod: 10
  minReplicaCount: 1
  maxReplicaCount: 10

  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-main.monitoring.svc.cluster.local:9090
      metricName: process_cpu_seconds_total
      threshold: '100'
      activationThreshold: '50'
      query: sum(process_cpu_seconds_total)

KEDA Operator日志报错：

kubectl logs -n keda keda-operator-fbfcd6cd-n7q2t -c keda-operator

2022-09-28T11:30:00Z    ERROR   prometheus_scaler       error executing prometheus query        {"type": "ScaledObject", "namespace": "default", "name": "prometheus-scaledobject", "error": "Get \"http://prometheus-main.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28process_cpu_seconds_total%29&time=2022-09-28T11:29:57Z\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
        /workspace/pkg/scaling/cache/scalers_cache.go:94
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:149
2022-09-28T11:30:00Z    ERROR   scalehandler    Error getting scale decision    {"scaledobject.Name": "prometheus-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "example-deploy", "error": "Get \"http://prometheus-main.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28process_cpu_seconds_total%29&time=2022-09-28T11:29:57Z\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:149

HPA描述信息：

kubectl describe hpa keda-hpa-prometheus-scaledobject

Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name:                                                                keda-hpa-prometheus-scaledobject
Namespace:                                                           default
Labels:                                                              app.kubernetes.io/managed-by=keda-operator
                                                                     app.kubernetes.io/name=keda-hpa-prometheus-scaledobject
                                                                     app.kubernetes.io/part-of=prometheus-scaledobject
                                                                     app.kubernetes.io/version=2.8.0
                                                                     scaledobject.keda.sh/name=prometheus-scaledobject
Annotations:                                                         <none>
CreationTimestamp:                                                   Wed, 28 Sep 2022 13:29:48 +0200
Reference:                                                           Deployment/example-deploy
Metrics:                                                             ( current / target )
  "s0-prometheus-process_cpu_seconds_total" (target average value):  <unknown> / 100
Min replicas:                                                        1
Max replicas:                                                        10
Deployment pods:                                                     1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric default/s0-prometheus-process_cpu_seconds_total/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: prometheus-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get s0-prometheus-process_cpu_seconds_total.external.metrics.k8s.io)
Events:
  Type     Reason                        Age               From                       Message
  ----     ------                        ----              ----                       -------
  Warning  FailedGetExternalMetric       3s (x7 over 93s)  horizontal-pod-autoscaler  unable to get external metric default/s0-prometheus-process_cpu_seconds_total/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: prometheus-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get s0-prometheus-process_cpu_seconds_total.external.metrics.k8s.io)
  Warning  FailedComputeMetricsReplicas  3s (x7 over 93s)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get s0-prometheus-process_cpu_seconds_total external metric: unable to get external metric default/s0-prometheus-process_cpu_seconds_total/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: prometheus-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get s0-prometheus-process_cpu_seconds_total.external.metrics.k8s.io)

1. 验证KEDA到Prometheus的网络连通性 #

从KEDA Operator Pod内部测试访问Prometheus服务：

kubectl exec -n keda keda-operator-fbfcd6cd-n7q2t -c keda-operator -- curl -v http://prometheus-main.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28process_cpu_seconds_total%29

• 如果连不通，检查Prometheus服务的ClusterIP是否正确：kubectl get svc -n monitoring prometheus-main
• 检查KEDA所在keda命名空间和Prometheus所在monitoring命名空间的网络策略，确认没有阻止跨namespace的Pod访问规则。

2. 调整超时与轮询配置 #

• 在ScaledObject的Prometheus触发器中添加timeout参数，延长查询超时时间：

  triggers:
  - type: prometheus
    metadata:
      # 保留原有参数
      timeout: "30s"
  

• 调大pollingInterval（当前为5秒），避免频繁查询给Prometheus造成压力：

  spec:
    pollingInterval: 15
  

3. 优化Prometheus查询与负载 #

• 优化查询语句：sum(process_cpu_seconds_total)是累计值，不适合作为扩缩容阈值，改用增长率更合理，同时提升查询效率：

  query: sum(rate(process_cpu_seconds_total[1m]))
  

• 检查Prometheus Pod的资源负载：kubectl top pod -n monitoring prometheus-main-xxx
• 通过Prometheus UI查看prometheus_http_request_duration_seconds指标，确认查询请求的响应时间是否过长。

4. 调整KEDA Operator资源配额 #

如果KEDA Operator本身资源不足，会导致请求处理超时：

• 查看当前资源配置：kubectl describe deployment keda-operator -n keda
• 调整资源请求与限制：

  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
    limits:
      cpu: "500m"
      memory: "512Mi"
  

5. 检查Prometheus认证与端口配置 #

• 确认Prometheus Service的9090端口正常暴露，端口名称匹配请求要求
• 如果Prometheus启用了认证，在ScaledObject中添加认证信息：

  triggers:
  - type: prometheus
    metadata:
      # 保留原有参数
      username: PROMETHEUS_USERNAME
      password: PROMETHEUS_PASSWORD
    authenticationRef:
      name: prometheus-auth-secret
  

  对应的Secret需创建在KEDA所在的keda命名空间或ScaledObject的default命名空间。

内容的提问来源于stack exchange，提问作者Nimander