GKE中HPA无法识别Prometheus/Stackdriver Adapter的untyped自定义指标

已完成操作 #

• 启用GKE托管式Prometheus
• 在JVM上部署JMX Exporter并配置导出目标指标
• 部署Stackdriver自定义指标适配器

此前基于JVM堆内存指标的HPA可正常扩缩容，配置如下：

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-autoscale
  namespace: somenamespace
spec:
  maxReplicas: 3
  metrics:
  - pods:
      metric:
        name: prometheus.googleapis.com|jvm_memory_bytes_used|gauge
        selector:
           matchLabels:
             metric.labels.area: heap
      target:
        averageValue: 2G
        type: AverageValue
  type: Pods
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-java-app

需求 #

希望HPA基于java_lang_threading_threadcount指标进行扩缩容，该指标类型为untyped：

# HELP java_lang_threading_threadcount java.lang:name=null,type=Threading,attribute=ThreadCount
# TYPE java_lang_threading_threadcount untyped
java_lang_threading_threadcount 123.0

尝试的操作 #

根据Stackdriver Adapter文档指引，从Metrics Explorer获取指标全名prometheus.googleapis.com/java_lang_threading_threadcount/unknown，并配置到HPA中：

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-autoscale
  namespace: somenamespace
spec:
  maxReplicas: 3
  metrics:
  - pods:
      metric:
        name: prometheus.googleapis.com|java_lang_threading_threadcount|unknown
        selector:
           matchLabels:
             metric.labels.area: heap
      target:
        averageValue: 200
        type: AverageValue
    type: Pods
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-java-app

问题 #

配置后HPA报错，提示无法找到该指标：

$ k get hpa -o yaml my-autoscale  | grep unable -A7
    message: 'the HPA was unable to compute the replica count: unable to get metric
      prometheus.googleapis.com|java_lang_threading_threadcount|unknown: unable to
      fetch metrics from custom metrics API: googleapi: Error 404: Cannot find metric(s)
      that match type = "prometheus.googleapis.com/java_lang_threading_threadcount/unknown"
      label = area label = pod. If a metric was created recently, it could take up
      to 10 minutes to become available. Please try again soon., notFound'
    reason: FailedGetPodsMetric
    status: "False"
    type: ScalingActive

正确的指标名称与配置 #

对于untyped类型的Prometheus指标，Stackdriver Adapter需要使用untyped作为类型标识，而非Metrics Explorer显示的unknown。同时注意，java_lang_threading_threadcount本身没有area: heap标签，需删除错误的selector配置。

正确的HPA配置如下：

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-autoscale
  namespace: somenamespace
spec:
  maxReplicas: 3
  metrics:
  - pods:
      metric:
        name: prometheus.googleapis.com|java_lang_threading_threadcount|untyped
      target:
        averageValue: 200
        type: AverageValue
    type: Pods
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-java-app

原因说明 #

1. Stackdriver Adapter对Prometheus指标的命名规则为prometheus.googleapis.com|<指标名>|<原指标类型>，untyped类型指标需用untyped作为后缀，而非Metrics Explorer展示的unknown。
2. 原配置中保留的metric.labels.area: heap selector与目标指标不匹配，会导致指标筛选失败，必须移除。

内容的提问来源于stack exchange，提问作者brandizzi