启用TLS后GKE中Google托管Prometheus监控失败问题

在GKE上运行应用，启用Google托管Prometheus（GMP）后原本监控正常。给应用启用外部通信TLS并重启后，监控服务出现证书验证失败的错误。按照GMP文档使用PodMonitor配置，但未找到对应的TLS配置项——仅Alert Manager和Prometheus Operator有相关配置，而GKE自行运行Operator，只需创建PodMonitor指向应用即可。

应用侧报错信息 #

2024-03-01 13:07:03,833 ERROR [prometheus-metrics-server-0] HOST: Failed to process request.
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.ReferenceCountedOpenSslEngine$OpenSslHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE

当前PodMonitor配置 #

# see https://github.com/GoogleCloudPlatform/prometheus-engine/blob/v0.7.0/doc/api.md  for more fields
# if we use TLS, we need to configure a TLSConfig
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: google-prom
  labels:
    examplelabel: test123
    examplelabel2: test1234
spec:
  selector:
    matchLabels:
      name: voltdb-cluster
      #app.kubernetes.io/name: voltdb-cluster
  endpoints:
  # this should be >= the interval that we create the metrics for
  - interval: 10s
    # the default is /metrics
    path: /metrics
    port: 11781
    scheme: https
    metricRelabeling:
    - action: drop
      regex: connection_.*
      sourceLabels: [__name__]
  targetLabels:
    metadata:
    - node
    - container
    - pod
    fromPod:
    - from: clustertype
      to: clustertype
    - from: testname
      to: testname
    - from: testid
      to: testid
    - from: branch
      to: branch

解决方案 #

GMP的PodMonitoring资源支持在endpoints字段中配置tlsConfig，用于处理HTTPS采集的证书验证问题，以下是几种常见场景的配置方式：

1. 临时禁用证书验证（仅测试环境） #

如果是自签名证书导致的验证失败，可临时跳过验证（生产环境不推荐）：

spec:
  endpoints:
  - interval: 10s
    path: /metrics
    port: 11781
    scheme: https
    tlsConfig:
      insecureSkipVerify: true  # 禁用证书验证
    metricRelabeling:
    - action: drop
      regex: connection_.*
      sourceLabels: [__name__]
  # 其他原有配置保持不变...

2. 使用自定义CA证书验证 #

如果应用使用自己的CA签发证书，需要将CA证书存入Kubernetes Secret，然后在PodMonitor中引用：

步骤1：创建CA证书Secret

kubectl create secret generic metrics-ca --from-file=ca.crt=/path/to/your/ca.crt

步骤2：修改PodMonitor配置

spec:
  endpoints:
  - interval: 10s
    path: /metrics
    port: 11781
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/secrets/metrics-ca/ca.crt  # 证书挂载路径
    secretMounts:  # 挂载Secret到采集器容器
    - name: metrics-ca
      mountPath: /etc/prometheus/secrets/metrics-ca
      readOnly: true
    metricRelabeling:
    - action: drop
      regex: connection_.*
      sourceLabels: [__name__]
  # 其他原有配置保持不变...

3. 客户端证书认证（双向TLS） #

如果应用要求采集端提供客户端证书，需将客户端证书、密钥和CA证书存入Secret，再配置PodMonitor：

步骤1：创建客户端证书Secret

kubectl create secret generic metrics-client --from-file=ca.crt=/path/to/ca.crt --from-file=tls.crt=/path/to/client.crt --from-file=tls.key=/path/to/client.key

步骤2：修改PodMonitor配置

spec:
  endpoints:
  - interval: 10s
    path: /metrics
    port: 11781
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/secrets/metrics-client/ca.crt
      certFile: /etc/prometheus/secrets/metrics-client/tls.crt
      keyFile: /etc/prometheus/secrets/metrics-client/tls.key
    secretMounts:
    - name: metrics-client
      mountPath: /etc/prometheus/secrets/metrics-client
      readOnly: true
    metricRelabeling:
    - action: drop
      regex: connection_.*
      sourceLabels: [__name__]
  # 其他原有配置保持不变...

内容的提问来源于stack exchange，提问作者Kajal Jadeja