Istio 1.21.4升级后无法移除Prometheus特定标签与指标

问题背景 #

之前通过IstioOperator配置中的telemetry.v2.prometheus.configOverride可以移除Prometheus指标标签，但升级至Istio 1.21.4后该配置提示configOverride无效。改用Telemetry资源配置后，仍无法移除指定标签，需排查是否遗漏配置项。

原IstioOperator配置 #

---
apiVersion: install.istio.io/v1alpha1 
kind: IstioOperator
metadata:
  <some metadata>
spec:
  <some specifications>
  values:
    global:
      # 确保Istio Pod仅调度到Linux节点
      autoscalingv2API: true
      defaultNodeSelector:
        beta.kubernetes.io/os: linux
      proxy:
        resources:
          requests:
            cpu: 10m
            memory: 40Mi     
        autoInject: "disabled"
      imagePullPolicy: "IfNotPresent"
      imagePullSecrets:
      <some secrets>
    sidecarInjectorWebhook:
    <some values>
    telemetry:
      v2:
        prometheus:
          configOverride:
            inboundSidecar:
              debug: false
              stat_prefix: istio
              metrics:
                # 移除所有Istio指标的这些标签
                - tags_to_remove: ["array of tags"]
                # 移除部分指标的响应标签
                - name: request_duration_milliseconds
                  tags_to_remove: ["response_code", "response_flags"]
                - name: request_bytes
                  tags_to_remove: ["response_code", "response_flags"]
                - name: response_bytes
                  tags_to_remove: ["response_code", "response_flags"]
            outboundSidecar:
              debug: false
              stat_prefix: istio
              metrics:
                # 移除所有Istio指标的这些标签
                - tags_to_remove: ["array of tags"]
                # 移除部分指标的响应标签
                - name: request_duration_milliseconds
                  tags_to_remove: ["response_code", "response_flags"]
                - name: request_bytes
                  tags_to_remove: ["response_code", "response_flags"]
                - name: response_bytes
                  tags_to_remove: ["response_code", "response_flags"]
            gateway:
              debug: false
              stat_prefix: istio
              disable_host_header_fallback: true
              metrics:
                # 移除所有Istio指标的这些标签
                - tags_to_remove: ["array of tags"]
                # 移除部分指标的响应标签
                - name: request_duration_milliseconds
                  tags_to_remove: ["response_code", "response_flags"]
                - name: request_bytes
                  tags_to_remove: ["response_code", "response_flags"]
                - name: response_bytes
                  tags_to_remove: ["response_code", "response_flags"]

改用的Telemetry配置 #

---
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: custom-telemetry-outbound
  namespace: istio-system
spec:
  metrics:
    - providers:
        - name: prometheus
          overrides:
            # 应用于所有出站Sidecar
            - match:
                mode: CLIENT_AND_SERVER
              tagOverrides:
                destination_canonical_service: 
                  operation: REMOVE
                source_canonical_service: 
                  operation: REMOVE
                destination_principal: 
                  operation: REMOVE
                source_principal: 
                  operation: REMOVE
                connection_security_policy:
                  operation: REMOVE
                grpc_response_status: 
                  operation: REMOVE
                source_version: 
                  operation: REMOVE
                destination_version:
                  operation: REMOVE
                request_protocol: 
                  operation: REMOVE
                source_canonical_revision: 
                  operation: REMOVE
                destination_canonical_revision: 
                  operation: REMOVE

            # 移除特定指标的响应标签
            - match:
                metric: REQUEST_DURATION
                mode: CLIENT_AND_SERVER
              tagOverrides:
                response_code: 
                  operation: REMOVE
                response_flags: 
                  operation: REMOVE

            - match:
                metric: REQUEST_SIZE
                mode: CLIENT_AND_SERVER
              tagOverrides:
                response_code: 
                  operation: REMOVE
                response_flags: 
                  operation: REMOVE

            - match:
                metric: RESPONSE_SIZE
                mode: CLIENT_AND_SERVER
              tagOverrides:
                response_code: 
                  operation: REMOVE
                response_flags: 
                  operation: REMOVE

问题分析与修正方案 #

1. 覆盖范围缺失网关场景 #

原IstioOperator配置同时覆盖了Sidecar（入站/出站）和Gateway，但新的Telemetry配置未明确针对Gateway设置匹配规则。需添加单独的Gateway匹配配置，或拆分Telemetry资源分别处理：

针对Sidecar的Telemetry配置

---
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: custom-telemetry-sidecar
  namespace: istio-system
spec:
  workloadSelector: {} # 匹配所有Sidecar
  metrics:
    - providers:
        - name: prometheus
          overrides:
            - match:
                mode: CLIENT_AND_SERVER
              tagOverrides:
                destination_canonical_service: { operation: REMOVE }
                source_canonical_service: { operation: REMOVE }
                destination_principal: { operation: REMOVE }
                source_principal: { operation: REMOVE }
                # 其他需移除的全局标签...
            - match:
                metric: REQUEST_DURATION
                mode: CLIENT_AND_SERVER
              tagOverrides:
                response_code: { operation: REMOVE }
                response_flags: { operation: REMOVE }
            - match:
                metric: REQUEST_SIZE
                mode: CLIENT_AND_SERVER
              tagOverrides:
                response_code: { operation: REMOVE }
                response_flags: { operation: REMOVE }
            - match:
                metric: RESPONSE_SIZE
                mode: CLIENT_AND_SERVER
              tagOverrides:
                response_code: { operation: REMOVE }
                response_flags: { operation: REMOVE }

针对Gateway的Telemetry配置

---
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: custom-telemetry-gateway
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway # 根据实际网关标签调整
  metrics:
    - providers:
        - name: prometheus
          overrides:
            - match:
                mode: GATEWAY
              tagOverrides:
                destination_canonical_service: { operation: REMOVE }
                source_canonical_service: { operation: REMOVE }
                # 其他需移除的全局标签...
            - match:
                metric: REQUEST_DURATION
                mode: GATEWAY
              tagOverrides:
                response_code: { operation: REMOVE }
                response_flags: { operation: REMOVE }
            - match:
                metric: REQUEST_SIZE
                mode: GATEWAY
              tagOverrides:
                response_code: { operation: REMOVE }
                response_flags: { operation: REMOVE }
            - match:
                metric: RESPONSE_SIZE
                mode: GATEWAY
              tagOverrides:
                response_code: { operation: REMOVE }
                response_flags: { operation: REMOVE }

2. 验证配置生效 #

• 应用配置后，重启相关Sidecar和Gateway Pod：

  kubectl rollout restart deployment <sidecar-deployment> -n <namespace>
  kubectl rollout restart deployment istio-ingressgateway -n istio-system
  

• 检查Proxy配置是否加载新规则：

  istioctl proxy-config bootstrap <pod-name> -n <namespace> | grep -A 10 -B 5 "tagOverrides"
  

• 查询Prometheus指标确认标签是否移除，例如：

  istio_request_duration_milliseconds_bucket
  

内容的提问来源于stack exchange，提问作者0ptimus