Istio 1.8.5自定义指标重复：如何为istio_requests_total添加request_path标签

需求 #

用EnvoyFilter给istio_requests_total指标加上request.url_path和request.method维度信息，当前使用Istio 1.8.5版本。

遇到的问题 #

现有EnvoyFilter配置会生成带envoy_request_path__$(path-seperated-with-dashes)__前缀的重复指标，示例如下：

# TYPE envoy_request_path____catalog_lint___istio_requests_total counter
envoy_request_path____catalog_lint___istio_requests_total{response_code="200",reporter="destination",source_workload="my-service",source_workload_namespace="my-namespace",source_principal="unknown",source_app="my-service",source_version="unknown",destination_workload="api-linter",destination_workload_namespace="my-namespace",destination_principal="unknown",destination_app="my-service",destination_version="unknown",destination_service="my-service",destination_service_name="my-service",destination_service_namespace="my-namespace",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="none",source_canonical_service="my-service",destination_canonical_service="api-linter",source_canonical_revision="latest",destination_canonical_revision="latest",request_method="GET"} 4204

当前使用的EnvoyFilter配置 #

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: customize-requests-total-metric
spec:
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_OUTBOUND
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: istio.stats
        proxy:
          proxyVersion: ^1\.8.*
      patch:
        operation: REPLACE
        value:
          name: istio.stats
          typed_config:
            "@type": type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
            value:
              config:
                configuration:
                  "@type": type.googleapis.com/google.protobuf.StringValue
                  value: |
                    {
                      "debug": "true",
                      "stat_prefix": "istio",
                      "metrics": [
                        {
                          "name": "requests_total",
                          "dimensions": {
                            "request_path": "request.url_path",
                            "request_method": "request.method"
                          }
                        }
                      ]
                    }
                root_id: stats_outbound
                vm_config:
                  code:
                    local:
                      inline_string: envoy.wasm.stats
                  runtime: envoy.wasm.runtime.null
                  vm_id: stats_outbound
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: istio.stats
        proxy:
          proxyVersion: ^1\.8.*
      patch:
        operation: REPLACE
        value:
          name: istio.stats
          typed_config:
            "@type": type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
            value:
              config:
                configuration:
                  "@type": type.googleapis.com/google.protobuf.StringValue
                  value: |
                    {
                      "debug": "true",
                      "stat_prefix": "istio",
                      "metrics": [
                        {
                          "name": "requests_total",
                          "dimensions": {
                            "request_path": "request.url_path",
                            "request_method": "request.method"
                          }
                        }
                      ]
                    }
                root_id: stats_inbound
                vm_config:
                  code:
                    local:
                      inline_string: envoy.wasm.stats
                  runtime: envoy.wasm.runtime.null
                  vm_id: stats_inbound

问询 #

怎么把request_path像request_method一样，直接作为标签加到主istio_requests_total指标里？

在Wasm stats的配置里加上emit_tags_as_labels: true，就能让路径直接成为指标标签，不会再生成带前缀的重复指标。修改后的EnvoyFilter配置如下：

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: customize-requests-total-metric
spec:
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_OUTBOUND
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: istio.stats
        proxy:
          proxyVersion: ^1\.8.*
      patch:
        operation: REPLACE
        value:
          name: istio.stats
          typed_config:
            "@type": type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
            value:
              config:
                configuration:
                  "@type": type.googleapis.com/google.protobuf.StringValue
                  value: |
                    {
                      "debug": "true",
                      "stat_prefix": "istio",
                      "emit_tags_as_labels": true,
                      "metrics": [
                        {
                          "name": "requests_total",
                          "dimensions": {
                            "request_path": "request.url_path",
                            "request_method": "request.method"
                          }
                        }
                      ]
                    }
                root_id: stats_outbound
                vm_config:
                  code:
                    local:
                      inline_string: envoy.wasm.stats
                  runtime: envoy.wasm.runtime.null
                  vm_id: stats_outbound
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: istio.stats
        proxy:
          proxyVersion: ^1\.8.*
      patch:
        operation: REPLACE
        value:
          name: istio.stats
          typed_config:
            "@type": type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
            value:
              config:
                configuration:
                  "@type": type.googleapis.com/google.protobuf.StringValue
                  value: |
                    {
                      "debug": "true",
                      "stat_prefix": "istio",
                      "emit_tags_as_labels": true,
                      "metrics": [
                        {
                          "name": "requests_total",
                          "dimensions": {
                            "request_path": "request.url_path",
                            "request_method": "request.method"
                          }
                        }
                      ]
                    }
                root_id: stats_inbound
                vm_config:
                  code:
                    local:
                      inline_string: envoy.wasm.stats
                  runtime: envoy.wasm.runtime.null
                  vm_id: stats_inbound

关键修改说明 #

• 新增emit_tags_as_labels: true：这行配置会让Wasm过滤器把你定义的维度直接转为Prometheus标签，不再把路径拼到指标名称里，从根源避免重复指标。
• 原dimensions配置无需改动：request_path和request_method的映射关系保持原样，开启上述配置后它们会直接出现在主istio_requests_total指标的标签列表中。

修改后验证，你会看到类似这样的指标：

istio_requests_total{response_code="200",reporter="destination",request_path="/catalog/lint",request_method="GET",...} 4204

内容的提问来源于stack exchange，提问作者ezileli