如何为System.Diagnostics.Metrics与OpenTelemetry添加全局标签

问题背景 #

需要为服务中所有上报的指标全局添加dc（数据中心）和env（环境）标签，避免开发人员手动为每个测量添加这些维度。尝试通过ConfigureResource添加属性，但Prometheus端点输出的指标中并未包含这些标签。

当前Prometheus输出 #

# TYPE click_c_clicks_total counter
# UNIT click_c_clicks_total clicks_total
# HELP click_c_clicks_total count the number of clicks
click_c_clicks_total{key="a"} 14 1682246108134
click_c_clicks_total{key="s"} 15 1682246108134
click_c_clicks_total{key="d"} 14 1682246108134
click_c_clicks_total{key="e"} 2 1682246108134
click_c_clicks_total{key="r"} 2 1682246108134
click_c_clicks_total{key="w"} 1 1682246108134

# TYPE click_t_ms histogram
# UNIT click_t_ms ms
# HELP click_t_ms time to click after prompted
click_t_ms_bucket{le="0"} 0 1682246108134
click_t_ms_bucket{le="5"} 0 1682246108134
click_t_ms_bucket{le="10"} 0 1682246108134
click_t_ms_bucket{le="25"} 0 1682246108134
click_t_ms_bucket{le="50"} 0 1682246108134
click_t_ms_bucket{le="75"} 17 1682246108134
click_t_ms_bucket{le="100"} 24 1682246108134
click_t_ms_bucket{le="250"} 42 1682246108134
click_t_ms_bucket{le="500"} 44 1682246108134
click_t_ms_bucket{le="750"} 47 1682246108134
click_t_ms_bucket{le="1000"} 47 1682246108134
click_t_ms_bucket{le="2500"} 47 1682246108134
click_t_ms_bucket{le="5000"} 47 1682246108134
click_t_ms_bucket{le="7500"} 47 1682246108134
click_t_ms_bucket{le="10000"} 47 1682246108134
click_t_ms_bucket{le="+Inf"} 48 1682246108134
click_t_ms_sum 29085 1682246108134
click_t_ms_count 48 1682246108134

# EOF

示例代码 #

var host = Host
    .CreateDefaultBuilder()
    .ConfigureLogging(logger => logger.AddConsole())
    .ConfigureServices(services =>
    {
        services.AddOpenTelemetry()
            .WithMetrics(metricsBuilder =>
            {
                metricsBuilder.ConfigureResource(resourceBuilder =>
                {
                    resourceBuilder
                        .AddService("ClickService")
                        .AddAttributes(new[]
                        {
                            new KeyValuePair<string, object>("dc", Environment.GetEnvironmentVariable("dc") ?? "us1"),
                            new KeyValuePair<string, object>("env", Environment.GetEnvironmentVariable("env") ?? "local"),
                        })
                        .AddTelemetrySdk();
                });
                var meterProviderBuilder = metricsBuilder.AddMeter("custom-console-meter");

                metricsBuilder.AddConsoleExporter();

                metricsBuilder.AddPrometheusHttpListener(httpListenerOptions =>
                {
                    httpListenerOptions.UriPrefixes = new[] { "http://localhost:9464/" };
                });
            });

        services.AddHostedService<ClickAppHost>();
    });

await host.RunConsoleAsync();

ClickAppHost代码 #

// 一个简单的循环应用，直到按下CTRL-C停止
// 统计按键点击次数
public class ClickAppHost : IHostedService
{
    private readonly CancellationTokenSource _work = new();

    public Task StartAsync(CancellationToken cancellationToken)
    {
        Console.CancelKeyPress += (sender, args) =>
        {
            Console.WriteLine("收到取消请求");
            _work.Cancel();
        };

        while (!_work.Token.IsCancellationRequested)
        {
            Console.WriteLine("按下任意按键");
            var sw = Stopwatch.GetTimestamp();
            
            var key = Console.ReadKey(intercept: true);
            
            var elapsed = Stopwatch.GetElapsedTime(sw);
            var combination = GetKeyLabel(key);
            Console.WriteLine($"已点击: {combination}");

            Meters.ClickInstrument.Add(1,
                new KeyValuePair<string, object?>("key", combination));
            Meters.DurationInstrumentation.Record((long)elapsed.TotalMilliseconds);
        }

        return Task.CompletedTask;

        static string GetKeyLabel(ConsoleKeyInfo keyInfo) 
            => keyInfo.Modifiers != ConsoleModifiers.None 
                ? $"{keyInfo.Modifiers}-{keyInfo.KeyChar}" 
                : keyInfo.KeyChar.ToString();
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        _work.Cancel();

        return Task.CompletedTask;
    }
}

public static class Meters
{
    private static readonly Meter _applicationMeter = new("custom-console-meter", "1.0.0");

    public static readonly Counter<int> ClickInstrument 
        = _applicationMeter.CreateCounter<int>("click_c", "clicks_total", "统计点击次数");

    public static readonly Histogram<long> DurationInstrumentation 
        = _applicationMeter.CreateHistogram<long>("click_t", "ms", "统计提示后到点击的耗时");
}

依赖项 #

<PackageReference Include="Microsoft.Extensions.Hosting" Version="8.0.0-preview.3.23174.8" />
    <PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.5.0-alpha.2" />
    <PackageReference Include="OpenTelemetry.Exporter.Prometheus.HttpListener" Version="1.5.0-alpha.1" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.5.0-alpha.2" />

解决方案 #

核心原因 #

OpenTelemetry的Resource属性默认不会自动转换为Prometheus的指标标签，需要通过指标处理器或指标转换逻辑手动注入全局标签。

实现思路 #

以下是两种可行方案：

方案1：使用TransformMetrics快速实现（OTEL 1.4+支持）

修改WithMetrics配置，添加全局标签转换逻辑：

services.AddOpenTelemetry()
    .WithMetrics(metricsBuilder =>
    {
        // 保留原有Resource配置（用于其他导出器或上下文）
        metricsBuilder.ConfigureResource(resourceBuilder =>
        {
            resourceBuilder
                .AddService("ClickService")
                .AddAttributes(new[]
                {
                    new KeyValuePair<string, object>("dc", Environment.GetEnvironmentVariable("dc") ?? "us1"),
                    new KeyValuePair<string, object>("env", Environment.GetEnvironmentVariable("env") ?? "local"),
                })
                .AddTelemetrySdk();
        });

        // 添加全局标签转换
        metricsBuilder.TransformMetrics(metric =>
        {
            var dc = Environment.GetEnvironmentVariable("dc") ?? "us1";
            var env = Environment.GetEnvironmentVariable("env") ?? "local";
            return metric
                .AddTag("dc", dc)
                .AddTag("env", env);
        });

        metricsBuilder.AddMeter("custom-console-meter");
        metricsBuilder.AddConsoleExporter();
        metricsBuilder.AddPrometheusHttpListener(httpListenerOptions =>
        {
            httpListenerOptions.UriPrefixes = new[] { "http://localhost:9464/" };
        });
    });

方案2：自定义MetricProcessor（兼容旧版本）

如果使用的OTEL版本不支持TransformMetrics，可以自定义处理器：

public class GlobalLabelInjectorProcessor : MetricProcessor
{
    private readonly string _dc;
    private readonly string _env;

    public GlobalLabelInjectorProcessor()
    {
        _dc = Environment.GetEnvironmentVariable("dc") ?? "us1";
        _env = Environment.GetEnvironmentVariable("env") ?? "local";
    }

    public override void OnEnd(Metric metric)
    {
        // 为每个指标添加全局标签
        metric.AddTag("dc", _dc);
        metric.AddTag("env", _env);
        base.OnEnd(metric);
    }
}

然后在配置中添加处理器：

metricsBuilder.AddProcessor(new GlobalLabelInjectorProcessor());

效果验证 #

修改后，Prometheus输出的指标会自动带上dc和env标签，示例如下：

click_c_clicks_total{key="a",dc="us1",env="local"} 14 1682246108134

内容的提问来源于stack exchange，提问作者Eugene Krapivin