.Net 6 API中OpenTelemetry指标无法显示在Prometheus问题排查

在.Net 6.0 API项目中使用Open Telemetry Protocol（OTLP）配置指标，这些指标能被OTLP接收器接收，但无法在Prometheus导出器中显示。其他来自OTLP目标的追踪和指标能正常在Prometheus中看到，只有应用自身产生的指标无法进入Prometheus（但能进入OTLP）。

相关配置代码 #

Startup.cs #

services.AddOpenTelemetryMetrics(opts => opts
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("otel-collector"))
    .AddMeter(metrics.MetricName)
    .AddAspNetCoreInstrumentation()
    .AddConsoleExporter()
    .AddOtlpExporter()
);

指标埋点类 #

public class OtelMetrics
{
    private Counter<int> MoveToLoadPositionCounter { get; }
    private Counter<int> MoveToSetupPositionCounter { get; }        
    private Histogram<double> MoveToPositionDelayHistogram { get; }
    public string MetricName { get; }

    public OtelMetrics(string meterName = "RealTimeChannelMeter")
    {
        var meter = new Meter(meterName);
        MetricName = meterName;

        MoveToLoadPositionCounter = meter.CreateCounter<int>("move_to_load", "Movement");
        MoveToSetupPositionCounter = meter.CreateCounter<int>("move_to_setup", "Movement");
        MoveToPositionDelayHistogram = meter.CreateHistogram<double>("move_to_position_delay", "Ms", "Move to position latency.");
    }
    
    public void AddToMoveToLoad() => MoveToLoadPositionCounter.Add(1);
    public void AddToMoveToSetup() => MoveToSetupPositionCounter.Add(1);
    public void MoveToPositionDelay(double delayMs) => MoveToPositionDelayHistogram.Record(delayMs);
}

触发指标的接口代码 #

[HttpPost]
[Route("Load")]
[ApiConventionMethod(typeof(DefaultApiConventions), nameof(DefaultApiConventions.Post))]
public async Task<ActionResult<MoveToPositionResponse>> LoadAsync(MoveToPositionRequest request)
{
    _metrics.AddToMoveToLoad();

    var watch = Stopwatch.StartNew();
    await Task.Delay(5000);
    watch.Stop();

    _metrics.MoveToPositionDelay(watch.ElapsedMilliseconds);

    return CreateActionResult(MovementResult.SUCCESS);
}

OTLP配置文件（config.yaml） #

receivers:
  otlp:
    protocols:
      http:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8888"

processors:
  batch:

extensions:
  health_check:

service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

OTLP启动命令 #

docker run -d -p 8888:8888 -p 8889:8889 -p 13133:13133 -p 4317:4317 -p 4318:4318 -v ./config.yaml:/etc/otel-collector-config.yamll otel/opentelemetry-collector:0.60.0

Prometheus配置文件（prometheus.yml） #

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'RealTimeChannelMeter'
    scrape_interval: 5s
    static_configs:
      - targets: ['host.docker.internal:8889']
      - targets: ['host.docker.internal:8888']

Prometheus Dockerfile #

FROM prom/prometheus:v2.38.0

ADD prometheus.yml /etc/prometheus/

Prometheus启动命令 #

docker build -t prometheus2 . 
docker run -d -p 9000:9000 prometheus2

日志对比 #

能正常进入Prometheus的OTLP指标日志 #

Resource SchemaURL: 
Resource attributes:
     -> service.name: STRING(otel-collector)
     -> service.instance.id: STRING(0.0.0.0:8888)
     -> net.host.port: STRING(8888)
     -> http.scheme: STRING(http)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope  
Metric #0
Descriptor:
     -> Name: otelcol_exporter_enqueue_failed_metric_points
     -> Description: Number of metric points failed to be added to the sending queue.
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
NumberDataPoints #0
Data point attributes:
     -> exporter: STRING(logging)
     -> service_instance_id: STRING(32709c55-4140-4bb7-98c3-94a09b557f4c)
     -> service_version: STRING(0.60.0)
StartTimestamp: 2023-02-23 16:24:28.395 +0000 UTC
Timestamp: 2023-02-23 16:24:28.395 +0000 UTC
Value: 0.000000

能进入OTLP但无法显示在Prometheus的应用指标日志 #

Resource SchemaURL: 
Resource attributes:
     -> service.name: STRING(otel-collector)
     -> service.instance.id: STRING(1e10dd35-13c6-4d1d-9f42-4a82611924da)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope RealTimeChannelMeter 
Metric #0
Descriptor:
     -> Name: move-to-load
     -> Description: 
     -> Unit: Movement
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
NumberDataPoints #0
StartTimestamp: 2023-02-23 16:07:04.4076488 +0000 UTC
Timestamp: 2023-02-23 16:08:04.3291068 +0000 UTC
Value: 2

怀疑问题与net.host.port和http.scheme属性有关，不知道如何在.Net应用中配置这些属性，希望得到排查建议。

1. 修正Prometheus的Scrape配置 #

Prometheus配置中同时配置了8889和8888两个target，但OTLP的Prometheus exporter仅监听8888，8889是OTLP自身的指标端口。去掉多余target，确保只抓取正确端口：

scrape_configs:
  - job_name: 'RealTimeChannelMeter'
    scrape_interval: 5s
    static_configs:
      - targets: ['host.docker.internal:8888']

2. 检查指标命名规范 #

Prometheus要求指标名称仅包含字母、数字、下划线和冒号，你的指标日志中显示为move-to-load（含连字符），旧版OTLP Collector可能存在转换兼容问题。建议创建指标时直接使用下划线命名（你的代码已符合，但需确保OTLP版本支持正确转换）。

3. 为.Net应用添加缺失的资源属性 #

若需手动添加net.host.port和http.scheme，可在ResourceBuilder中配置：

services.AddOpenTelemetryMetrics(opts => opts
    .SetResourceBuilder(ResourceBuilder.CreateDefault()
        .AddService("otel-collector")
        .AddAttributes(new Dictionary<string, object>
        {
            ["net.host.port"] = 5000, // 替换为你的API实际端口
            ["http.scheme"] = "http"
        }))
    .AddMeter(metrics.MetricName)
    .AddAspNetCoreInstrumentation()
    .AddConsoleExporter()
    .AddOtlpExporter()
);

4. 升级OTLP Collector版本 #

你使用的0.60.0版本较旧，存在Prometheus exporter兼容性问题。建议升级到最新稳定版：

docker run -d -p 8888:8888 -p 4317:4317 -p 4318:4318 -v ./config.yaml:/etc/otel-collector-config.yaml otel/opentelemetry-collector:latest

5. 验证OTLP Pipeline的指标流转 #

添加logging exporter，确认应用指标是否被正确发送到Prometheus exporter：

exporters:
  prometheus:
    endpoint: "0.0.0.0:8888"
  logging:
    loglevel: debug

service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus, logging]

查看OTLP容器日志，确认应用指标是否进入Prometheus exporter环节。

内容的提问来源于stack exchange，提问作者qhyseni