Kubernetes属性未在OpenTelemetry指标中显示问题求助

在Kubernetes项目中使用OpenTelemetry Collector采集并导出指标，已通过Go语言配置指标提供者，利用环境变量设置K8S_NODE_NAME、K8S_NAMESPACE等Kubernetes专属属性，但这些属性并未出现在导出至Prometheus的指标中。

Go语言指标提供者简化配置 #

func initMeterProvider(serviceName string) (*sdkmetric.MeterProvider, error) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

otelCollectorAddr := getEnv("OTEL_COLLECTOR_ADDR", "otel-collector-opentelemetry-collector.telemetry.svc.cluster.local:4317")
//otelCollectorAddr := getEnv("OTEL_COLLECTOR_ADDR", "otel-collector-lb.telemetry.svc.cluster.local:4317")

// Set up the gRPC connection to the OpenTelemetry Collector
conn, err := grpc.DialContext(ctx, otelCollectorAddr, grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithBlock())
if err != nil {
    return nil, fmt.Errorf("failed to connect to OpenTelemetry Collector: %w", err)
}

// Create the OTLP gRPC-based metric exporter
exporter, err := otlpmetricgrpc.New(ctx, otlpmetricgrpc.WithGRPCConn(conn))
if err != nil {
    return nil, fmt.Errorf("failed to create metric exporter: %w", err)
}

res, err := resource.New(ctx,
    resource.WithSchemaURL(semconv.SchemaURL),
    resource.WithFromEnv(),
    resource.WithAttributes(
        semconv.ServiceNameKey.String(serviceName),
        semconv.ServiceVersionKey.String(os.Getenv("SERVICE_VERSION")),
        // Add your environment variables as attributes
        attribute.String("K8S_NODE_NAME", os.Getenv("K8S_NODE_NAME")),
        attribute.String("K8S_NAMESPACE", os.Getenv("K8S_NAMESPACE")),
        attribute.String("K8S_POD_IP", os.Getenv("K8S_POD_IP")),
        attribute.String("K8S_POD_UID", os.Getenv("K8S_POD_UID")),
        attribute.String("K8S_POD_NAME", os.Getenv("K8S_POD_NAME")),
    ),
)
fmt.Println("Resource Attributes:", res.Attributes())

// Store resource attributes for later use
ResourceAttrsKub = res.Attributes()

if err != nil {
    return nil, fmt.Errorf("failed to create resource: %w", err)
}

// Return new MeterProvider with periodic metric reader
return sdkmetric.NewMeterProvider(
    sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exporter)),
    sdkmetric.WithResource(res),
), nil
}

OpenTelemetry Collector Helm Values配置 #

image:
  repository: "otel/opentelemetry-collector-contrib"
  tag: "latest"

mode: "deployment"

presets:
  kubernetesAttributes:
    enabled: true

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"

  exporters:
    prometheus:
      endpoint: "0.0.0.0:8889"
      namespace: ""

  service:
    pipelines:
      metrics:
        receivers: [otlp]
        processors: [batch]
        exporters: [prometheus]

resources:
  requests:
    memory: "256Mi"
    cpu: "200m"
  limits:
    memory: "512Mi"
    cpu: "500m"

enabled: true

ports:
  metrics:
    enabled: true
    containerPort: 8889
    servicePort: 8889
    protocol: TCP

已尝试操作 #

• 确认Go应用可正确读取并访问相关环境变量
• 验证OpenTelemetry Collector配置中kubernetesAttributes已启用
• 确认其他指标可正常导出至Prometheus

核心问题 #

1. OpenTelemetry Collector配置是否缺少相关设置以包含Kubernetes属性？
2. 在initMeterProvider中设置属性的方式是否合适？如需让属性在Prometheus中显示，是否需要调整设置方式？

一、OpenTelemetry Collector配置问题 #

你启用了kubernetesAttributes预设，但未将对应的k8sattributes处理器加入metrics pipeline，同时Prometheus exporter默认不会自动将资源属性转为标签，这是两个核心缺失项。

修改后的Collector关键配置 #

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"

  processors:
    batch: {}

  exporters:
    prometheus:
      endpoint: "0.0.0.0:8889"
      namespace: ""
      # 开启资源属性转Prometheus标签
      resource_to_telemetry_conversion:
        enabled: true

  service:
    pipelines:
      metrics:
        receivers: [otlp]
        processors: [k8sattributes, batch]  # 加入k8sattributes处理器，顺序在batch之前
        exporters: [prometheus]

配置说明 #

• k8sattributes处理器需要加入pipeline才能生效（启用预设后会自动创建该处理器，但需手动加入pipeline）
• resource_to_telemetry_conversion.enabled: true：强制将资源属性转换为Prometheus可识别的标签

二、Go代码属性设置优化 #

当前设置方式可行，但需调整为符合OTel语义化规范的键，避免自定义键可能导致的兼容问题：

修改后的资源属性配置 #

res, err := resource.New(ctx,
    resource.WithSchemaURL(semconv.SchemaURL),
    resource.WithFromEnv(),
    resource.WithAttributes(
        semconv.ServiceNameKey.String(serviceName),
        semconv.ServiceVersionKey.String(os.Getenv("SERVICE_VERSION")),
        // 使用OTel语义化标准键
        semconv.K8SNodeNameKey.String(os.Getenv("K8S_NODE_NAME")),
        semconv.K8SNamespaceNameKey.String(os.Getenv("K8S_NAMESPACE")),
        semconv.K8SPodIPKey.String(os.Getenv("K8S_POD_IP")),
        semconv.K8SPodUIDKey.String(os.Getenv("K8S_POD_UID")),
        semconv.K8SPodNameKey.String(os.Getenv("K8S_POD_NAME")),
    ),
)

额外优化点 #

无需手动通过metric.WithAttributes(telemetry.ResourceAttrsKub...)传递资源属性，关联了资源的MeterProvider会自动将资源属性附加到所有指标上。

三、排查思路 #

1. 检查Collector日志：查看Collector Pod日志，确认k8sattributes处理器是否正常启动，排查RBAC权限问题（该处理器需访问K8S API权限）
2. 直接验证Collector端点：通过curl http://<collector-service-ip>:8889/metrics查看指标是否已携带Kubernetes标签，排除Prometheus抓取环节问题
3. 用debug exporter验证数据：临时将Prometheus exporter替换为debug exporter，确认Collector是否接收到Go应用发送的资源属性
4. 确认环境变量注入正确性：在Go应用Pod中执行echo $K8S_NODE_NAME等命令，验证环境变量是否被正确注入

内容的提问来源于stack exchange，提问作者har17bar