无需Grafana Agent，Python应用直连Grafana Cloud Tempo遇404错误

需求说明 #

• 从Lambda环境直接发送追踪数据至Grafana Cloud的Tempo实例
• 无需部署Grafana Agent（避免24/7运行）
• 已通过Grafana Agent验证该Tempo实例可用

问题现象 #

使用基于opentelemetry-exporter-otlp-proto-http库的Python脚本发送测试追踪数据时，尝试多个Tempo端点均返回错误：Failed to export batch code: 404, reason: 404 page not found

测试脚本 #

import base64
from typing import Tuple

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import \
    OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import Tracer, TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

def create_span_exporter_for_tempo() -> Tuple[Tracer, BatchSpanProcessor]:

    resource = Resource(attributes={"service.name": constants.APP_NAME})

    headers = {
        "Authorization": f"Basic {base64.b64encode(f'{constants.GRAFANA_USERNAME}:{constants.GRAFANA_PASSWORD}'.encode('ascii')).decode('ascii')}"
    }
    tempo_exporter = OTLPSpanExporter(endpoint=constants.TEMPO_ENDPOINT, headers=headers)

    trace.set_tracer_provider(TracerProvider(resource=resource))
    tracer = trace.get_tracer(__name__)
    span_processor = BatchSpanProcessor(tempo_exporter)

    trace.get_tracer_provider().add_span_processor(span_processor)
    return tracer, span_processor

tracer, span_processor = create_span_exporter_for_tempo()
with tracer.start_as_current_span("test") as span:
        print("test!")
        span.set_attribute("check", "works")

尝试过的Tempo端点 #

• https://tempo-prod-04-prod-us-east-0.grafana.net/tempo
• https://tempo-prod-04-prod-us-east-0.grafana.net/tempo/v1/traces
• https://tempo-prod-04-prod-us-east-0.grafana.net/v1/traces
• https://tempo-prod-04-prod-us-east-0.grafana.net:443/tempo/v1/traces
• https://tempo-prod-04-prod-us-east-0.grafana.net/tempo/v1/trace

解决方案 #

1. 修正Tempo端点路径 #

Grafana Cloud Tempo的OTLP HTTP标准端点格式是https://<tempo-region-endpoint>/otlp/v1/traces，你之前尝试的路径都带了错误的/tempo前缀或路径。针对你的实例，正确端点为：
https://tempo-prod-04-prod-us-east-0.grafana.net/otlp/v1/traces

2. 确认认证信息正确性 #

确保constants.GRAFANA_USERNAME是Grafana Cloud中Tempo实例的实例ID（不是账号邮箱），constants.GRAFANA_PASSWORD是对应的API密钥，这两个值可在Grafana Cloud的Tempo实例详情页获取。

3. 适配Lambda短运行时特性 #

Lambda是短生命周期环境，BatchSpanProcessor可能在函数结束前无法完成异步导出，需做两处调整：

• 改用SimpleSpanProcessor同步导出span
• 函数结束前手动触发强制导出

修正后的完整脚本 #

import base64
from typing import Tuple

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import \
    OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import Tracer, TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

# 替换为你的实际配置
class constants:
    APP_NAME = "lambda-test-service"
    GRAFANA_USERNAME = "your-tempo-instance-id"
    GRAFANA_PASSWORD = "your-grafana-api-key"
    TEMPO_ENDPOINT = "https://tempo-prod-04-prod-us-east-0.grafana.net/otlp/v1/traces"

def create_span_exporter_for_tempo() -> Tuple[Tracer, SimpleSpanProcessor]:

    resource = Resource(attributes={"service.name": constants.APP_NAME})

    headers = {
        "Authorization": f"Basic {base64.b64encode(f'{constants.GRAFANA_USERNAME}:{constants.GRAFANA_PASSWORD}'.encode('ascii')).decode('ascii')}"
    }
    tempo_exporter = OTLPSpanExporter(endpoint=constants.TEMPO_ENDPOINT, headers=headers)

    trace.set_tracer_provider(TracerProvider(resource=resource))
    tracer = trace.get_tracer(__name__)
    span_processor = SimpleSpanProcessor(tempo_exporter)

    trace.get_tracer_provider().add_span_processor(span_processor)
    return tracer, span_processor

tracer, span_processor = create_span_exporter_for_tempo()
with tracer.start_as_current_span("test") as span:
        print("test!")
        span.set_attribute("check", "works")

# 强制导出所有未发送的span
span_processor.force_flush()

内容的提问来源于stack exchange，提问作者Utsav Chokshi