.NET Web API集成OpenTelemetry后内存持续增长问题求助
.NET Web API集成OpenTelemetry后内存泄漏问题排查
我在.NET Web API中集成OpenTelemetry库后,出现内存占用持续增长直至触发Pod内存限制导致重启的问题:开发环境下API流量极低且稳定(约4请求/秒,90%为FrontDoor健康检查),但Pod内存6小时内从375MB增至506MB(涨幅35%),后续10小时内进一步升至710MB后Pod重启。
对比间隔6小时的两份GC Dump,发现第二份Dump中新增了106,078个Activity对象、1,060,780个DiagNode<KeyValuePair<String, Object>>对象等。在Visual Studio中对比GC Dump,得到包含15个最大包含大小差异对象的列表(截图:
)
使用的OpenTelemetry相关库版本
<PackageReference Include="OpenTelemetry" Version="1.6.0" /> <PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.6.0" /> <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.6.0" /> <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.6.0" /> <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.5.1-beta.1" /> <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.5.1-beta.1" /> <PackageReference Include="OpenTelemetry.Instrumentation.Runtime" Version="1.5.1" /> <PackageReference Include="OpenTelemetry.Instrumentation.SqlClient" Version="1.5.1-beta.1" />
Startup.cs中OpenTelemetry配置方法
public static IServiceCollection SetupOpenTelemetry(IServiceCollection services, IConfiguration configuration, TracingOptions options) { if (!options.OpenTelemetry.Enabled) return services; if (string.IsNullOrWhiteSpace(options.OpenTelemetry.EndpointUrl)) throw new InvalidOperationException("OpenTelemetry EndpointUrl is not configured."); if (string.IsNullOrWhiteSpace(options.OpenTelemetry.ServiceName)) throw new InvalidOperationException("OpenTelemetry ServiceName is not configured."); var environmentOptions = configuration.GetRequiredSection(EnvironmentOptions.SectionName).Get<EnvironmentOptions>() ?? throw new InvalidOperationException($"Cannot bind {EnvironmentOptions.SectionName} section."); if (string.IsNullOrWhiteSpace(environmentOptions.Name)) throw new InvalidOperationException("Environment Name is not configured."); var serviceAttributes = new[] { new KeyValuePair<string, object>("service.version", Assembly.GetExecutingAssembly().GetName().Version.ToString()), new KeyValuePair<string, object>("service.environment", environmentOptions.Name) }; var resourceBuilder = ResourceBuilder.CreateDefault() .AddService(options.OpenTelemetry.ServiceName) .AddAttributes(serviceAttributes); services.AddOpenTelemetry() .WithMetrics(builder => { builder .SetResourceBuilder(resourceBuilder) .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation() .AddRuntimeInstrumentation() .AddOtlpExporter(exporter => { exporter.Endpoint = new Uri(options.OpenTelemetry.EndpointUrl); exporter.Headers = $"lightstep-access-token={options.OpenTelemetry.AccessToken}"; }); }) .WithTracing(builder => { builder.AddSource("Npgsql") .SetResourceBuilder(resourceBuilder) .SetSampler(new AlwaysOnSampler()) .AddAspNetCoreInstrumentation(o => { o.RecordException = true; o.EnrichWithHttpRequest = (activity, request) => { activity.SetTag("size", request.HttpContext.Response.ContentLength ?? 0); }; o.EnrichWithException = (activity, exception) => { activity.SetTag("stackTrace", exception.StackTrace); }; }) .AddHttpClientInstrumentation(o => { o.RecordException = true; o.EnrichWithException = (activity, exception) => { activity.SetTag("stackTrace", exception.StackTrace); }; }) .AddSqlClientInstrumentation(o => { o.RecordException = true; o.SetDbStatementForText = true; o.EnableConnectionLevelAttributes = true; }) .AddOtlpExporter(o => { o.Endpoint = new Uri(options.OpenTelemetry.EndpointUrl); o.Headers = $"lightstep-access-token={options.OpenTelemetry.AccessToken}"; }); if (options.OpenTelemetry.EnableConsoleExport) { builder.AddConsoleExporter(); } }); return services; }
OtlpExporter的Endpoint为https://ingest.lightstep.com
请问是什么原因导致Activity等对象无法从内存中释放?是否有类似的经验可以分享?
内容的提问来源于stack exchange,提问作者Petofi
相关产品推荐
相关产品推荐

