ASP.NET 6中OpenTelemetry OTLP Exporter内存访问异常求助
问题描述
我有一个ASP.NET 6应用,已集成OpenTelemetry并配置OTLP Exporter,配置代码如下:
services.AddOpenTelemetryTracing(tracerProviderBuilder => { tracerProviderBuilder .AddOtlpExporter(opt => { opt.Protocol = OtlpExportProtocol.Grpc; opt.BatchExportProcessorOptions = new BatchExportProcessorOptions<Activity> { ////MaxExportBatchSize = 16 // Set a low batch size }; }) .AddSource(serviceName) .AddSource("MySqlConnector") // Or remove this .SetResourceBuilder( ResourceBuilder.CreateDefault() .AddService(serviceName: serviceName, serviceVersion: serviceVersion) .AddDetector(new AWSEC2ResourceDetector())) .AddAWSInstrumentation() .AddHttpClientInstrumentation() .AddAspNetCoreInstrumentation() .AddRedisInstrumentation(); });
使用的依赖包及版本:
<PackageReference Include="OpenTelemetry" Version="1.4.0-alpha.2" /> <PackageReference Include="OpenTelemetry.Contrib.Extensions.AWSXRay" Version="1.2.0" /> <PackageReference Include="OpenTelemetry.Contrib.Instrumentation.AWS" Version="1.0.1" /> <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.3.0" /> <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc9.6" /> <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.6" /> <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc9.6" /> <PackageReference Include="OpenTelemetry.Instrumentation.StackExchangeRedis" Version="1.0.0-rc9.7" />
运行集成测试时触发致命异常:
Exit code is -1073741819 (Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. at System.Runtime.CompilerServices.CastHelpers.IsInstanceOfClass(Void*, System.Object) at OpenTelemetry.Internal.TagTransformer`1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TryTransformTag(System.Collections.Generic.KeyValuePair`2<System.String,System.Object>, System.__Canon ByRef) at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ActivityExtensions+TagEnumerationState.ForEach(System.Collections.Generic.KeyValuePair`2<System.String,System.Object>) at DynamicClass.AllocationFreeForEachDelegate(System.Collections.Generic.IEnumerable`1<System.Collections.Generic.KeyValuePair`2<System.String,System.Object>>, TagEnumerationState ByRef, ForEachDelegate<System.Collections.Generic.IEnumerable`1<System.Collections.Generic.KeyValuePair`2<System.String,System.Object>>,System.Collections.Generic.KeyValuePair`2<System.String,System.Object>,TagEnumerationState>) at OpenTelemetry.Trace.ActivityHelperExtensions+ActivityTagsEnumeratorFactory`1[[OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ActivityExtensions+TagEnumerationState, OpenTelemetry.Exporter.OpenTelemetryProtocol, Version=1.0.0.0, Culture=neutral, PublicKeyToken=7bd6737fe5b67e3c]].Enumerate(System.Diagnostics.ActivityEvent, TagEnumerationState ByRef) at OpenTelemetry.Trace.ActivityHelperExtensions.EnumerateTags[[OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ActivityExtensions+TagEnumerationState, OpenTelemetry.Exporter.OpenTelemetryProtocol, Version=1.0.0.0, Culture=neutral, PublicKeyToken=7bd6737fe5b67e3c]](System.Diagnostics.ActivityEvent, TagEnumerationState ByRef) at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ActivityExtensions.ToOtlpEvent(System.Diagnostics.ActivityEvent) at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ActivityExtensions+EventEnumerationState.ForEach(System.Diagnostics.ActivityEvent) at OpenTelemetry.Trace.ActivityHelperExtensions+ActivityEventsEnumeratorFactory`1[[OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ActivityExtensions+EventEnumerationState, OpenTelemetry.Exporter.OpenTelemetryProtocol, Version=1.0.0.0, Culture=neutral, PublicKeyToken=7bd6737fe5b67e3c]].ForEachEventCallback(EventEnumerationState ByRef, System.Diagnostics.ActivityEvent) at DynamicClass.AllocationFreeForEachDelegate(System.Collections.Generic.IEnumerable`1<System.Diagnostics.ActivityEvent>, EventEnumerationState ByRef, ForEachDelegate<System.Collections.Generic.IEnumerable`1<System.Diagnostics.ActivityEvent>,System.Diagnostics.ActivityEvent,EventEnumerationState>) at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ActivityExtensions.AddBatch(Opentelemetry.Proto.Collector.Trace.V1.ExportTraceServiceRequest, Opentelemetry.Proto.Resource.V1.Resource, OpenTelemetry.Batch`1<System.Diagnostics.Activity> ByRef) at OpenTelemetry.Exporter.OtlpTraceExporter.Export(OpenTelemetry.Batch`1<System.Diagnostics.Activity> ByRef) at OpenTelemetry.BatchExportProcessor`1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ExporterProc() at System.Threading.Thread.StartCallback())
已知以下操作可临时规避问题:
- 将
MaxExportBatchSize设置为较小值(如16,默认512) - 移除"MySqlConnector"数据源(推测因减少了追踪导出数量)
解决方案
1. 统一OpenTelemetry依赖版本
当前依赖包版本混杂,既有Alpha版核心包,又有不同版本的工具包,版本不匹配是引发内存访问异常的核心原因之一。将所有OpenTelemetry相关包升级到稳定且统一的版本,示例配置如下:
<PackageReference Include="OpenTelemetry" Version="1.5.1" /> <PackageReference Include="OpenTelemetry.Contrib.Extensions.AWSXRay" Version="1.3.1" /> <PackageReference Include="OpenTelemetry.Contrib.Instrumentation.AWS" Version="1.1.0" /> <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.5.1" /> <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.5.1" /> <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.5.1" /> <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.5.1" /> <PackageReference Include="OpenTelemetry.Instrumentation.StackExchangeRedis" Version="1.5.1" />
2. 修复批量导出选项的初始化方式
原代码直接新建BatchExportProcessorOptions<Activity>实例的方式可能存在内存管理问题,建议通过配置委托修改默认选项,而非完全替换:
.AddOtlpExporter(opt => { opt.Protocol = OtlpExportProtocol.Grpc; // 修改默认批量导出参数,无需新建实例 opt.BatchExportProcessorOptions.MaxExportBatchSize = 64; // 可根据实际流量调整,无需过小 opt.BatchExportProcessorOptions.MaxQueueSize = 2048; })
3. 过滤MySqlConnector追踪的异常标签
异常发生在标签转换阶段,推测MySqlConnector生成的追踪标签中存在特殊类型或未正确释放的对象。可通过自定义处理器过滤无效标签:
// 注册自定义标签过滤处理器 services.AddOpenTelemetryTracing(tracerProviderBuilder => { // ... 其他配置 .AddProcessor(new InvalidTagFilterProcessor()) }); // 自定义处理器实现 public class InvalidTagFilterProcessor : BaseProcessor<Activity> { public override void OnEnd(Activity activity) { // 移除非基础类型的标签,避免转换时内存异常 foreach (var tag in activity.Tags.ToList()) { if (tag.Value == null || !IsValidTagType(tag.Value)) { activity.RemoveTag(tag.Key); } } base.OnEnd(activity); } private bool IsValidTagType(object value) { return value is string || value is int || value is long || value is double || value is bool; } }
4. 禁用分配自由枚举特性
该异常与OpenTelemetry的分配自由枚举机制相关,可通过环境变量临时禁用此特性验证:
在应用启动前添加:
Environment.SetEnvironmentVariable("OTEL_DOTNET_TRACES_ALLOCATION_FREE_ENABLED", "false");
若禁用后异常消失,说明是该机制的内存问题,后续可等待OpenTelemetry官方修复。
内容的提问来源于stack exchange,提问作者Mihai
相关产品推荐
相关产品推荐

