AKS部署Identity Server：健康检查日志过滤代码失效，注释探针后日志消失

我在部署于AKS集群的Identity Server应用中，尝试通过代码将健康检查日志排除在Application Insights之外，但健康检查日志仍会被记录。当注释掉deployment.yaml中的存活探针（livenessProbe）和就绪探针（readinessProbe）配置后，健康检查日志就不再出现了。

Startup 代码 #

var applicationInsightsConfig = Configuration.GetSection("LoggingConfigurations:ApplicationInsights");
var applicationInsights = LoggingConfiguration.Get<ApplicationInsights>();
services.Configure<ApplicationInsights>(applicationInsightsConfig);
services.AddApplicationInsights(); 

public static void AddApplicationInsights(this IServiceCollection services)
{
    var aiOptions = new ApplicationInsightsServiceOptions();
    aiOptions.EnableAdaptiveSampling = false;

    services.AddApplicationInsightsTelemetry(aiOptions);

    services.Configure<TelemetryConfiguration>((config) =>
    {
        var builder = config.DefaultTelemetrySink.TelemetryProcessorChainBuilder;
        builder.UseAdaptiveSampling(new SamplingPercentageEstimatorSettings
        {
            MinSamplingPercentage = 0.01,
            MaxSamplingPercentage = 100,
            MaxTelemetryItemsPerSecond = 5
        }, null, excludedTypes: "Exception");

        var serviceProvider = services.BuildServiceProvider();
        var options = serviceProvider.GetRequiredService<IOptions<ApplicationInsights>>();
        var httpContextAccessor = serviceProvider.GetRequiredService<IHttpContextAccessor>();

        config.DefaultTelemetrySink.TelemetryProcessorChainBuilder
            .Use(next => new HealthCheckFilterProcessor(next, options, httpContextAccessor))
            .Build();
    });
}

HealthCheckFilterProcessor 代码 #

public class HealthCheckFilterProcessor : ITelemetryProcessor
{
    private ITelemetryProcessor _next { get; set; }
    private readonly IOptions<ApplicationInsights> _config;
    private readonly IHttpContextAccessor _httpContextAccessor;
    List<string> mandatoryExclusionPaths = new List<string> { "/favicon.ico", "/health" };
    List<string> defaultExclusionPaths = new List<string> { "home/index", "/swagger", "/health", ".js", ".css", ".html", ".chtml", ".png", ".jpg", ".jpeg", ".gif", ".svg", ".ico", ".woff", ".woff2", ".ttf", ".map" };

    public HealthCheckFilterProcessor(ITelemetryProcessor next, IOptions<ApplicationInsights> config, IHttpContextAccessor httpContextAccessor)
    {
        _next = next;
        _config = config;
        _httpContextAccessor = httpContextAccessor;
    }

    public void Process(ITelemetry item)
    {
        switch (item)
        {
            case RequestTelemetry request:
                if (item is RequestTelemetry request1)
                {
                    var path = request1.Url.AbsolutePath.ToLowerInvariant();

                    if (path.Contains("/health"))
                    {
                        return; // Do not log this telemetry
                    }
                }
                break;

            case TraceTelemetry trace:
                var message = trace.Message;
                var operationName = trace.Context.Operation?.Name;
                Console.WriteLine("message" + message);
                Console.WriteLine("operationName" + operationName);
                if ((message?.Contains("/health", StringComparison.OrdinalIgnoreCase) ?? false) ||
                    (operationName?.Contains("/health", StringComparison.OrdinalIgnoreCase) ?? false))
                    return;
                break;

            case ExceptionTelemetry exception:
                Console.WriteLine("exception.Exception?.Message" + exception.Context.Operation?.Name);
                Console.WriteLine("exception.Exception?.Message" + exception.Exception?.Message);
                if ((exception.Context.Operation?.Name?.Contains("/health", StringComparison.OrdinalIgnoreCase) ?? false) ||
                    (exception.Exception?.Message?.Contains("/health", StringComparison.OrdinalIgnoreCase) ?? false))
                    return;
                break;

            case DependencyTelemetry dependency:
                Console.WriteLine("dependency.Target" + dependency.Context.Operation?.Name);
                Console.WriteLine("dependency.Target" + dependency.Target);
                if ((dependency.Context.Operation?.Name?.Contains("/health", StringComparison.OrdinalIgnoreCase) ?? false) ||
                    (dependency.Target?.Contains("/health", StringComparison.OrdinalIgnoreCase) ?? false))
                    return;
                break;
        }

        _next.Process(item);
    }
}

deployment.yaml 配置片段 #

livenessProbe: 
  httpGet:
    path: /health
    port: 8080
  periodSeconds: {{ .Values.iam.web.probes.liveness.periodSeconds }}
  initialDelaySeconds: {{ .Values.iam.web.probes.liveness.initialDelaySeconds }}
  timeoutSeconds: {{ .Values.iam.web.probes.liveness.timeoutSeconds }}
readinessProbe: 
  httpGet:
    path: /health
    port: 8080
  periodSeconds: {{ .Values.iam.web.probes.readiness.periodSeconds }}
  initialDelaySeconds: {{ .Values.iam.web.probes.readiness.initialDelaySeconds }}
  timeoutSeconds: {{ .Values.iam.web.probes.readiness.timeoutSeconds }}

问题核心是AKS的存活/就绪探针请求不经过ASP.NET Core的中间件管道，直接由Kubelet发起，绕过了你写的Telemetry Processor过滤逻辑，导致这些请求的日志仍会进入Application Insights。以下是几种可行的解决方法：

方法1：识别Kubelet探针请求并过滤 #

Kubelet发起的探针请求会带有特定的User-Agent头，你可以在Telemetry Processor中检查该头来过滤：

case RequestTelemetry request:
    if (item is RequestTelemetry request1)
    {
        var path = request1.Url.AbsolutePath.ToLowerInvariant();
        // 检查请求是否来自Kubelet
        var isKubeletProbe = request1.Context.Properties.TryGetValue("AspNetCore.HttpContext.Request.Headers.User-Agent", out var userAgent) 
                             && userAgent.Contains("kubelet");
        
        if (path.Contains("/health") || isKubeletProbe)
        {
            return;
        }
    }
    break;

方法2：在采样规则中直接排除健康检查请求 #

修改Application Insights的采样配置，针对/health路径的请求设置采样率为0：

builder.UseAdaptiveSampling(new SamplingPercentageEstimatorSettings
{
    MinSamplingPercentage = 0.01,
    MaxSamplingPercentage = 100,
    MaxTelemetryItemsPerSecond = 5
}, 
// 添加自定义采样过滤器
samplingFilter: (telemetry) =>
{
    if (telemetry is RequestTelemetry request && request.Url.AbsolutePath.ToLowerInvariant().Contains("/health"))
    {
        return false; // 完全排除该请求的遥测
    }
    return true;
},
excludedTypes: "Exception");

方法3：在健康检查配置中标记忽略遥测 #

在Startup的Configure方法中配置健康检查时，直接标记该请求不需要被Application Insights追踪：

app.UseHealthChecks("/health", new HealthCheckOptions
{
    AllowCachingResponses = false,
    ResponseWriter = async (context, report) =>
    {
        context.Response.ContentType = "application/json";
        await context.Response.WriteAsync(JsonSerializer.Serialize(new
        {
            status = report.Status.ToString(),
            checks = report.Entries.Select(e => new { name = e.Key, status = e.Value.Status.ToString(), description = e.Value.Description })
        }));
    },
    // 标记该请求忽略遥测
    OnBeforeHealthCheck = (context) =>
    {
        context.Items["ApplicationInsights:IgnoreTelemetry"] = true;
        return Task.CompletedTask;
    }
});

内容的提问来源于stack exchange，提问作者Gouthami Shivakoti