.NET异步Task性能诊断：如何获取生产环境待执行Task指标

Absolutely, you can monitor Task execution metrics and diagnose the production issues you’ve described—whether it’s runaway CPU from backlogged Tasks or blocked executors hogging resources. Let’s break this down into actionable, ready-to-use solutions and custom approaches to get the visibility you need.

Ready-Made Tools & Metrics #

You don’t have to build everything from scratch—.NET and established monitoring tools provide out-of-the-box visibility tailored to your scenarios:

1. .NET Built-In Performance Counters #

The .NET CLR maintains a set of counters that map directly to your problem areas:

• Tasks Waiting to Run: Tracks the number of Tasks queued for an available ThreadPool thread. This is exactly the "pending Task count" you need to catch scenario 1 (Task creation outpacing completion) before CPU hits 100%.
• Tasks Created/sec: Helps you correlate incoming load with Task generation rates.
• ThreadPool Work Items Queued: Complementary to the Task counter, since most Tasks use the ThreadPool by default.
• ThreadPool Threads in Use: Spikes or sustained high values here indicate blocked Tasks (like your scenario 2, where long-running Tasks hold onto threads instead of yielding).

You can access these counters via the System.Diagnostics.PerformanceCounter class, or view them in real-time with tools like Performance Monitor (perfmon) on Windows.

2. TPL EventSource #

The System.Threading.Tasks.TplEventSource class exposes detailed events for the full Task lifecycle (creation, scheduling, execution, completion). You can subscribe to these events to build custom metrics:

• Listen for TaskStarted and TaskCompleted events to track pending Task counts (increment on start, decrement on completion).
• Capture TaskScheduled events to monitor queue growth.

This is lightweight enough for production use, as it’s part of the TPL’s built-in diagnostic infrastructure.

3. APM & Monitoring Tools #

Tools like New Relic, Datadog, or Application Insights automatically track Task execution out of the box. They’ll:

• Visualize pending Task queues and execution times.
• Alert you when queue lengths exceed thresholds.
• Identify long-running or blocked Tasks by flagging execution time outliers.

These tools eliminate the need to write custom code for basic monitoring and alerting.

Custom Task Tracking (Injecting Code at Task Start/End) #

If you need full control over logging, metrics, or custom logic, you can wrap Task creation to inject code at the start and end of every Task. Here are practical approaches:

1. Wrap Task Creation with a Helper Class #

Create a centralized helper method for all Task creation in your system. This ensures every Task gets tracked:

public static class TrackedTask
{
    private static readonly ConcurrentDictionary<Guid, DateTime> _activeTasks = new();
    private static long _pendingTaskCount;

    // Expose pending count as a metric
    public static long PendingTaskCount => Interlocked.Read(ref _pendingTaskCount);

    public static async Task<T> Run<T>(Func<Task<T>> taskFunc, string taskName = "Unnamed Task")
    {
        var taskId = Guid.NewGuid();
        Interlocked.Increment(ref _pendingTaskCount);
        _activeTasks.TryAdd(taskId, DateTime.UtcNow);

        try
        {
            // Log task start (use a high-performance logger to avoid overhead)
            Logger.LogDebug($"Task {taskId} ({taskName}) started");
            return await taskFunc();
        }
        finally
        {
            Interlocked.Decrement(ref _pendingTaskCount);
            if (_activeTasks.TryRemove(taskId, out var startTime))
            {
                var duration = DateTime.UtcNow - startTime;
                Logger.LogDebug($"Task {taskId} ({taskName}) completed in {duration.TotalMilliseconds:F2}ms");
                
                // Trigger alert if duration exceeds threshold
                if (duration.TotalSeconds > 30)
                {
                    Logger.LogWarning($"Long-running task detected: {taskName} took {duration.TotalSeconds:F2}s");
                }
            }
        }
    }

    // Overload for non-generic Tasks
    public static async Task Run(Func<Task> taskFunc, string taskName = "Unnamed Task")
    {
        await Run(() => taskFunc().ContinueWith(_ => (object)null), taskName);
    }
}

Use this helper everywhere you create Tasks (e.g., await TrackedTask.Run(async () => { /* your async logic */ }, "SocketRequestHandler")).

2. Custom TaskScheduler #

For deeper control over Task scheduling, implement a custom TaskScheduler that tracks queue and execution metrics:

public class TrackingTaskScheduler : TaskScheduler
{
    private readonly TaskScheduler _innerScheduler;
    private long _queuedTaskCount;

    public long QueuedTaskCount => Interlocked.Read(ref _queuedTaskCount);

    public TrackingTaskScheduler(TaskScheduler innerScheduler)
    {
        _innerScheduler = innerScheduler ?? TaskScheduler.Default;
    }

    protected override void QueueTask(Task task)
    {
        Interlocked.Increment(ref _queuedTaskCount);
        _innerScheduler.QueueTask(task);
        // Log or alert if queue exceeds threshold
        if (QueuedTaskCount > 1000)
        {
            Logger.LogWarning($"Task queue is backlogged: {QueuedTaskCount} pending tasks");
        }
    }

    protected override bool TryExecuteTaskInline(Task task, bool taskWasPreviouslyQueued)
    {
        if (taskWasPreviouslyQueued)
        {
            Interlocked.Decrement(ref _queuedTaskCount);
        }
        return _innerScheduler.TryExecuteTaskInline(task, taskWasPreviouslyQueued);
    }

    protected override IEnumerable<Task> GetScheduledTasks()
    {
        return _innerScheduler.GetScheduledTasks();
    }
}

You can set this as the default scheduler for your app or specific Task factories to capture all Task activity.

3. AsyncLocal for Context Tracking #

Use AsyncLocal<T> to track context across async calls (e.g., request IDs) and link Task logs to specific requests:

public static class AsyncContext
{
    private static readonly AsyncLocal<string> _requestId = new();

    public static string RequestId
    {
        get => _requestId.Value ?? Guid.NewGuid().ToString();
        set => _requestId.Value = value;
    }
}

Then include AsyncContext.RequestId in your Task logs to trace back to the original request causing long-running Tasks.

Addressing Your Specific Scenarios #

• Scenario 1 (CPU spikes on high load): Monitor the PendingTaskCount (from your custom helper or performance counters). If it’s rising steadily while load increases, you’re creating Tasks faster than they can complete—alert before the queue gets too large.
• Scenario 2 (Blocked Tasks causing slowdowns): Track Task execution durations (via the helper class) and ThreadPool thread usage. Look for Tasks that take far longer than average—these are likely blocking on third-party code. When ThreadPool threads are maxed out and most long-running Tasks are active, you’ll see the "slowdown point" you described.

Testing Tips #

• Use ThreadPool.SetMinThreads and ThreadPool.SetMaxThreads to simulate limited executor resources during testing.
• Avoid excessive logging in production—use batch logging or a high-performance library like Serilog to minimize overhead.
• Test with load generators to replicate high-traffic scenarios and validate your metrics/alerting.

内容的提问来源于stack exchange，提问作者Mike Roibu