.NET 4.7.2 Windows服务中HttpClient SendAsync请求莫名超时延迟问题排查求助

Background #

We have a Windows service built on .NET 4.7.2 with NancyFX that both receives and sends REST requests. There's a mix of async and sync code (we know this isn't ideal). The thread pool stays healthy with only 70 threads in the process, but some HTTP responses suffer delays of tens or even hundreds of seconds, eventually leading to task cancellation.

Our code includes patterns like:

public async Task<Guid> SomeFunction() { 
    ... 
    var response = await _httpClient.SendAsync(request, cancellationToken); 
    ... 
}

// Called synchronously with:
SomeFunction().Result

We've already ruled out several possibilities:

• Network issues: ETW traces (Microsoft-Windows-NDIS-PacketCapture/PacketFragment) confirm packets return almost immediately
• Thread pool starvation: Thread count remains stable at 70, and Microsoft-Windows-DotNETRuntime/ThreadPoolWorkerThreadAdjustment/Adjustment traces show no starvation events
• Deadlocks: Deadlocks typically cause permanent hangs, not delayed completion, so this is unlikely
• Single HttpClient instance misuse: We've confirmed we're using only one instance
• HTTP connection exhaustion: We added <connectionManagement><add address="*" maxconnection="1024"/></connectionManagement> which resolved queuing, but the request completion issue persists

Potential Root Causes & Solutions #

1. Synchronous Blocking on Async Code Causing Context Bottlenecks

Even without thread pool exhaustion, using .Result to block on async methods can create hidden bottlenecks. If your code runs in a context that has limited thread availability (even if NancyFX doesn't use a sync context by default, it's worth verifying), blocking threads can prevent async tasks from resuming promptly. This doesn't always cause a full deadlock, but it can lead to significant delays as the runtime waits for blocked threads to free up.

Solution:
Push forward with migrating to full async/await everywhere, including your NancyFX route handlers (Nancy supports async natively). Example of an async route:

Get["/api/your-endpoint"] = async (parameters, ct) =>
{
    var result = await SomeFunction(ct);
    return Response.AsJson(result);
};

Ensure all callers up the stack are async to eliminate blocking entirely.

2. HttpClient Handler Misconfiguration

Even with sufficient connections, the underlying HttpClientHandler might have settings causing delays:

• MaxConnectionsPerServer: Double-check that your connectionManagement config isn't being overridden by a custom handler (explicitly set MaxConnectionsPerServer if using a custom handler)
• Timeout & Cancellation: Verify HttpClient.Timeout is set appropriately, and ensure cancellation tokens are propagated to every async call in your stack. A missing token could let operations run longer than intended before cancellation triggers.
• Proxy or Redirect Issues: If a proxy is configured (even unintentionally), it could add hidden delays. Disable proxy usage in your handler if it's not needed.

Solution:
Explicitly configure your HttpClient handler to avoid ambiguity:

var handler = new HttpClientHandler
{
    MaxConnectionsPerServer = 1024,
    UseProxy = false,
    AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
var httpClient = new HttpClient(handler) { Timeout = TimeSpan.FromSeconds(30) };

3. Unidentified Gaps in the .NET HTTP Stack

You've confirmed packets return quickly, but the response isn't reaching your C# code. Dig deeper into the handoff between the OS HTTP stack and .NET:

• Enable Microsoft-Windows-HttpService/HttpTrace events to track when the OS receives the response and when it's passed to .NET
• Add System.Net.Http provider traces in PerfView to pinpoint delays between HttpClientHandler.SendAsyncStart and HttpResponseMessage.Complete events

Solution:
Run PerfView with these extended providers to fill in the trace gaps:

PerfView.exe collect -Providers:Microsoft-Windows-HttpService,System.Net.Http,Microsoft-Windows-DotNETRuntime

Analyze the timeline to see exactly where the response is stuck between network receipt and your code execution.

4. .NET 4.7.2 Async Framework Limitations

Older .NET Framework versions have known edge cases with async task scheduling and context handling. .NET 4.7.2 has fixes, but it still lacks some of the async improvements introduced in .NET Framework 4.8.

Solution:
Apply the latest cumulative updates for .NET Framework 4.7.2 to ensure you have all bug fixes. If feasible, upgrade to .NET Framework 4.8, which includes significant improvements to async performance and thread pool management.

内容的提问来源于stack exchange，提问作者Mark