Azure ASP.NET Core应用随机出现502.3错误，如何排查与修复？

First off, let's break down what these errors mean: the 502.3 code specifically points to a connection failure between your ARR (Application Request Routing) proxy and the backend web application/CGI process. The randomness and "works first, fails second" behavior suggest intermittent issues with connection pooling, process stability, or TCP-level problems—even if you haven't touched configs, environmental changes (like traffic spikes, memory leaks, or cached connection corruption) can trigger this.

Here's a step-by-step troubleshooting plan to narrow down the root cause:

1. Verify Backend Application/Process Health #

Even if you didn't change code, your app or CGI process might be crashing or leaking memory silently:

• Open Event Viewer (eventvwr.msc) and check the Windows Logs > Application section for errors from W3SVC, ASP.NET, or your application's executable. Look for events indicating process crashes, unhandled exceptions, or memory exhaustion.
• Check your IIS application pool's recycling logs (found in %SystemDrive%\inetpub\logs\W3SVC1 or your site's log directory) to see if the pool is recycling unexpectedly—this would cause in-flight requests to fail with 502s.
• For CGI apps: Enable CGI logging in IIS (go to your site > CGI > Enable Logging, set a log path) to capture exit codes and partial output from the CGI process. This will tell you if the process is exiting before returning valid HTTP headers.

2. Inspect ARR Proxy Configuration & Logs #

ARR's connection pooling or timeout settings might be misaligned with your backend:

• In IIS Manager, go to your server node > Application Request Routing Cache > Server Proxy Settings:
  • Increase the Connection timeout (default is 30s) to 60s or higher—sometimes backend processes take longer to respond on subsequent requests.
  • Ensure Maximum response buffer size is large enough for your app's responses.
• Check ARR's detailed logs in %SystemDrive%\inetpub\logs\LogFiles\W3SVC1 (filter for sc-substatus=502.3). Look for fields like cs-host (backend server) and time-taken to see if failures correlate with specific backend instances or slow responses.
• Enable Failed Request Tracing Rules for your site: Create a rule to capture 502 status codes. This will generate a detailed trace log showing every step of the request, including ARR's attempt to connect to the backend and any errors during that process.

3. Check TCP Connection & Port Exhaustion #

Intermittent connection failures often stem from TCP-level issues:

• Run netstat -ano | findstr :<backend-port> (replace <backend-port> with your app's port, e.g., 80 or 443) to check for a high number of TIME_WAIT connections. These are closed connections that are still reserving ports, which can block new connections.
• If port exhaustion is an issue, adjust TCP registry settings (note: requires server restart):
  • Open regedit and navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
  • Set TcpTimedWaitDelay to 30 (reduces how long ports stay in TIME_WAIT)
  • Set MaxUserPort to 65534 (increases the range of available ephemeral ports)

4. Rule Out ARP Cache Corruption #

Rarely, ARP cache mismatches can prevent ARR from reaching the backend:

• Run arp -a to view the ARP cache and verify that your backend server's IP maps to the correct MAC address.
• If there's a mismatch, clear the cache with arp -d <backend-server-ip> and let it refresh automatically.

5. Isolate ARR from the Backend #

To confirm if the issue is with ARR or the backend itself:

• Temporarily bypass ARR by accessing the backend server directly (via its IP/port) and repeat the failing operation. If the error still occurs, the problem is with your app/CGI process or backend server—not ARR. If it works, focus back on ARR's connection settings or proxy configuration.

Potential Fixes Based on Root Cause #

• App/CGI crashes: Use tools like DebugDiag to capture crash dumps and analyze unhandled exceptions or memory leaks in your application.
• ARR timeouts: Adjust the proxy connection timeout as mentioned earlier, or enable ARR's connection pooling settings to reuse connections more efficiently.
• CGI header issues: Ensure your CGI app outputs a valid HTTP header first (e.g., Content-Type: text/html\r\n\r\n) before sending content. Missing or malformed headers trigger the "did not return a valid set of HTTP errors" message.
• Port exhaustion: Implement load balancing across multiple backend servers, or adjust TCP settings to reduce port reservation time.

内容的提问来源于stack exchange，提问作者amsDeveloper