如何捕获并诊断Kestrel.Core.BadHttpRequestException异常

核心异常：Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content
运行环境：Ubuntu 18.04.5 + Nginx + ASP.NET Core，Bugsnag、NewRelic未采集到异常信息，异常发生时全量请求返回网关超时

完整异常堆栈 #

at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content.
   at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken)
   at System.Text.Json.JsonSerializer.ReadAsync[TValue](Stream utf8Json, Type returnType, JsonSerializerOptions options, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
   at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
   at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
   at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value)
   at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.

问题追踪方法 #

• 补全日志采集：将Microsoft.AspNetCore.Server.Kestrel日志级别调整为Debug，开启Nginx访问日志、错误日志的全字段记录，新增$request_time、$upstream_response_time、$content_length、$body_bytes_sent等字段，对比异常发生时间窗口的请求特征，定位异常请求的来源、请求体大小、客户端IP等信息。
• 实时抓包验证：异常发生时用tcpdump抓取服务器80/443端口的网络流量，校验请求头声明的Content-Length和实际传输的请求体大小是否匹配，排查是否存在客户端主动断连、网络丢包、链路中断的情况。
• 进程资源监控：配置监控项持续采集dotnet进程的CPU、内存、线程池使用率、TCP连接数、等待队列长度，异常触发时同步记录上述指标，排查是否存在资源耗尽的情况。
• 场景复现测试：模拟大请求体上传、上传中途断连、短时间大量异常请求涌入的场景，验证是否能复现相同异常和全局网关超时现象。

可能的诱因 #

• 客户端侧问题：客户端上传请求时声明的Content-Length大于实际传输的请求体大小，或者大文件/大JSON上传过程中主动中断请求、客户端网络波动断开连接，都会触发Kestrel抛出该异常。
• Nginx配置问题：client_max_body_size配置小于实际请求体大小，Nginx主动截断请求；proxy_read_timeout/proxy_send_timeout配置过短，Nginx提前断开和Kestrel的连接，导致Kestrel读取请求体不完整。
• Kestrel配置问题：MaxRequestBodySize设置小于实际请求体大小，或者请求体读取超时时间配置不合理，Kestrel主动中断未完成的请求读取。
• 服务器资源问题：dotnet进程CPU、内存占用过高，或者线程池耗尽，无法及时处理请求读取逻辑，导致请求被中间层超时断开。
• 网络层问题：服务器所在网络存在限流、丢包，或者防火墙规则主动切断长连接，导致请求体传输不完整。

全局网关超时合理性判断 #

该异常本身为单请求级别的错误，正常场景下不会触发全局网关超时。如果出现全量请求网关超时，说明异常发生时伴随了以下问题：

1. 异常触发了应用进程的死锁、资源泄漏，导致所有后续请求都无法被处理，Nginx等待上游响应超时返回504
2. 短时间内大量该类异常耗尽了Kestrel的连接池，新请求无法建立连接，导致全局不可用
3. 服务器网络、系统资源被耗尽，所有上游请求无法得到正常响应

内容的提问来源于stack exchange，提问作者Faisal