启用TLS双向认证的Azure App Service POST大小限制异常

问题背景 #

在Azure App Service中启用TLS双向认证后，用于接收IoT设备通过multipart/form-data格式POST上传的小于300KB的图片时，出现异常拦截问题。

问题现象 #

• 启用客户端认证后，仅小于约100KB的文件可正常透传到后端：已验证100000字节文件可正常到达业务代码，150000字节文件上传失败
• 超过阈值的请求会在负载均衡器层面直接返回403 Forbidden响应，完全无法触达后端业务代码
• 关闭客户端认证后，所有大小的请求均可正常到达后端：仅因缺少X-ARR-ClientCert请求头触发业务逻辑报错，链路连通符合预期
• 未自行配置过任何文件上传大小限制规则，公开资料中未检索到关于启用客户端认证存在请求大小限制的官方说明

复现日志 #

100KB文件上传日志（请求正常到达后端） #

$ curl --cert my.crt --key my.key https://my-site.azurewebsites.net/Upload/uploadImage -F image=@delme-100k.jpg --cookie-jar sys-cookies.jar --cookie sys-cookies.jar --tlsv1.2 -v
*   Trying x.x.x.x:443...
* Connected to my-site.azurewebsites.net (x.x.x.x) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=*.azurewebsites.net
*  start date: Mar 14 18:39:55 2022 GMT
*  expire date: Mar  9 18:39:55 2023 GMT
*  subjectAltName: host "my-site.azurewebsites.net" matched cert's "*.azurewebsites.net"
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure TLS Issuing CA 01
*  SSL certificate verify ok.
> POST /Upload/uploadImage
> Host: my-site.azurewebsites.net
> User-Agent: curl/7.74.0
> Accept: */*
> Cookie: ARRAffinitySameSite=b[...]7; ARRAffinity=b[...]7
> Content-Length: 100193
> Content-Type: multipart/form-data; boundary=------------------------e6811f73870ec90c
>
* We are completely uploaded and fine
* TLSv1.2 (IN), TLS handshake, Hello request (0):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< Content-Type: application/json; charset=utf-8
< Date: Thu, 09 Jun 2022 06:46:01 GMT
< Server: Microsoft-IIS/10.0
< Access-Control-Allow-Origin: *
* Replaced cookie ARRAffinity="b[...]7" for domain my-site.azurewebsites.net, path /, expire 0
< Set-Cookie: ARRAffinity=b[...]7;Path=/;HttpOnly;Secure;Domain=my-site.azurewebsites.net
* Replaced cookie ARRAffinitySameSite="b[...]7" for domain my-site.azurewebsites.net, path /, expire 0
< Set-Cookie: ARRAffinitySameSite=b[...]7;Path=/;HttpOnly;SameSite=None;Secure;Domain=my-site.azurewebsites.net
< Transfer-Encoding: chunked
< X-Powered-By: ASP.NET
<
* Connection #0 to host my-site.azurewebsites.net left intact
{"error":"Error on uploading image!"}

该场景返回500为预期行为：测试用100KB文件为截断生成的非法JPEG，业务代码主动抛出上传错误。

150KB文件上传日志（负载均衡层拦截返回403） #

$ curl --cert my.crt --key my.key https://my-site.azurewebsites.net/Upload/uploadImage -F image=@delme-150k.jpg --cookie-jar sys-cookies.jar --cookie sys-cookies.jar --tlsv1.2 -v
*   Trying x.x.x.x:443...
* Connected to my-site.azurewebsites.net (x.x.x.x) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=*.azurewebsites.net
*  start date: Mar 14 18:39:55 2022 GMT
*  expire date: Mar  9 18:39:55 2023 GMT
*  subjectAltName: host "my-site.azurewebsites.net" matched cert's "*.azurewebsites.net"
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure TLS Issuing CA 01
*  SSL certificate verify ok.
> POST /Upload/uploadImage HTTP/1.1
> Host: my-site.azurewebsites.net
> User-Agent: curl/7.74.0
> Accept: */*
> Cookie: ARRAffinitySameSite=b[...]7; ARRAffinity=b[...]7
> Content-Length: 150193
> Content-Type: multipart/form-data; boundary=------------------------8f78ee43724d4b8d
>
* TLSv1.2 (IN), TLS handshake, Hello request (0):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Content-Length: 0
< Connection: close
< Date: Thu, 09 Jun 2022 06:45:39 GMT
<
* we are done reading and this is set to close, stop send
* Closing connection 0

从日志可确认：大文件场景下交互流程明显缩短，cURL未完成文件上传，上传流量未达100KB就被前置拦截。

问题根因 #

这是Azure App Service前置ARR（Application Request Routing）层的已知默认行为，和自行配置的上传大小限制没有关系：

1. App Service启用mTLS时，前置负载均衡不会在初始TLS握手阶段就请求客户端证书，而是等初始连接建立、收到请求头后，才会发起TLS重新协商流程要求客户端提供证书
2. 重新协商过程中，ARR需要把已经收到的请求体缓存下来，等带客户端证书的二次握手完成后，再把缓存的请求体转发给后端实例
3. ARR默认的请求体缓存阈值就是100KB：如果请求体大小超过该阈值，ARR不会缓存完整请求体，直接在协商阶段返回403 Forbidden断开连接，这就是观察到的拦截现象。
   日志细节也能佐证这个逻辑：小文件场景下能看到完整的Request CERT握手流程，大文件场景下刚触发重新协商就直接返回403，根本走不到请求客户端证书的步骤。

解决方案 #

按落地优先级排序：

• 优先方案：强制客户端在初始握手阶段就发送客户端证书
  客户端发起请求时不要复用已有TLS会话，在初始Client Hello阶段就携带客户端证书，避免触发ARR的请求后重新协商流程，就不会碰到100KB缓存限制。curl可通过添加--tls-max 1.2 --no-sessionid参数验证效果，大部分IoT设备的TLS栈也支持配置该行为。
• 轻量改造方案：调整客户端请求逻辑，先完成mTLS握手再发送大请求体
  客户端可以先向同域名发一个空GET请求完成带客户端证书的TLS握手，复用该TLS会话再发POST上传请求，此时不会触发重新协商，也不会被大小限制拦截。
• 架构调整方案：如果需要支持更大的单次上传请求，替换内置mTLS实现
  可以在App Service前挂载API Management或者Application Gateway，在网关层终止mTLS连接，再把请求转发给后端App Service，网关层的mTLS实现没有这个100KB的缓存限制。

内容的提问来源于stack exchange，提问作者Bogdan Stăncescu