AKS上Windows容器间歇性CrashLoopBackOff问题（.NET 6应用）

问题背景 #

• 运行环境：AKS集群，部署11个Linux容器应用与11个Windows容器应用（Windows应用依赖Windows平台C++库），均基于.NET 6开发
• 触发场景：通过Azure DevOps流水线并行部署全部22个应用后，随机出现1-2个应用Pod进入间歇性CrashLoopBackOff状态；部分多实例应用会出现一个实例启动正常、另一个实例启动失败的情况
• 失败根源：所有异常Pod均因连接依赖Azure Key Vault存储密钥的Azure App Configuration服务失败，日志中出现两种Socket异常：

异常1：连接超时（System.Net.Sockets.SocketException (10060)） #

---> Azure.RequestFailedException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (my-demo-dev-appconfig-ac.azconfig.io:443)
 ---> System.Net.Http.HttpRequestException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (my-demo-dev-appconfig-ac.azconfig.io:443)
 ---> System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

异常2：未知主机（System.Net.Sockets.SocketException (11001)） #

Unhandled exception. System.AggregateException: Retry failed after 3 tries. Retry settings can be adjusted in ClientOptions.Retry. (No such host is known. (my-demo-dev-appconfig-ac.azconfig.io:443)) (No such host is known. (my-demo-dev-appconfig-ac.azconfig.io:443)) (No such host is known. (my-demo-dev-appconfig-ac.azconfig.io:443))
 ---> Azure.RequestFailedException: No such host is known. (my-demo-dev-appconfig-ac.azconfig.io:443)
 ---> System.Net.Http.HttpRequestException: No such host is known. (my-demo-dev-appconfig-ac.azconfig.io:443)
 ---> System.Net.Sockets.SocketException (11001): No such host is known.

已尝试的修复措施 #

1. Kubernetes Pod DNS配置优化 #

最初Linux与Windows容器均出现该问题，在部署清单中添加以下dnsConfig配置后，Linux容器的CrashLoopBackOff问题彻底解决，但Windows容器问题仍存在：

template:
    metadata:
      labels:
        app: ${aks_app_name}$
        service: ${aks_app_name}$
    spec:
      nodeSelector:
        "kubernetes.io/os": ${aks_app_nodeselector}$
      priorityClassName: ${aks_app_container_priorityclass_name}$
      # 优化Pod DNS策略以提升解析效率
      dnsConfig:
        options:
          - name: ndots
            value: "2"
          - name: timeout
            value: "15"
          - name: attempts
            value: "3"
          - name: use-vc
          - name: single-request-reopen
      volumes:

2. .NET客户端重试策略调整 #

调整Azure App Configuration与Key Vault客户端的重试参数，仅延长了Pod进入CrashLoopBackOff的时间，未解决根本问题：

Azure App Configuration客户端配置

return configurationBuilder.AddAzureAppConfiguration(options =>
            {
                options
                    .Connect(appConfigurationEndpoint)
                    .ConfigureClientOptions(clientOptions =>
                    {
                        clientOptions.Retry.Delay = TimeSpan.FromSeconds(10);
                        clientOptions.Retry.MaxDelay = TimeSpan.FromSeconds(40);
                        clientOptions.Retry.MaxRetries = 5;
                        clientOptions.Retry.Mode = RetryMode.Exponential;
                    });

Key Vault SecretClient配置

SecretClientOptions secretClientOptions = new()
            {
                Retry =
                {
                    Delay= TimeSpan.FromSeconds(10),
                    MaxDelay = TimeSpan.FromSeconds(40),
                    MaxRetries = 5,
                    Mode = RetryMode.Exponential
                }
            };

临时缓解方案 #

• 手动或通过自动化工具删除异常Pod，重新创建的Pod可正常启动
• 部分异常Pod等待45分钟至1小时后可自动恢复

需求 #

寻求针对Windows容器场景的AKS配置优化方案或**.NET应用层面的修复措施**，彻底解决并行部署时的Azure App Configuration连接失败问题。

内容的提问来源于stack exchange，提问作者ChamindaC