Elastic Beanstalk上ASP.NET WebApi因get_RetryMode()死锁无响应问题排查

问题背景 #

在Elastic Beanstalk上运行的.NET Framework ASP.NET WebApi应用偶尔出现无响应，通过w3wp.exe进程转储分析发现：阻塞线程卡在Amazon.Runtime.ClientConfig.get_RetryMode()调用上，该线程未释放Amazon.Runtime.Internal.Util.Logger.GetLogger中获取的锁，引发死锁，后续调用因等待锁释放持续阻塞。核心疑问：为何Amazon.Runtime.ClientConfig.get_RetryMode()无法返回？

阻塞调用栈 #

Amazon.Runtime.ClientConfig.get_RetryMode()+2f
Amazon.Runtime.AmazonServiceClient.BuildRuntimePipeline()+1cd
AWS.Logger.Core.AWSLoggerCore..ctor(AWS.Logger.AWSLoggerConfig, System.String)+1b0
AWS.Logger.Log4net.AWSAppender.ActivateOptions()+131
log4net.Repository.Hierarchy.XmlHierarchyConfigurator.ParseAppender(System.Xml.XmlElement)+47b
log4net.Repository.Hierarchy.XmlHierarchyConfigurator.FindAppenderByReference(System.Xml.XmlElement)+1dc
log4net.Repository.Hierarchy.XmlHierarchyConfigurator.ParseChildrenOfLoggerElement(System.Xml.XmlElement, log4net.Repository.Hierarchy.Logger, Boolean)+110
log4net.Repository.Hierarchy.XmlHierarchyConfigurator.ParseRoot(System.Xml.XmlElement)+5f
log4net.Repository.Hierarchy.XmlHierarchyConfigurator.Configure(System.Xml.XmlElement)+554
log4net.Repository.Hierarchy.Hierarchy.XmlRepositoryConfigure(System.Xml.XmlElement)+c9
log4net.Config.XmlConfigurator.InternalConfigure(log4net.Repository.ILoggerRepository, System.IO.Stream)+2ad
log4net.Config.XmlConfigurator.InternalConfigure(log4net.Repository.ILoggerRepository, System.IO.FileInfo)+18f
log4net.Config.XmlConfigurator.Configure(log4net.Repository.ILoggerRepository, System.Uri)+77
log4net.Core.DefaultRepositorySelector.ConfigureRepository(System.Reflection.Assembly, log4net.Repository.ILoggerRepository)+2d1
log4net.Core.DefaultRepositorySelector.CreateRepository(System.Reflection.Assembly, System.Type, System.String, Boolean)+2bf
log4net.Core.DefaultRepositorySelector.GetRepository(System.Reflection.Assembly)+3e
log4net.Core.LoggerManager.GetLogger(System.Reflection.Assembly, System.Type)+45
[[DebuggerU2MCatchHandlerFrame]]
[[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
mscorlib_ni!System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])+84
mscorlib_ni!System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)+92
Amazon.Runtime.Internal.Util.InternalLog4netLogger..ctor(System.Type)+d8
Amazon.Runtime.Internal.Util.Logger..ctor(System.Type)+5f
Amazon.Runtime.Internal.Util.Logger.GetLogger(System.Type)+af
Amazon.Runtime.AppConfigAWSCredentials..ctor()+33
Amazon.Runtime.FallbackCredentialsFactory+<>c.b__10_0()+1f
Amazon.Runtime.FallbackCredentialsFactory.GetCredentials(Boolean)+c7
Amazon.SimpleSystemsManagement.AmazonSimpleSystemsManagementClient..ctor(Amazon.RegionEndpoint)+3b
// UserCode new Amazon.SimpleSystemsManagement.AmazonSimpleSystemsManagementClient(region)

被阻塞调用栈 #

System.Threading.Monitor.Enter(System.Object)
Amazon.Runtime.Internal.Util.Logger.GetLogger(System.Type)+68
Amazon.Runtime.AmazonServiceClient..ctor(Amazon.Runtime.AWSCredentials, Amazon.Runtime.ClientConfig)+8d
// UserCode new Amazon.SimpleSystemsManagement.AmazonSimpleSystemsManagementClient(region)

死锁原因分析 #

从调用栈可以明确这是循环依赖+锁顺序倒置引发的典型死锁：

1. 第一个线程执行路径：

   • 调用new AmazonSimpleSystemsManagementClient(region)，触发凭证加载逻辑FallbackCredentialsFactory.GetCredentials
   • 实例化AppConfigAWSCredentials时调用Logger.GetLogger并获取锁
   • 后续初始化AWS日志组件（AWSAppender）时，需要创建AmazonServiceClient实例，进而调用ClientConfig.get_RetryMode()
   • 此时线程持有Logger锁，但get_RetryMode()内部可能等待其他资源，或触发的逻辑尝试再次获取日志相关锁，导致线程无法推进、锁无法释放

2. 第二个线程执行路径：

   • 同样尝试创建AmazonSimpleSystemsManagementClient，调用Logger.GetLogger时需要等待第一个线程释放锁
   • 但第一个线程卡在get_RetryMode()无法继续，两个线程互相等待，形成死锁

核心触发点 #

• AWS SDK的凭证加载逻辑（AppConfigAWSCredentials初始化）与日志组件（AWS.Logger.Log4net）初始化存在循环依赖
• Logger.GetLogger使用静态锁，多线程并发初始化SDK客户端时，极易触发锁顺序冲突

解决方案建议 #

• 提前单线程初始化：在应用启动阶段（如Application_Start）完成日志配置和AWS客户端的初始化，避免并发初始化操作
• 解耦日志与AWS客户端依赖：调整日志配置，确保AWS日志组件初始化过程中不创建其他AWS服务客户端
• 升级AWS SDK版本：旧版本AWS SDK存在此类循环依赖bug，升级到最新稳定版可修复该问题
• 显式指定凭证：绕过FallbackCredentialsFactory自动加载逻辑，直接在代码中指定凭证，避免初始化凭证时触发日志初始化流程

内容的提问来源于stack exchange，提问作者Sam Sippe