Service Fabric集群频繁崩溃转储：Method CompareOrdinal不支持

环境信息 #

• Service Fabric集群版本：10.1.2493.9590
• 节点规格：Standard_D2ds_v4
• 操作系统：Ubuntu 20.04-LTS-Gen2

问题现象 #

集群每小时生成新的崩溃转储，.NET ThreadPool.dmp文件大量写入日志，引发内存占用过高问题。

崩溃转储堆栈跟踪 #

{"Method CompareOrdinal not supported."}        System.NotSupportedException
 Message    Method CompareOrdinal not supported.    string
 Thread [1971120] .NET ThreadPool Worker    System.Threading.Thread
 StackTrace     System.Diagnostics.StackFrame[]
 [0]    Azure.Data.Tables.Queryable.ExpressionNormalizer.VisitMethodCallNoRewrite(System.Linq.Expressions.MethodCallExpression call)    System.Diagnostics.StackFrame
 [1]    Azure.Data.Tables.Queryable.ExpressionNormalizer.VisitMethodCall(System.Linq.Expressions.MethodCallExpression call) System.Diagnostics.StackFrame
 [2]    [External Code] System.Diagnostics.StackFrame
 [3]    Azure.Data.Tables.Queryable.ExpressionNormalizer.VisitBinary(System.Linq.Expressions.BinaryExpression b)    System.Diagnostics.StackFrame
 [4]    [External Code] System.Diagnostics.StackFrame
 [5]    Azure.Data.Tables.Queryable.ExpressionNormalizer.VisitBinary(System.Linq.Expressions.BinaryExpression b)    System.Diagnostics.StackFrame
 [6]    [External Code] System.Diagnostics.StackFrame
 [7]    Azure.Data.Tables.Queryable.ExpressionNormalizer.VisitBinary(System.Linq.Expressions.BinaryExpression b)    System.Diagnostics.StackFrame
 [8]    [External Code] System.Diagnostics.StackFrame
 [9]    Azure.Data.Tables.Queryable.ExpressionNormalizer.Normalize(System.Linq.Expressions.Expression expression, System.Collections.Generic.Dictionary<System.Linq.Expressions.Expression, System.Linq.Expressions.Expression> rewrites)   System.Diagnostics.StackFrame
 [10]   Azure.Data.Tables.TableClient.Bind(System.Linq.Expressions.Expression expression)   System.Diagnostics.StackFrame
 [11]   FabricDCA.AzureTableQueryableCsvUploaderDefault.DeleteOldEntities(Azure.Data.Tables.TableClient tableClient, System.DateTime cutoffTime, string partitionKey, FabricDCA.AzureTableQueryableCsvUploaderDefault.StatusEntity statusEntity)    System.Diagnostics.StackFrame
 [12]   FabricDCA.AzureTableQueryableCsvUploaderDefault.DeleteOldLogsHandler(object state)  System.Diagnostics.StackFrame
 [13]   [External Code] System.Diagnostics.StackFrame
 InnerException null    
 Additional details     string
 

   at Azure.Data.Tables.Queryable.ExpressionNormalizer.VisitMethodCallNoRewrite(MethodCallExpression call)
   at Azure.Data.Tables.Queryable.ExpressionNormalizer.VisitMethodCall(MethodCallExpression call)
   at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at System.Linq.Expressions.ExpressionVisitor.VisitBinary(BinaryExpression node)
   at Azure.Data.Tables.Queryable.ExpressionNormalizer.VisitBinary(BinaryExpression b)
   at System.Linq.Expressions.BinaryExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at System.Linq.Expressions.ExpressionVisitor.VisitBinary(BinaryExpression node)
   at Azure.Data.Tables.Queryable.ExpressionNormalizer.VisitBinary(BinaryExpression b)
   at System.Linq.Expressions.BinaryExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at System.Linq.Expressions.ExpressionVisitor.VisitBinary(BinaryExpression node)
   at Azure.Data.Tables.Queryable.ExpressionNormalizer.VisitBinary(BinaryExpression b)
   at System.Linq.Expressions.BinaryExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at System.Linq.Expressions.ExpressionVisitor.VisitLambda[T](Expression`1 node)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Azure.Data.Tables.Queryable.ExpressionNormalizer.Normalize(Expression expression, Dictionary`2 rewrites)
   at Azure.Data.Tables.TableClient.Bind(Expression expression)
   at Azure.Data.Tables.TableClient.Query[T](Expression`1 filter, Nullable`1 maxPerPage, IEnumerable`1 select, CancellationToken cancellationToken)
   at FabricDCA.AzureTableQueryableCsvUploaderDefault.DeleteOldEntities(TableClient tableClient, DateTime cutoffTime, String partitionKey, StatusEntity statusEntity)
   at FabricDCA.AzureTableQueryableCsvUploaderDefault.DeleteOldLogsHandler(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.TimerQueue.FireNextTimers()
   at System.Threading.UnmanagedThreadPoolWorkItem.ExecuteUnmanagedThreadPoolWorkItem(IntPtr callback, IntPtr state)
   at System.Threading.UnmanagedThreadPoolWorkItem.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

临时缓解方案 #

执行以下PowerShell命令禁用崩溃转储收集：

Set-AzServiceFabricSetting -ResourceGroupName 'API-PROD' -Name 'admiral-prod-api'  -Section 'AzureWinFabCrashDump' -Parameter 'IsEnabled' -Value "false"

内容的提问来源于stack exchange，提问作者ODrobyazko