StackExchange.Redis超时异常致ASP.NET MVC网站请求阻塞求助
ASP.NET MVC网站使用StackExchange.Redis高流量下请求卡顿问题
我在基于.NET Framework的ASP.NET MVC网站中使用StackExchange.Redis NuGet包,当流量稍有增长时,网站无法加载,请求卡在IIS工作进程中。
Redis连接助手代码
public class RedisCacheHelper { public static T Get<T>(string cacheKey) where T : class { var cache = RedisConnectorHelper.Connection.GetDatabase(); var itemValue = cache.StringGetAsync(cacheKey).Result; T item = null; if (!string.IsNullOrEmpty(itemValue)) item = JsonConvert.DeserializeObject<T>(itemValue); //T item = cache.StringGet(cacheKey) as T; return item; } public static void Set(string cacheKey, object ObjValue, int cacheMinutes = 60) { var cache = RedisConnectorHelper.Connection.GetDatabase(); if (!string.IsNullOrEmpty(cacheKey) && ObjValue != null) { var Expiration = TimeSpan.FromMinutes(cacheMinutes); var redisVal = JsonConvert.SerializeObject(ObjValue); cache.StringSetAsync(cacheKey, redisVal, Expiration, When.NotExists, CommandFlags.FireAndForget); } } public static void ClearRedisCache(string Key = null) { if (!string.IsNullOrEmpty(Key)) { var cache = RedisConnectorHelper.Connection.GetDatabase(); var endpoints = RedisConnectorHelper.Connection.GetEndPoints(); var server = RedisConnectorHelper.Connection.GetServer(endpoints[0]); var keys = server.Keys(); var _ObjKey = keys.Where(k => k.ToString().ToLower().Equals(Key.ToLower())).FirstOrDefault(); if (!string.IsNullOrEmpty(_ObjKey)) cache.KeyDelete(_ObjKey); } else { var endpoints = RedisConnectorHelper.Connection.GetEndPoints(); var server = RedisConnectorHelper.Connection.GetServer(endpoints[0]); server.FlushAllDatabases(); } } } public class RedisConnectorHelper { static RedisConnectorHelper() { RedisConnectorHelper.lazyConnection = new Lazy<ConnectionMultiplexer>(() => { return ConnectionMultiplexer.Connect("SomeConnection"); }); } private static Lazy<ConnectionMultiplexer> lazyConnection; public static ConnectionMultiplexer Connection { get { return lazyConnection.Value; } } }
服务器事件查看器捕获的异常信息
Event code: 3005 Event message: An unhandled exception has occurred. Event time: 12/3/2022 4:53:53 AM Event time (UTC): 12/3/2022 2:53:53 AM Event ID: d5f0dd48b0554c508c5111b92599081e Event sequence: 56886 Event occurrence: 5 Event detail code: 0 Application information: Application domain: /LM/W3SVC/3/ROOT-1-133144634776972114 Trust level: Full Application Virtual Path: / Application Path: Some Path Machine name: EC2AMAZ-VG4448R Process information: Process ID: 8412 Process name: w3wp.exe Account name: EC2AMAZ-VG4448R\MyASPMVC_web Exception information: Exception type: RedisTimeoutException Exception message: Timeout performing SETEX (5000ms), next: GET AvailableResources, inst: 1345, qu: 0, qs: 852, aw: False, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, serverEndpoint: MyASPMVC-redis.gfqxrw.0001.use1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: EC2AMAZ-VG4448R, IOCP: (Busy=0,Free=1000,Min=16,Max=1000), WORKER: (Busy=647,Free=32120,Min=16,Max=32767), v: 2.2.50.36290 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) at MyASPMVCRevamp.Service.RedisCacheHelper.Set(String cacheKey, Object ObjValue, Int32 cacheMinutes) in D:\Work\\MyASPMVCFronts\MyASPMVCRevamp.sln\Master\MyASPMVCRevamp.Service\RedisService.cs:line 40 at MyASPMVCRevamp.Service.GeneralServices.GetStaticPageByID(Int32 ID) in D:\Work\\MyASPMVCFronts\MyASPMVCRevamp.sln\Master\MyASPMVCRevamp.Service\GeneralServices.cs:line 353 at lambda_method(Closure , ControllerBase , Object[] ) at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters) at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c.<BeginInvokeSynchronousActionMethod>b__9_0(IAsyncResult asyncResult, ActionInvocation innerInvokeState) at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`2.CallEndDelegate(IAsyncResult asyncResult) at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult) at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<InvokeActionMethodFilterAsynchronouslyRecursive>b__11_0() at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_1.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2() at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_1.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2() at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult) at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass3_6.<BeginInvokeAction>b__3() at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass3_1.<BeginInvokeAction>b__5(IAsyncResult asyncResult) at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult) at System.Web.Mvc.Controller.<>c.<BeginExecuteCore>b__152_1(IAsyncResult asyncResult, ExecuteCoreState innerState) at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult) at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult) at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult) at System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult) at System.Web.Mvc.MvcHandler.<>c.<BeginProcessRequest>b__20_1(IAsyncResult asyncResult, ProcessRequestState innerState) at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult) at System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult) at System.Web.Mvc.HttpHandlerUtil.ServerExecuteHttpHandlerWrapper.<>c__DisplayClass5_0.<Wrap>b__0() at System.Web.Mvc.HttpHandlerUtil.ServerExecuteHttpHandlerWrapper.Wrap[TResult](Func`1 func) at System.Web.HttpServerUtility.ExecuteInternal(IHttpHandler handler, TextWriter writer, Boolean preserveForm, Boolean setPreviousPage, VirtualPath path, VirtualPath filePath, String physPath, Exception error, String queryStringOverride) Request information: Request URL: https://www.MyASPMVC.com:443/en-us/elogin Request path: /en-us/elogin User host address: User: Is authenticated: False Authentication Type: Thread account name: EC2AMAZ-VG4448R\MyASPMVC_web Thread information: Thread ID: 640 Thread account name: EC2AMAZ-VG4448R\MyASPMVC_web Is impersonating: False Stack trace: at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) at MyASPMVCRevamp.Service.RedisCacheHelper.Set(String cacheKey, Object ObjValue, Int32 cacheMinutes) in D:\Work\\MyASPMVCFronts\MyASPMVCRevamp.sln\Master\MyASPMVCRevamp.Service\RedisService.cs:line 40 at MyASPMVCRevamp.Service.GeneralServices.GetStaticPageByID(Int32 ID) in D:\Work\\MyASPMVCFronts\MyASPMVCRevamp.sln\Master\MyASPMVCRevamp.Service\GeneralServices.cs:line 353 at lambda_method(Closure , ControllerBase , Object[] ) at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters) at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c.<BeginInvokeSynchronousActionMethod>b__9_0(IAsyncResult asyncResult, ActionInvocation innerInvokeState) at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`2.CallEndDelegate(IAsyncResult asyncResult) at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult) at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<InvokeActionMethodFilterAsynchronouslyRecursive>b__11_0() at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_1.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2() at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_1.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2() at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult) at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass3_6.<BeginInvokeAction>b__3() at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass3_1.<BeginInvokeAction>b__5(IAsyncResult asyncResult) at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult) at System.Web.Mvc.Controller.<>c.<BeginExecuteCore>b__152_1(IAsyncResult asyncResult, ExecuteCoreState innerState) at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult) at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult) at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult) at System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult) at System.Web.Mvc.MvcHandler.<>c.<BeginProcessRequest>b__20_1(IAsyncResult asyncResult, ProcessRequestState innerState) at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult) at System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult) at System.Web.Mvc.HttpHandlerUtil.ServerExecuteHttpHandlerWrapper.<>c__DisplayClass5_0.<Wrap>b__0() at System.Web.Mvc.HttpHandlerUtil.ServerExecuteHttpHandlerWrapper.Wrap[TResult](Func`1 func) at System.Web.HttpServerUtility.ExecuteInternal(IHttpHandler handler, TextWriter writer, Boolean preserveForm, Boolean setPreviousPage, VirtualPath path, VirtualPath filePath, String physPath, Exception error, String queryStringOverride) Custom event details:
问题分析与解决方案
从异常信息来看,核心问题是RedisTimeoutException,请求队列堆积(qs: 852),工作线程繁忙(WORKER: Busy=647),结合代码,主要修复点如下:
1. 替换异步方法的同步阻塞调用
Get方法中使用.Result阻塞线程,高并发下会耗尽线程池资源,引发队列堆积。改为异步方法并使用await调用:
public static async Task<T> GetAsync<T>(string cacheKey) where T : class { var cache = RedisConnectorHelper.Connection.GetDatabase(); var itemValue = await cache.StringGetAsync(cacheKey); T item = null; if (!string.IsNullOrEmpty(itemValue)) item = JsonConvert.DeserializeObject<T>(itemValue); return item; }
控制器中调用示例:
var data = await RedisCacheHelper.GetAsync<YourDataType>(cacheKey);
2. 优化Redis连接配置
在连接字符串中添加关键参数,提升稳定性:
return ConnectionMultiplexer.Connect("SomeConnection,connectTimeout=10000,syncTimeout=10000,abortOnConnectFail=false,keepAlive=60,responseTimeout=10000");
3. 避免全量遍历Redis键
ClearRedisCache方法中server.Keys()全量遍历会严重阻塞Redis,删除单个键直接调用KeyDeleteAsync即可:
public static async Task ClearRedisCacheAsync(string key = null) { if (!string.IsNullOrEmpty(key)) { var cache = RedisConnectorHelper.Connection.GetDatabase(); await cache.KeyDeleteAsync(key); } else { var endpoints = RedisConnectorHelper.Connection.GetEndPoints(); var server = RedisConnectorHelper.Connection.GetServer(endpoints[0]); await server.FlushAllDatabasesAsync(); } }
4. 调整线程池配置
在Global.asax的Application_Start中设置线程池最小线程数,避免线程不足:
protected void Application_Start() { int minWorkerThreads = Environment.ProcessorCount * 10; int minIoThreads = Environment.ProcessorCount * 10; ThreadPool.SetMinThreads(minWorkerThreads, minIoThreads); // 其他初始化代码 }
5. 升级StackExchange.Redis版本
当前使用的2.2.50版本存在已知性能问题,升级到最新稳定版可修复大量bug。
内容的提问来源于stack exchange,提问作者Ahmed Kamal
相关产品推荐
相关产品推荐

