Heroku部署的.NET Core应用连接Heroku Data Redis超时问题排查

问题 #

在Heroku使用Docker托管的.NET Core应用中，从Heroku Data Redis获取数据时出现超时错误，本地缓存功能正常，但部署到Heroku后报错：

StackExchange.Redis.RedisTimeoutException: The timeout was reached before the message could be written to the output buffer, and it was not sent, command=HMGET, timeout: 60000, inst: 0, qu: 1, qs: 0, aw: False, bw: CheckingForTimeout, serverEndpoint: [amazon aws host:port], mc: 1/1/0, mgr: 10 of 10 available, clientName: d3d51ae0-3bd0-4434-8767-2028ec9f6c41(SE.Redis-v2.6.66.47313), IOCP: (Busy=0,Free=1000,Min=10,Max=1000), WORKER: (Busy=0,Free=32767,Min=10,Max=32767), POOL: (Threads=5,QueuedItems=0,CompletedItems=986), v: 2.6.66.47313

堆栈跟踪 #

2023-01-30T16:49:20.014624+00:00 app[web.1]: Stack Trace:
2023-01-30T16:49:20.014625+00:00 app[web.1]:
2023-01-30T16:49:20.014625+00:00 app[web.1]: at Microsoft.Extensions.Caching.StackExchangeRedis.RedisExtensions.HashMemberGetAsync(IDatabase cache, String key, String[] members)
2023-01-30T16:49:20.014626+00:00 app[web.1]: at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefreshAsync(String key, Boolean getData, CancellationToken token)
2023-01-30T16:49:20.014626+00:00 app[web.1]: at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAsync(String key, CancellationToken token)
2023-01-30T16:49:20.014626+00:00 app[web.1]: at SudokuCollective.Cache.CacheService.GetByUserNameWithCacheAsync(IUsersRepository`1 repo, IDistributedCache cache, String cacheKey, DateTime expiration, ICacheKeys keys, String username, String license, IResult result) in /src/SudokuCollective.Api/SudokuCollective.Cache/CacheService.cs:line 1029
2023-01-30T16:49:20.014627+00:00 app[web.1]: at SudokuCollective.Data.Services.AuthenticateService.AuthenticateAsync(ILoginRequest request) in /src/SudokuCollective.Api/SudokuCollective.Data/Services/AuthenticateService.cs:line 89
2023-01-30T16:49:20.014627+00:00 app[web.1]: at SudokuCollective.Api.Controllers.V1.LoginController.PostAsync(LoginRequest request) in /src/SudokuCollective.Api/SudokuCollective.Api/Controllers/V1/LoginController.cs:line 80, License: , AppId: 0, RequestorId: 0
2023-01-30T16:49:20.014638+00:00 app[web.1]: StackExchange.Redis.RedisTimeoutException: The timeout was reached before the message could be written to the output buffer, and it was not sent, command=HMGET, timeout: 60000, inst: 0, qu: 0, qs: 0, aw: False, bw: CheckingForTimeout, serverEndpoint: ec2-3-221-27-134.compute-1.amazonaws.com:19570, mc: 1/1/0, mgr: 10 of 10 available, clientName: d3d51ae0-3bd0-4434-8767-2028ec9f6c41(SE.Redis-v2.6.66.47313), IOCP: (Busy=0,Free=1000,Min=10,Max=1000), WORKER: (Busy=1,Free=32766,Min=10,Max=32767), POOL: (Threads=5,QueuedItems=0,CompletedItems=1277), v: 2.6.66.47313
2023-01-30T16:49:20.014638+00:00 app[web.1]: at Microsoft.Extensions.Caching.StackExchangeRedis.RedisExtensions.HashMemberGetAsync(IDatabase cache, String key, String[] members)
2023-01-30T16:49:20.014640+00:00 app[web.1]: at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefreshAsync(String key, Boolean getData, CancellationToken token)
2023-01-30T16:49:20.014642+00:00 app[web.1]: at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAsync(String key, CancellationToken token)
2023-01-30T16:49:20.014643+00:00 app[web.1]: at SudokuCollective.Cache.CacheService.GetByUserNameWithCacheAsync(IUsersRepository`1 repo, IDistributedCache cache, String cacheKey, DateTime expiration, ICacheKeys keys, String username, String license, IResult result) in /src/SudokuCollective.Api/SudokuCollective.Cache/CacheService.cs:line 1029
2023-01-30T16:49:20.014644+00:00 app[web.1]: at SudokuCollective.Data.Services.AuthenticateService.AuthenticateAsync(ILoginRequest request) in /src/SudokuCollective.Api/SudokuCollective.Data/Services/AuthenticateService.cs:line 89
2023-01-30T16:49:20.014655+00:00 app[web.1]: at SudokuCollective.Api.Controllers.V1.LoginController.PostAsync(LoginRequest request) in /src/SudokuCollective.Api/SudokuCollective.Api/Controllers/V1/LoginController.cs:line 80

当前缓存配置（Startup.cs） #

string cacheConnectionString = "";

ConfigurationOptions options;

if (!_environment.IsStaging())
{
    cacheConnectionString = Configuration.GetConnectionString("CacheConnection");

    options = ConfigurationOptions.Parse(Configuration.GetConnectionString("CacheConnection"));
}
else
{
    options = GetHerokuRedisConfigurationOptions();

    cacheConnectionString = options.ToString();
}

services.AddStackExchangeRedisCache(redisOptions => {
    redisOptions.Configuration = cacheConnectionString;
    redisOptions.ConfigurationOptions = options;
});

GetHerokuRedisConfigurationOptions方法 #

private static ConfigurationOptions GetHerokuRedisConfigurationOptions()
{
    // Get the connection string from the ENV variables
    var redisUrlString = Environment.GetEnvironmentVariable("REDIS_URL");

    // parse the connection string
    var redisUri = new Uri(redisUrlString);
    var userInfo = redisUri.UserInfo.Split(':');

    var config = new ConfigurationOptions
    {
        EndPoints = { { redisUri.Host, redisUri.Port } },
        Password = userInfo[1],
        AbortOnConnectFail = false,
        ConnectRetry = 3,
        ConnectTimeout = 60000,
        SyncTimeout = 60000,
        AsyncTimeout = 60000,
        Ssl = true,
        SslProtocols = System.Security.Authentication.SslProtocols.Tls12,
    };

    // Add the tls certificate
    config.CertificateSelection += delegate
    {
        var cert = new X509Certificate2("[path to pfx file]");
        return cert;
    };

    return config;
}

已尝试将超时时间设置为60000毫秒，但问题仍未解决，需要正确注册IDistributedCache服务以避免超时错误。

解决方案 #

1. 修复配置冲突问题 #

当前代码同时设置redisOptions.Configuration和redisOptions.ConfigurationOptions会引发配置冲突，StackExchange.Redis优先使用ConfigurationOptions，冗余配置可能导致异常。修改Startup.cs中的缓存注册代码，仅保留ConfigurationOptions：

services.AddStackExchangeRedisCache(redisOptions => {
    redisOptions.ConfigurationOptions = options;
});

2. 移除不必要的证书配置 #

Heroku Data Redis的SSL连接无需手动指定客户端证书，CertificateSelection委托会干扰默认验证流程，直接删除该代码块：

// 删除以下代码
config.CertificateSelection += delegate
{
    var cert = new X509Certificate2("[path to pfx file]");
    return cert;
};

3. 优化连接池与线程配置 #

适配Heroku容器环境，调整连接池大小和线程参数，避免连接闲置或线程耗尽：

var config = new ConfigurationOptions
{
    EndPoints = { { redisUri.Host, redisUri.Port } },
    Password = userInfo[1],
    AbortOnConnectFail = false,
    ConnectRetry = 3,
    ConnectTimeout = 60000,
    SyncTimeout = 30000,
    AsyncTimeout = 30000,
    Ssl = true,
    SslProtocols = System.Security.Authentication.SslProtocols.Tls12,
    PoolSize = 20, // 增加连接池大小
    MinIoThreads = 20,
    MinWorkerThreads = 20,
    KeepAlive = 60, // 保持连接活跃，避免闲置断开
    DefaultDatabase = 0
};

4. 增加环境变量校验 #

确保REDIS_URL环境变量正确获取，避免Uri解析失败：

var redisUrlString = Environment.GetEnvironmentVariable("REDIS_URL");
if (string.IsNullOrEmpty(redisUrlString))
{
    throw new InvalidOperationException("REDIS_URL环境变量未设置");
}

5. 强制使用异步调用 #

业务代码中优先使用Redis缓存的异步方法（如GetAsync），避免同步阻塞导致线程池资源耗尽。

内容的提问来源于stack exchange，提问作者Joe King