Redis Stream订阅触发RedisCommandTimeoutException故障排查求助

我有一个运行了两年的服务，每秒从Redis Stream消费数据并通过WebSocket推送给客户端，此前一直正常，但近两周开始频繁抛出RedisCommandTimeoutException错误，完整堆栈信息如下：

org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: Command timed out after 20 second(s)
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:70)
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41)
at org.springframework.data.redis.connection.lettuce.LettuceReactiveRedisConnection.lambda$translateException$0(LettuceReactiveRedisConnection.java:293)
at reactor.core.publisher.Flux.lambda$onErrorMap$28(Flux.java:7070)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onError(MonoFlatMapMany.java:255)
at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:134)
at io.lettuce.core.RedisPublisher$ImmediateSubscriber.onError(RedisPublisher.java:891)
at io.lettuce.core.RedisPublisher$State.onError(RedisPublisher.java:712)
at io.lettuce.core.RedisPublisher$RedisSubscription.onError(RedisPublisher.java:357)
at io.lettuce.core.RedisPublisher$SubscriptionCommand.onError(RedisPublisher.java:797)
at io.lettuce.core.RedisPublisher$SubscriptionCommand.doOnError(RedisPublisher.java:793)
at io.lettuce.core.protocol.CommandWrapper.completeExceptionally(CommandWrapper.java:128)
at io.lettuce.core.protocol.CommandExpiryWriter.lambda$potentiallyExpire$0(CommandExpiryWriter.java:175)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)
Caused by: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 20 second(s)
at io.lettuce.core.internal.ExceptionFactory.createTimeoutException(ExceptionFactory.java:59)
at io.lettuce.core.protocol.CommandExpiryWriter.lambda$potentiallyExpire$0(CommandExpiryWriter.java:176)
... 8 more

异常特征 #

• 所有订阅的Stream（如"A""B""C"）会同时触发超时，尽管每个Stream使用独立订阅者
• 服务代码、配置均未变更，CPU、堆内存等指标正常
• AWS ElastiCache无异常（已咨询AWS支持），Stream数量与WebSocket连接数无明显增长
• 另外两个向Redis Stream写入数据的服务运行正常

使用的配置（ReactiveRedisConnectionFactory） #

@Bean(name = "streamRedisLettuceConnectionFactory")
public RedisConnectionFactory connectionFactory() {
    log.info("Connecting to redis with configuration : {} ", chpStreamRedisConfiguration);
    LettuceClientConfiguration configuration = getLettuceClientConfiguration();
    return new LettuceConnectionFactory(redisStandaloneConfiguration, configuration);
}

@Bean(name = "streamRedisStandaloneConfiguration")
public RedisStandaloneConfiguration getRedisStandaloneConfiguration() {
    return new RedisStandaloneConfiguration(chpStreamRedisConfiguration.getHost(), chpStreamRedisConfiguration.getPort());
}

@Bean(name = "streamLettuceClientConfiguration")
public LettuceClientConfiguration getLettuceClientConfiguration() {
    return LettucePoolingClientConfiguration.builder()
            .poolConfig(getLettucePoolConfig())
            .commandTimeout(Duration.ofMillis(chpStreamRedisConfiguration.getTimeout()))
            .build();
}

@Bean(name = "streamLettucePoolConfig")
public GenericObjectPoolConfig getLettucePoolConfig() {
    GenericObjectPoolConfig genericObjectPoolConfig = new GenericObjectPoolConfig();
    genericObjectPoolConfig.setMaxTotal(chpStreamRedisConfiguration.getConnectionPool().getMaxTotal());
    genericObjectPoolConfig.setMaxIdle(chpStreamRedisConfiguration.getConnectionPool().getMaxTotal());
    genericObjectPoolConfig.setMinIdle(chpStreamRedisConfiguration.getConnectionPool().getMinIdle());
    return genericObjectPoolConfig;
}

依赖版本 #

• spring-data-redis:2.3.9.RELEASE
• lettuce-core:6.1.2.RELEASE
• spring-boot:5.2.18.RELEASE

已尝试操作 #

• 将lettuce升级至6.2.2.RELEASE，问题未解决
• 调试io.lettuce.core.protocol得到以下信息：

Completing command LatencyMeteredCommand [type=XREAD, output=StreamReadOutput [output=[], error='Command timed out after 20 second(s)'], commandType=io.lettuce.core.RedisPublisher$SubscriptionCommand]

20秒超时已足够长，不希望增加超时时间，恳请协助排查故障原因与解决办法。

内容的提问来源于stack exchange，提问作者Doron Shaul