Quarkus Redis客户端切换高可用模式后负载测试报错求助

错误日志 #

level=ERROR time="07-03-2023 20:54:35" traceId="5a65a85d2f6ccaa8" logger=GlobalExceptionMapper message="GlobalExceptionMapper: java.util.concurrent.CompletionException: io.vertx.core.impl.NoStackTraceThrowable: No more endpoints in chain.
        at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:73)
        at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:61)
        at io.quarkus.redis.client.runtime.RedisClientImpl.await(RedisClientImpl.java:1026)
        at io.quarkus.redis.client.runtime.RedisClientImpl.set(RedisClientImpl.java:672)
        at io.quarkus.redis.client.RedisClient_761b9a6e5f634178e3291b09c1921f229025da0c_Synthetic_ClientProxy.set(RedisClient_761b9a6e5f634178e3291b09c1921f229025da0c_Synthetic_ClientProxy.zig:2298)
        at <package>.services.SequenceService.isKeyPresentForDuplicityCheck(SequenceService.java:34)
        at <package>.services.SequenceService_ClientProxy.isKeyPresentForDuplicityCheck(SequenceService_ClientProxy.zig:256)
        at <package>.services.AccountsApiServiceImpl.createCustomerAccount(AccountsApiServiceImpl.java:128)
        at <package>.services.AccountsApiServiceImpl_Subclass.createCustomerAccount$$superforward1(AccountsApiServiceImpl_Subclass.zig:197)
        at <package>.services.AccountsApiServiceImpl_Subclass$$function$$2.apply(AccountsApiServiceImpl_Subclass$$function$$2.zig:33)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:54)
        at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorBase.invokeInOurTx(TransactionalInterceptorBase.java:132)
        at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorBase.invokeInOurTx(TransactionalInterceptorBase.java:103)
        at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorRequired.doIntercept(TransactionalInterceptorRequired.java:38)
        at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorBase.intercept(TransactionalInterceptorBase.java:57)
        at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorRequired.intercept(TransactionalInterceptorRequired.java:32)
        at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorRequired_Bean.intercept(TransactionalInterceptorRequired_Bean.zig:340)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:41)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:41)
        at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:32)
        at <package>.services.AccountsApiServiceImpl_Subclass.createCustomerAccount(AccountsApiServiceImpl_Subclass.zig:404)
        at <package>.services.AccountsApiServiceImpl_ClientProxy.createCustomerAccount(AccountsApiServiceImpl_ClientProxy.zig:659)
        at <package>.resources.AccountsApi.createCustomerAccount(AccountsApi.java:48)
        at <package>.resources.AccountsApi$quarkusrestinvoker$createCustomerAccount_0b915408532d6a09a8c6a63ae490a49fe854ecb6.invoke(AccountsApi$quarkusrestinvoker$createCustomerAccount_0b915408532d6a09a8c6a63ae490a49fe854ecb6.zig:39)
        at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
        at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:7)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:141)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$13.runWith(VertxCoreRecorder.java:543)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:829)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:567)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: io.vertx.core.impl.NoStackTraceThrowable: No more endpoints in chain.

负载测试方式 #

• 通过JMeter发送30000个请求，在请求处理过程中逐个删除Redis主节点（需等前一个删除的Pod完全重启后再操作），删除命令：kubectl -n <namespace> delete pod redis-0
• Sentinel可完成故障转移并选出新主节点，多数场景仅少量请求失败，短时间后恢复正常；但部分场景下JMeter会挂起，后续所有请求均报错io.vertx.core.impl.NoStackTraceThrowable: No more endpoints in chain

系统配置 #

• Quarkus进程连接Redis的配置均存储在application.properties中
• Redis、Sentinel配置见对应截图

已尝试方案 #

• 调整Quarkus Redis参数进行负载测试，20次测试仅1次未出现问题
• 考虑Vertx Redis客户端重连方案，但改造成本过高已放弃

Quarkus进程详情 #

• API预期TPS约300
• 测试时开发环境仅运行1个Pod，生产环境部署5个
• Quarkus版本：2.3.0.Final
• Redis依赖：

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-redis-client</artifactId>
</dependency>

• Redis配置（会被K8s部署文件覆盖）：

quarkus.redis.hosts=redis://sentinel:5000
quarkus.redis.client-type=sentinel
quarkus.redis.password=
quarkus.redis.timeout=5S
quarkus.redis.max-pool-size=50
quarkus.redis.max-pool-waiting=5000
quarkus.redis.pool-cleaner-interval=5S
quarkus.redis.pool-recycle-timeout=5S
quarkus.redis.reconnect-attempts=5
quarkus.redis.reconnect-interval=30S
quarkus.redis.max-waiting-handlers=20

更新内容 #

• 通过存活探针崩溃Redis主节点进行测试，仍出现相同错误
• 增加Redis超时参数后问题依旧：

quarkus.redis.timeout.connect=60_000
quarkus.redis.timeout.read=70_000
quarkus.redis.timeout.write=80_000
quarkus.redis.timeout=100S

• 使用Jedis库可实现自动重连，推测Quarkus Redis客户端存在问题

现寻求解决该问题的方案，包括负载测试方法和配置优化建议。

内容的提问来源于stack exchange，提问作者KVS