如何解决Spring集成Lettuce时Tomcat9与Netty的线程内存泄漏问题

我使用基于GenericObjectPoolConfig的io.lettuce.core.api.StatefulRedisConnection连接PaaS Redis，本地开发环境启动Tomcat时无异常，但通过Jenkins部署启动时出现线程未关闭的内存泄漏警告，目前不影响系统访问和使用，但需要解决这些警告。

相关配置代码 #

@Bean
public GenericObjectPoolConfig redisPoolConfig() {
    GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
    poolConfig.setMaxTotal(redisProperties.getPool().getMaxTotal());
    poolConfig.setMaxIdle(redisProperties.getPool().getMaxIdle());
    poolConfig.setMinIdle(redisProperties.getPool().getMinIdle());
    poolConfig.setMaxWaitMillis(redisProperties.getPool().getMaxWaitMillis());
    poolConfig.setMinEvictableIdleTimeMillis(redisProperties.getPool().getMinEvictableIdleTimeMillis());
    return poolConfig;
}

@Bean(destroyMethod = "close")
public GenericObjectPool<StatefulRedisConnection<String, Object>> redisObjectPool(RedisClient redisClient, GenericObjectPoolConfig genericObjectPoolConfig) {
    RedisCodecForObject redisCodecForObject = new RedisCodecForObject(Object.class, redisProperties.getKeyPrefix());
    ObjectMapper om = new ObjectMapper();
    om.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
    TypeResolverBuilder<?> typeResolver = new ObjectMapper.DefaultTypeResolverBuilder(ObjectMapper.DefaultTyping.NON_FINAL) {
        {
            init(JsonTypeInfo.Id.CLASS, null);
            inclusion(JsonTypeInfo.As.WRAPPER_ARRAY);
        }
    };
    om.setDefaultTyping(typeResolver);
    redisCodecForObject.setObjectMapper(om);
    return ConnectionPoolSupport.createGenericObjectPool(() -> redisClient.connect(redisCodecForObject), genericObjectPoolConfig);
}

@Bean(destroyMethod = "close")
public StatefulRedisConnection<String, Object> objectRedisConnection(GenericObjectPool<StatefulRedisConnection<String, Object>> redisObjectPool) throws Exception {
    StatefulRedisConnection<String, Object> connection = redisObjectPool.borrowObject();
    return connection;
}

启动警告日志 #

SEVERE [main] org.apache.catalina.core.StandardContext.listenerStart
Exception sending context initialized event to listener instance of
class [ch.qos.logback.ext.spring.web.LogbackConfigListener]
java.lang.IllegalStateException: Web app root system property already set to different value:
'webapp.root' = [/app/tomcat/webapps/ROOT/] instead of [/app/tomcat/webapps/demo/] -

Choose unique values for the 'webAppRootKey' context-param in your web.xml files!

WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [my-app] appears to have started a thread named [lettuce-timer-3-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)

02-Jun-2023 19:35:11.842 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [my-app] appears to have started a thread named [parallel-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)

02-Jun-2023 19:35:11.842 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [my-app] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@381d724e]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@4736bb9f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

pom.xml相关依赖 #

<dependency>
  <groupId>org.springframework.data</groupId>
  <artifactId>spring-data-redis</artifactId>
  <version>1.5.2.RELEASE</version>
</dependency>

<dependency>
  <groupId>org.apache.commons</groupId>
  <artifactId>commons-pool2</artifactId>
  <version>2.9.0</version>
</dependency>

<dependency>
  <groupId>io.lettuce</groupId>
  <artifactId>lettuce-core</artifactId>
  <version>6.1.0.RELEASE</version>
</dependency>

解决方案 #

1. 解决webapp.root重复问题 #

该错误由LogbackConfigListener重复设置webapp.root系统属性导致，可通过以下方式解决：

• 在web.xml中添加唯一的webAppRootKey参数：

<context-param>
    <param-name>webAppRootKey</param-name>
    <param-value>my-app.webapp.root</param-value>
</context-param>

• 或者直接禁用webAppRoot属性设置：

<context-param>
    <param-name>logbackDisableWebAppRoot</param-name>
    <param-value>true</param-value>
</context-param>

2. 解决Lettuce与Netty线程/内存泄漏问题 #

Lettuce基于Netty实现，容器停止时未正确销毁Redis客户端和连接池是核心原因，需从资源管理入手：

（1）完善Redis客户端与连接池的销毁逻辑

当前配置仅给连接池和单个连接设置了destroyMethod="close"，但RedisClient也需要在容器关闭时调用shutdown停止Netty相关线程。同时，直接暴露单个连接Bean会导致连接无法归还到池，建议改为通过连接池管理连接或使用RedisTemplate：

@Bean(destroyMethod = "shutdown")
public RedisClient redisClient(RedisProperties redisProperties) {
    return RedisClient.create(RedisURI.create(redisProperties.getHost(), redisProperties.getPort()));
}

@Bean(destroyMethod = "close")
public GenericObjectPool<StatefulRedisConnection<String, Object>> redisObjectPool(RedisClient redisClient, GenericObjectPoolConfig genericObjectPoolConfig, RedisProperties redisProperties) {
    RedisCodecForObject redisCodecForObject = new RedisCodecForObject(Object.class, redisProperties.getKeyPrefix());
    ObjectMapper om = new ObjectMapper();
    om.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
    TypeResolverBuilder<?> typeResolver = new ObjectMapper.DefaultTypeResolverBuilder(ObjectMapper.DefaultTyping.NON_FINAL) {
        {
            init(JsonTypeInfo.Id.CLASS, null);
            inclusion(JsonTypeInfo.As.WRAPPER_ARRAY);
        }
    };
    om.setDefaultTyping(typeResolver);
    redisCodecForObject.setObjectMapper(om);
    return ConnectionPoolSupport.createGenericObjectPool(() -> redisClient.connect(redisCodecForObject), genericObjectPoolConfig);
}

// 改用RedisTemplate管理连接，避免直接暴露单个连接
@Bean
public RedisTemplate<String, Object> redisTemplate(GenericObjectPool<StatefulRedisConnection<String, Object>> redisObjectPool, ObjectMapper om) {
    RedisTemplate<String, Object> template = new RedisTemplate<>();
    template.setConnectionFactory(new LettuceConnectionFactory() {
        @Override
        protected StatefulRedisConnection<String, byte[]> doCreateConnection() {
            try {
                // 根据实际Codec适配，此处仅为示例
                return (StatefulRedisConnection<String, byte[]>) redisObjectPool.borrowObject();
            } catch (Exception e) {
                throw new RuntimeException("获取Redis连接失败", e);
            }
        }

        @Override
        public void destroy() throws Exception {
            redisObjectPool.close();
        }
    });
    template.setDefaultSerializer(new GenericJackson2JsonRedisSerializer(om));
    template.afterPropertiesSet();
    return template;
}

（2）使用ClientResources管理Netty资源

通过Lettuce的ClientResources统一管理Netty线程池，确保容器关闭时自动清理：

@Bean(destroyMethod = "shutdown")
public ClientResources clientResources() {
    return DefaultClientResources.create();
}

@Bean(destroyMethod = "shutdown")
public RedisClient redisClient(RedisProperties redisProperties, ClientResources clientResources) {
    return RedisClient.create(clientResources, RedisURI.create(redisProperties.getHost(), redisProperties.getPort()));
}

（3）清理Netty ThreadLocal

添加容器销毁监听器，手动清理Netty的InternalThreadLocalMap：

@Component
public class NettyResourceCleanupListener implements ServletContextListener {
    @Override
    public void contextDestroyed(ServletContextEvent sce) {
        // 清理Netty ThreadLocal资源
        io.netty.util.internal.InternalThreadLocalMap.destroy();
        
        // 停止Lettuce的HashedWheelTimer线程
        try {
            Field timerField = io.lettuce.core.resource.DefaultClientResources.class.getDeclaredField("timer");
            timerField.setAccessible(true);
            io.netty.util.HashedWheelTimer timer = (io.netty.util.HashedWheelTimer) timerField.get(null);
            if (timer != null) {
                timer.stop();
            }
        } catch (NoSuchFieldException | IllegalAccessException e) {
            // 忽略异常或记录日志
        }
    }
}

3. 解决parallel-1线程泄漏 #

该线程属于ScheduledThreadPoolExecutor，大概率是老旧版本spring-data-redis内部定时任务未正确关闭导致。建议升级spring-data-redis到与lettuce-core兼容的版本：

• spring-data-redis 1.5.2.RELEASE适配的Lettuce版本为3.x，而你使用的是6.1.0.RELEASE，版本严重不兼容，这是资源泄漏的重要原因。
• 建议将spring-data-redis升级到2.6.x及以上（对应Spring Boot 2.6+），同时保持lettuce-core版本在6.x范围，确保版本匹配。

内容的提问来源于stack exchange，提问作者Felix