Tomcat高CPU占用及OutOfMemory问题排查求助

应用栈 #

• Ubuntu 18.04.6 LTS
• JDK 1.8.0_201
• Spring Boot: 2.6.6
• Oracle 12c
• Tomcat 8.5.31

问题现象 #

添加了每6小时运行的批量任务，从数据库生成6个最大12MB的Excel文件。任务执行时CPU飙升，几天后CPU无法回落至5-10%的正常水平，持续升高最终触发java.lang.OutOfMemoryError: GC overhead limit exceeded异常。优化代码后问题仍会出现，仅延迟了触发时间。

CPU使用率截图：

无法在本地或其他服务器复现该问题，寻求原因分析。

相关配置代码 #

异步任务配置 #

/**
* Used to cache few tables using CompletableFuture.
*/
@Configuration
@EnableAsync
public class AsyncConfig extends AsyncConfigurerSupport {

    @Override
    public Executor getAsyncExecutor() {
        ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
        taskExecutor.setCorePoolSize(5);
        taskExecutor.setMaxPoolSize(25);
        taskExecutor.setQueueCapacity(25);
        taskExecutor.setThreadNamePrefix("Async-thread-");
        taskExecutor.initialize();
        return taskExecutor;
    }
}

调度任务配置 #

/**
 * 
 * Configures the scheduler to allow multiple concurrent pools.
 * Used for Batches
 */
@Configuration
@EnableScheduling
public class SchedulerConfig implements SchedulingConfigurer
{  
    @Override
    public void configureTasks(ScheduledTaskRegistrar taskRegistrar)
    {
        ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler();
        threadPoolTaskScheduler.setPoolSize(25);
        threadPoolTaskScheduler.setThreadNamePrefix("task-pool-");
        threadPoolTaskScheduler.initialize();
        taskRegistrar.setTaskScheduler(threadPoolTaskScheduler);
    }    
}

Excel批量任务代码 #

@EnableBatchProcessing
@Configuration
public class ExceltReportBatchProcessor {
...

@Scheduled(cron = "0 0 0/6 * * *", zone = "UTC")
    public void printMessage() {
            JobParameters statusJobParameters = new JobParametersBuilder().addLong("time", System.currentTimeMillis())
                    .toJobParameters();
            jobLauncher.run(excelReportNotify(), statusJobParameters);
    }
}

Tomcat停止时的警告信息 #

WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [excel-app] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:327)
07-Jan-2023 03:02:27.855 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [excel-app] appears to have started a thread named [task-pool-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)
07-Jan-2023 03:02:27.860 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [excel-app] appears to have started a thread named [task-pool-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
...
...

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007faa3008b800 nid=0x1876 in Object.wait() [0x00007fa9fa9a6000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
    - locked <0x00000000c0044f40> (a java.lang.ref.Reference$Lock)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"VM Thread" os_prio=0 tid=0x00007faa30082000 nid=0x1875 runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007faa30021800 nid=0x186d runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007faa30023800 nid=0x186e runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007faa30025000 nid=0x186f runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007faa30027000 nid=0x1870 runnable 

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007faa30028800 nid=0x1871 runnable 

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007faa3002a800 nid=0x1872 runnable 

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007faa3002c000 nid=0x1873 runnable 

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007faa3002e000 nid=0x1874 runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007faa300d2000 nid=0x187e waiting on condition 

JNI global references: 2470

Heap
 PSYoungGen      total 328192K, used 295629K [0x00000000eab00000, 0x00000000ffd00000, 0x0000000100000000)
  eden space 312320K, 94% used [0x00000000eab00000,0x00000000fcba3790,0x00000000fdc00000)
  from space 15872K, 0% used [0x00000000fed80000,0x00000000fed90000,0x00000000ffd00000)
  to   space 16896K, 0% used [0x00000000fdc00000,0x00000000fdc00000,0x00000000fec80000)
 ParOldGen       total 427520K, used 169725K [0x00000000c0000000, 0x00000000da180000, 0x00000000eab00000)
  object space 427520K, 39% used [0x00000000c0000000,0x00000000ca5bf630,0x00000000da180000)
 Metaspace       used 119147K, capacity 124996K, committed 125132K, reserved 1159168K
  class space    used 14369K, capacity 15321K, committed 15360K, reserved 1048576K

本地SpringBoot应用每5分钟运行Excel批量任务截图：

可能的原因与排查方向 #

1. 线程池资源泄漏
   从Tomcat停止时的警告可以看到，task-pool-*线程和Oracle JDBC的BlockReleaser线程未被正常销毁，长期运行会导致线程堆积，占用CPU和内存资源。

   • 给ThreadPoolTaskScheduler配置销毁方法，确保Bean销毁时关闭线程池：

     @Bean(destroyMethod = "shutdown")
     public ThreadPoolTaskScheduler threadPoolTaskScheduler() {
         ThreadPoolTaskScheduler scheduler = new ThreadPoolTaskScheduler();
         scheduler.setPoolSize(25);
         scheduler.setThreadNamePrefix("task-pool-");
         return scheduler;
     }
     

   • 升级Oracle JDBC驱动到适配JDK1.8的最新稳定版本，检查连接池配置（如HikariCP）的maximumPoolSize、idleTimeout等参数，确保连接被及时回收。

2. Excel生成过程中的内存泄漏
   若生成Excel时未正确释放POI相关资源（如Workbook、Sheet等），会导致内存无法回收，长期积累引发OOM。

   • 用try-with-resources语法包裹POI对象，确保资源自动释放：

     try(XSSFWorkbook workbook = new XSSFWorkbook()) {
         // 生成Excel逻辑
     } catch (IOException e) {
         // 异常处理
     }
     

   • 检查是否存在静态集合持有Excel相关对象的引用，导致GC无法回收。

3. GC配置不合理
   从Heap信息看，Young区使用率高达94%，Old区占用39%，且出现GC overhead异常，说明GC无法有效回收内存或内存分配不足。

   • 调整JVM堆内存参数（如-Xms1g -Xmx2g），调整新生代和老年代比例（如-XX:NewRatio=2），减少GC频率。
   • 开启GC日志（-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/path/to/gc.log），分析GC频率和回收效率。

4. 数据库查询与数据处理问题
   生产环境数据量可能远大于本地，批量查询若一次性加载全量数据到内存，会导致内存占用过高。

   • 采用分页查询，分批处理数据，避免一次性加载全量数据。
   • 检查是否存在未关闭的ResultSet或Statement，导致数据库资源泄漏。

5. 环境差异导致的问题
   无法在本地复现可能是生产环境数据量、并发量、硬件资源与本地不同，或存在其他抢占资源的进程。

   • 用jstack导出线程快照，分析CPU占用高的线程执行逻辑，排查死循环、阻塞问题。
   • 用jmap导出堆快照（jmap -dump:format=b,file=heap.hprof <pid>），用MAT工具分析内存泄漏点。

内容的提问来源于stack exchange，提问作者Tusar