使用Hibernate批量操作MySQL时触发Java堆内存溢出问题

我用Hibernate向MySQL推送/更新大量数据集，JVM堆内存已设置为4GB，但执行commit操作时仍抛出java.lang.OutOfMemoryError: Java heap space错误。我是Hibernate新手，以下是错误堆栈及相关代码，求解决办法。

错误信息 #

Exception in thread "Thread-7" java.lang.OutOfMemoryError: Java heap space at java.base/java.util.Arrays.copyOf(Arrays.java:3745) at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:172) at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:748) at java.base/java.lang.StringBuilder.append(StringBuilder.java:245) at com.mysql.cj.ClientPreparedQueryBindings.setString(ClientPreparedQueryBindings.java:651) at com.mysql.cj.jdbc.ClientPreparedStatement.setString(ClientPreparedStatement.java:1752) at org.hibernate.type.descriptor.sql.VarcharTypeDescriptor$1.doBind(VarcharTypeDescriptor.java:46) at org.hibernate.type.descriptor.sql.BasicBinder.bind(BasicBinder.java:73) at org.hibernate.type.descriptor.converter.AttributeConverterSqlTypeDescriptorAdapter$1.bind(AttributeConverterSqlTypeDescriptorAdapter.java:88) at org.hibernate.type.AbstractStandardBasicType.nullSafeSet(AbstractStandardBasicType.java:276) at org.hibernate.type.AbstractStandardBasicType.nullSafeSet(AbstractStandardBasicType.java:271) at org.hibernate.type.AbstractSingleColumnStandardBasicType.nullSafeSet(AbstractSingleColumnStandardBasicType.java:39) at org.hibernate.persister.entity.AbstractEntityPersister.dehydrate(AbstractEntityPersister.java:3073) at org.hibernate.persister.entity.AbstractEntityPersister.dehydrate(AbstractEntityPersister.java:3048) at org.hibernate.persister.entity.AbstractEntityPersister$2.bindValues(AbstractEntityPersister.java:3271) at org.hibernate.id.insert.AbstractReturningDelegate.performInsert(AbstractReturningDelegate.java:42) at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3279) at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3885) at org.hibernate.action.internal.EntityIdentityInsertAction.execute(EntityIdentityInsertAction.java:84) at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:645) at org.hibernate.engine.spi.ActionQueue.addResolvedEntityInsertAction(ActionQueue.java:282) at org.hibernate.engine.spi.ActionQueue.addInsertAction(ActionQueue.java:263) at org.hibernate.engine.spi.ActionQueue.addAction(ActionQueue.java:317) at org.hibernate.event.internal.AbstractSaveEventListener.addInsertAction(AbstractSaveEventListener.java:330) at org.hibernate.event.internal.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:287) at org.hibernate.event.internal.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:193) at org.hibernate.event.internal.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:123) at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:194) at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:179) at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.performSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:100) at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:75) at org.hibernate.internal.SessionImpl$$Lambda$432/0x000000084074ac40.accept(Unknown Source)

相关代码 #

public String addNewReport( int runId , Collection<ReportStorage> reportStorage )
{
    int overAllCases = 0 ;
    int overAllPass  = 0 ;
    int overAllFail  = 0 ;
    try {
        if (isRunDetailExistsById(runId)) {
            
            HibernateConfig.getInstance().begin();
            RunDetails runDetails = HibernateConfig.getSession().get( RunDetails.class , runId);
            for( ReportStorage repStore : reportStorage )
            {
                
                repStore.setRun_id( runId );
                repStore.getReportJson().setRun_id(runId);
                overAllCases += repStore.getTotal_cases() ;
                overAllPass  += repStore.getTotal_pass() ;
                overAllFail  += repStore.getTotal_fail() ;
            }
            Collection<ReportStorage> existingReports = runDetails.getReports() ;
            existingReports.addAll( reportStorage ) ;
            runDetails.setReports( existingReports );
            runDetails.setEnd_time( CommonUtils.epocheTime(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS").format(new Date())) );
            String timetaken = CommonUtils.timeConverter(runDetails.getEnd_time() - runDetails.getStart_time() ) ;
            runDetails.setOver_all_cases( Integer.toString(overAllCases ) );
            runDetails.setOver_all_pasess( Integer.toString( overAllPass ) );
            runDetails.setOver_all_failures( Integer.toString( overAllFail ) );
            runDetails.setTotal_time_taken( timetaken );
            runDetails.setRun_status("SUCCESS");
            
            HibernateConfig.getSession().saveOrUpdate(runDetails);
            HibernateConfig.getInstance().commit();
            
            HibernateConfig.close();
            return "SUCCESS" ;
    }else {
        return "Run Id Mismatch" ;
    }
    }
    catch (Exception e) {
        REPORTREPOLOGGER.info("addNewReport-"+ e.toString() );
        HibernateConfig.getInstance().rollback();
        HibernateConfig.close();
        throw new CustomException("addNewReport" , e)  ;
    }
    
}

从错误堆栈和代码来看，内存溢出的核心原因是一次性将所有ReportStorage对象加载到Session缓存中，Hibernate在commit时需要处理所有对象的持久化操作，导致内存占用过高。以下是针对性优化方案：

1. 分批处理数据 #

不要一次性将所有ReportStorage加入集合，拆分成小批次提交并清理Session缓存：

public String addNewReport( int runId , Collection<ReportStorage> reportStorage )
{
    int overAllCases = 0 ;
    int overAllPass  = 0 ;
    int overAllFail  = 0 ;
    try {
        List<ReportStorage> reportList = new ArrayList<>(reportStorage);
        // 定义每批处理数量，可根据内存情况调整
        int batchSize = 500;
        
        HibernateConfig.getInstance().begin();
        Session session = HibernateConfig.getSession();
        RunDetails runDetails = session.get(RunDetails.class, runId);
        
        if(runDetails == null){
            HibernateConfig.close();
            return "Run Id Mismatch";
        }
        
        for(int i=0; i<reportList.size(); i++){
            ReportStorage repStore = reportList.get(i);
            repStore.setRun_id(runId);
            repStore.getReportJson().setRun_id(runId);
            overAllCases += repStore.getTotal_cases();
            overAllPass += repStore.getTotal_pass();
            overAllFail += repStore.getTotal_fail();
            
            runDetails.getReports().add(repStore);
            
            // 每批处理完成后提交并清理缓存
            if((i+1) % batchSize == 0){
                session.flush();
                session.clear();
                // 重新加载RunDetails，避免脱离Session管理
                runDetails = session.get(RunDetails.class, runId);
            }
        }
        
        // 更新RunDetails统计字段
        runDetails.setEnd_time(CommonUtils.epocheTime(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS").format(new Date())));
        String timetaken = CommonUtils.timeConverter(runDetails.getEnd_time() - runDetails.getStart_time());
        runDetails.setOver_all_cases(Integer.toString(overAllCases));
        runDetails.setOver_all_pasess(Integer.toString(overAllPass));
        runDetails.setOver_all_failures(Integer.toString(overAllFail));
        runDetails.setTotal_time_taken(timetaken);
        runDetails.setRun_status("SUCCESS");
        
        session.saveOrUpdate(runDetails);
        HibernateConfig.getInstance().commit();
        HibernateConfig.close();
        return "SUCCESS";
    }
    catch (Exception e) {
        REPORTREPOLOGGER.info("addNewReport-"+ e.toString());
        HibernateConfig.getInstance().rollback();
        HibernateConfig.close();
        throw new CustomException("addNewReport", e);
    }
}

要点：

• 拆分数据集为小批次，每批处理后调用flush()发送SQL到数据库，clear()清空Session缓存释放内存
• 清理后重新加载RunDetails，避免对象处于 detached 状态引发异常

2. 开启Hibernate批量操作配置 #

在Hibernate配置文件中添加以下配置，减少JDBC调用次数：

# 设置批量处理大小，和代码中的batchSize保持一致
hibernate.jdbc.batch_size = 500
# 支持带版本控制数据的批量更新
hibernate.jdbc.batch_versioned_data = true
# 关闭二级缓存（如果不需要）
hibernate.cache.use_second_level_cache = false

3. 优化关联关系加载 #

确保RunDetails和ReportStorage的一对多关联使用懒加载，避免一次性加载所有历史数据：

@Entity
public class RunDetails {
    // 其他字段
    
    @OneToMany(mappedBy = "runDetails", fetch = FetchType.LAZY)
    private Collection<ReportStorage> reports;
    
    // getter/setter
}

4. 优化大文本字段存储 #

错误堆栈显示绑定字符串时内存溢出，说明ReportJson中的文本字段可能过大。将其映射为大文本类型：

@Entity
public class ReportJson {
    // 其他字段
    
    @Lob
    @Column(columnDefinition = "TEXT")
    private String jsonContent;
    
    // getter/setter
}

避免MySQL驱动处理超长VARCHAR时占用过多内存。

内容的提问来源于stack exchange，提问作者prabuBJV