大数据库检索中内存膨胀问题的排查与解决

需求背景 #

需全量检索某数据库中近400万行（约10列）的数据，写入HTTP响应的OutputStream供下游应用反序列化；采用Jackson将数据序列化为JSON输出。应用基于Spring Boot构建，使用Spring Data JPA（搭配Hibernate）操作数据库，无关联表查询。

初始实现及问题 #

初始代码 #

try (Stream<TableEntry> tableEntryStream = tableEntryRepository.streamAllBy();
        SequenceWriter outputStreamSequenceWriter = jsonMapper.writerFor(MappedTableEntry.class)
                .writeValues(request.getOutputStream());
        Session hibernateSession = entityManager.unwrap(Session.class)) {
    Iterator<TableEntry> tableEntryIterator = tableEntryStream.iterator();
    log.debug("Opened database connection.");

    for (int i = 0; ; i++) {
        if (!tableEntryIterator.hasNext()) {
            break;
        }
        TableEntry nextTableEntry = tableEntryIterator.next();

        // 尝试从EntityManager分离实体并从Hibernate一级缓存逐出，期望对象可被GC回收
        hibernateSession.evict(nextTableEntry);
        this.entityManager.detach(nextTableEntry);

        if (i % 1000 == 0) {
            log.debug("Writing element n={} to OutputStream using jsonGenerator.", i);
        }
        MappedTableEntry nextMappedTableEntry = tableEntryMapper.convertTableEntryToMappedTableEntry(nextTableEntry);
        outputStreamSequenceWriter.write(nextMappedTableEntry);
        if (i % 1000 == 0) {
            log.debug("Finished pushing element n={} to OutputStream.", i);
        }
    }        
} catch (IOException ioException) {
    throw new UncheckedIOException(ioException);
}

出现的问题 #

启动检索后，应用内存占用持续攀升至3-4GB，超出Docker容器3GB的内存限制导致崩溃。性能分析显示，内存主要消耗在字符串、字节数组、TableEntry对象及EntityDeleteAction对象上。

优化后实现及效果 #

优化后代码（基于M. Deinum建议） #

try (Stream<TableEntry> tableEntryStream = tableEntryRepository.streamAllBy();
        OutputStream oos = new ObjectOutputStream(request.getOutputStream());
        SequenceWriter outputStreamSequenceWriter = jsonMapper.writerFor(MappedTableEntry.class)
                .writeValues(oos)) {
    log.debug("Opened database connection.");
    final AtomicInteger counter = new AtomicInteger();
    final AtomicInteger errorCounter = new AtomicInteger();
    tableEntryStream.map(tableEntryMapper::convertTableEntryToMappedTableEntry).forEach(mappedTableEntry -> {
        try {
            outputStreamSequenceWriter.write(mappedTableEntry);

            int count = counter.incrementAndGet();
            if (count % CLEAR_ENTITY_MANAGER_THRESHOLD == 0) {
                log.debug("Clearing entityManager after {} writes to OutputStream.", count);
                outputStreamSequenceWriter.flush();
                entityManager.clear();
            }
            errorCounter.set(0);
        } catch (IOException ex) {
            errorCounter.incrementAndGet();
            if (errorCounter.get() > ERROR_THRESHOLD) {
                throw new UncheckedIOException(ex);
            } else {
                log.warn(
                        "Encountered IOException {}; Processing will continue until {} more exceptions are encountered.",
                        ex.getMessage(), errorCounter.get() - ERROR_THRESHOLD, ex);
            }
        }
    });
} catch (IOException ioException) {
    throw new UncheckedIOException(ioException);
}

优化效果 #

优化后的方案有效控制了内存占用，满足容器内存限制，可稳定运行。

内容的提问来源于stack exchange，提问作者Alexander Kirk Jørgensen