如何在Spring Batch中使用流式处理或分页功能？

一、用你写的JPA Stream方式完全可行 #

你可以在Spring Batch里直接调用这个自定义的Repository Stream方法，核心是写一个简单的自定义ItemReader来封装Stream的读取逻辑，同时要注意Stream的资源释放：

@Bean
public ItemReader<Employee> employeeStreamReader(EmployeeRepository employeeRepository) {
    return new ItemReader<>() {
        private Stream<Employee> employeeStream;
        private Iterator<Employee> iterator;

        @Override
        public Employee read() throws Exception {
            if (employeeStream == null) {
                // 替换成你的实际日期参数
                employeeStream = employeeRepository.findByHireDateBetween(LocalDate.of(2000, 1, 1), LocalDate.of(2024, 1, 1));
                iterator = employeeStream.iterator();
            }
            return iterator.hasNext() ? iterator.next() : null;
        }

        @Override
        public void close() throws Exception {
            if (employeeStream != null) {
                employeeStream.close();
            }
        }
    };
}

不过更推荐用Spring Batch自带的JpaCursorItemReader，它本身就封装了游标式流式读取的逻辑，和你Repository里的QueryHint配置逻辑完全匹配，配置更规范也更易维护：

@Bean
public JpaCursorItemReader<Employee> jpaCursorReader(EntityManagerFactory entityManagerFactory) {
    return new JpaCursorItemReaderBuilder<Employee>()
            .name("employeeCursorReader")
            .entityManagerFactory(entityManagerFactory)
            .queryString("SELECT DISTINCT e from Employee e LEFT JOIN FETCH e.salaries LEFT JOIN FETCH e.titles WHERE e.hireDate BETWEEN :from AND :to ORDER BY e.employeeId")
            .parameterValues(Map.of("from", LocalDate.of(2000, 1, 1), "to", LocalDate.of(2024, 1, 1)))
            .queryHint(HINT_FETCH_SIZE, String.valueOf(Integer.MIN_VALUE))
            .queryHint(HINT_CACHEABLE, "false")
            .queryHint(HINT_READONLY, "true")
            .queryHint(HINT_PASS_DISTINCT_THROUGH, "false")
            .build();
}

二、分页读取同样支持 #

Spring Batch专门提供了JpaPagingItemReader来处理分页读取场景，它会自动按配置的页大小循环查询数据，避免一次性加载大量数据到内存：

@Bean
public JpaPagingItemReader<Employee> jpaPagingReader(EntityManagerFactory entityManagerFactory) {
    return new JpaPagingItemReaderBuilder<Employee>()
            .name("employeePagingReader")
            .entityManagerFactory(entityManagerFactory)
            .queryString("SELECT DISTINCT e from Employee e LEFT JOIN FETCH e.salaries LEFT JOIN FETCH e.titles WHERE e.hireDate BETWEEN :from AND :to ORDER BY e.employeeId")
            .parameterValues(Map.of("from", LocalDate.of(2000, 1, 1), "to", LocalDate.of(2024, 1, 1)))
            .pageSize(1000) // 根据你的内存情况调整每页数据量
            .build();
}

要是你想直接用Repository里的findAll(Pageable)方法，也可以自己写个自定义ItemReader来循环调用分页查询：

@Bean
public ItemReader<Employee> employeePagingReader(EmployeeRepository employeeRepository) {
    return new ItemReader<>() {
        private int currentPage = 0;
        private final int pageSize = 1000;
        private List<Employee> currentPageData;
        private int dataIndex = 0;

        @Override
        public Employee read() throws Exception {
            // 当前页数据读完了，就取下一页
            if (currentPageData == null || dataIndex >= currentPageData.size()) {
                Page<Employee> pageResult = employeeRepository.findAll(
                        PageRequest.of(currentPage++, pageSize, Sort.by("employeeId"))
                );
                currentPageData = pageResult.getContent();
                dataIndex = 0;
                // 没有更多数据了，返回null结束读取
                if (currentPageData.isEmpty()) {
                    return null;
                }
            }
            return currentPageData.get(dataIndex++);
        }
    };
}

两种方式怎么选 #

• 流式读取（游标）：适合超大数据量场景，内存占用极低，但要求数据库支持游标，且查询必须有唯一排序字段（防止数据重复或遗漏）。
• 分页读取：兼容性更强，不需要数据库游标支持，但每次分页都会生成新的查询SQL，性能略低于游标方式，同样要确保排序字段唯一，避免跨页数据重复。

内容的提问来源于stack exchange，提问作者Sha