Spring Batch处理大MultipartFile：替换静态变量传参方案咨询

问题背景 #

开发Spring图片批处理系统时，当前流程通过静态变量将MultipartFile[]从Service传递给自定义ItemReader，但处理GB级大文件时静态变量会带来内存占用过高、线程安全等问题，需要找到无静态变量的传参方式。

现有代码：

Service类 #

@Service
public class ImageService {
    @Autowired
    private JobLauncher jobLauncher;
    @Autowired
    private Job job;
    public static MultipartFile[] files = null;

    public void imageFiles(MultipartFile[] files) {
        JobParameters jobParameters = new JobParametersBuilder()
                .addLong("startAt", System.currentTimeMillis()).toJobParameters();
        try {
            ImageService.files = files;
            jobLauncher.run(job, jobParameters);
        } catch (JobExecutionAlreadyRunningException | JobRestartException | JobInstanceAlreadyCompleteException |
                 JobParametersInvalidException e) {
            e.printStackTrace();
        }
    }
}

自定义Reader类 #

@Service
public class MyCustomReader implements ItemReader<MultipartFile> {
    private int index = 0;

    @Override
    public MultipartFile read() throws Exception {
        MultipartFile[] files = ImageService.files;
        if (index >= files.length)
            return null;
        return files[index++];
    }
}

配置类 #

@Configuration
@EnableBatchProcessing
@AllArgsConstructor
public class BatchConfig {
    @Autowired
    private JobBuilderFactory jobBuilderFactory;
    @Autowired
    private StepBuilderFactory stepBuilderFactory;
    @Autowired
    private MyCustomReader myCustomReader;
    @Autowired
    private MyCustomProcessor myCustomProcessor;
    @Autowired
    private MyCustomWriter myCustomWriter;

    @Bean
    public Step step() {
        return stepBuilderFactory.get("MyStep").<MultipartFile, MultipartFile>chunk(3)
                .reader(myCustomReader)
                .processor(myCustomProcessor)
                .writer(myCustomWriter)
                .build();
    }

    @Bean
    public Job runBatchJob() {
        return jobBuilderFactory.get("batch")
                .flow(step()).end().build();
    }
}

可行解决方案 #

方案一：转存临时文件，通过JobParameters传递路径（推荐用于大文件） #

将MultipartFile转存到系统临时目录，把文件路径作为Job参数传递，Reader从Job参数中读取路径并加载文件，避免内存中持有大文件对象。

1. 修改ImageService

@Service
public class ImageService {
    @Autowired
    private JobLauncher jobLauncher;
    @Autowired
    private Job job;
    // 定义临时文件存储目录
    private final Path tempDir = Paths.get(System.getProperty("java.io.tmpdir"), "batch-image-temp");

    public void imageFiles(MultipartFile[] files) throws IOException {
        // 创建临时目录
        Files.createDirectories(tempDir);
        List<String> filePaths = new ArrayList<>();

        // 将MultipartFile转存为临时文件
        for (MultipartFile file : files) {
            // 生成唯一文件名避免冲突
            String tempFileName = UUID.randomUUID() + "-" + file.getOriginalFilename();
            Path tempFile = tempDir.resolve(tempFileName);
            Files.copy(file.getInputStream(), tempFile, StandardCopyOption.REPLACE_EXISTING);
            filePaths.add(tempFile.toString());
        }

        // 将路径列表转为逗号分隔字符串，作为Job参数传递
        JobParameters jobParameters = new JobParametersBuilder()
                .addLong("startAt", System.currentTimeMillis())
                .addString("imageFilePaths", String.join(",", filePaths))
                .toJobParameters();

        try {
            jobLauncher.run(job, jobParameters);
        } catch (Exception e) {
            e.printStackTrace();
            // 异常时清理临时文件
            filePaths.forEach(path -> {
                try {
                    Files.deleteIfExists(Paths.get(path));
                } catch (IOException ex) {
                    ex.printStackTrace();
                }
            });
        }
    }
}

2. 修改自定义Reader（实现StepExecutionListener）

@Service
@StepScope // 必须添加StepScope才能获取StepExecution上下文
public class MyCustomReader implements ItemReader<File>, StepExecutionListener {
    private int currentIndex = 0;
    private List<File> imageFiles;

    @Override
    public void beforeStep(StepExecution stepExecution) {
        // 从Job参数中读取文件路径列表
        String pathStr = stepExecution.getJobParameters().getString("imageFilePaths");
        String[] paths = pathStr.split(",");
        imageFiles = Arrays.stream(paths)
                .map(File::new)
                .collect(Collectors.toList());
    }

    @Override
    public File read() throws Exception {
        if (currentIndex >= imageFiles.size()) {
            return null;
        }
        return imageFiles.get(currentIndex++);
    }

    @Override
    public ExitStatus afterStep(StepExecution stepExecution) {
        // 批处理完成后清理临时文件
        imageFiles.forEach(File::delete);
        return ExitStatus.COMPLETED;
    }
}

3. 更新配置类

需要将Reader注册为Step监听器，并调整Processor/Writer适配File类型：

@Configuration
@EnableBatchProcessing
@AllArgsConstructor
public class BatchConfig {
    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;
    private final MyCustomReader myCustomReader;
    private final MyCustomProcessor myCustomProcessor;
    private final MyCustomWriter myCustomWriter;

    @Bean
    public Step imageProcessStep() {
        return stepBuilderFactory.get("ImageProcessStep")
                .<File, File>chunk(3) // 调整泛型为File
                .reader(myCustomReader)
                .processor(myCustomProcessor) // 需要修改Processor处理File类型
                .writer(myCustomWriter) // 需要修改Writer处理File类型
                .listener(myCustomReader) // 注册Step监听器
                .build();
    }

    @Bean
    public Job imageBatchJob() {
        return jobBuilderFactory.get("ImageBatchJob")
                .flow(imageProcessStep())
                .end()
                .build();
    }
}

方案二：将Reader设为原型Bean，动态注入文件（适合小文件） #

把Reader改为原型作用域，每次批处理时获取新的Reader实例，直接设置MultipartFile[]，避免静态变量共享。

1. 修改自定义Reader

@Service
@Scope("prototype") // 原型作用域，每次获取都是新实例
public class MyCustomReader implements ItemReader<MultipartFile> {
    private int currentIndex = 0;
    private MultipartFile[] files;

    // 提供setter方法注入文件数组
    public void setFiles(MultipartFile[] files) {
        this.files = files;
        this.currentIndex = 0; // 重置索引
    }

    @Override
    public MultipartFile read() throws Exception {
        if (files == null || currentIndex >= files.length) {
            return null;
        }
        return files[currentIndex++];
    }
}

2. 修改ImageService动态创建Job/Step

@Service
public class ImageService {
    @Autowired
    private JobLauncher jobLauncher;
    @Autowired
    private JobBuilderFactory jobBuilderFactory;
    @Autowired
    private StepBuilderFactory stepBuilderFactory;
    @Autowired
    private ApplicationContext applicationContext;
    @Autowired
    private MyCustomProcessor myCustomProcessor;
    @Autowired
    private MyCustomWriter myCustomWriter;

    public void imageFiles(MultipartFile[] files) {
        // 获取原型Reader实例
        MyCustomReader reader = applicationContext.getBean(MyCustomReader.class);
        reader.setFiles(files);

        // 动态创建Step（每次用唯一名称避免重复）
        Step step = stepBuilderFactory.get("ImageProcessStep-" + System.currentTimeMillis())
                .<MultipartFile, MultipartFile>chunk(3)
                .reader(reader)
                .processor(myCustomProcessor)
                .writer(myCustomWriter)
                .build();

        // 动态创建Job
        Job job = jobBuilderFactory.get("ImageBatchJob-" + System.currentTimeMillis())
                .flow(step)
                .end()
                .build();

        JobParameters jobParameters = new JobParametersBuilder()
                .addLong("startAt", System.currentTimeMillis())
                .toJobParameters();

        try {
            jobLauncher.run(job, jobParameters);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

3. 简化配置类

不需要提前创建Job和Step Bean，只需提供工厂类：

@Configuration
@EnableBatchProcessing
public class BatchConfig {
    @Bean
    public JobBuilderFactory jobBuilderFactory(JobRepository jobRepository) {
        return new JobBuilderFactory(jobRepository);
    }

    @Bean
    public StepBuilderFactory stepBuilderFactory(JobRepository jobRepository, PlatformTransactionManager transactionManager) {
        return new StepBuilderFactory(jobRepository, transactionManager);
    }
}

方案对比 #

• 方案一：适合GB级大文件，通过临时文件减少内存占用，需处理文件清理，线程安全。
• 方案二：实现简单，适合小文件场景，直接传递MultipartFile，但大文件会导致内存溢出，需注意线程安全（原型Bean保证每次实例独立）。

内容的提问来源于stack exchange，提问作者Nirupam Sur