如何在不同Spring Boot应用间传递10-100MB的字节数组

Hey there! Let's tackle this large data transfer problem between your Spring Boot apps. 10-100MB is definitely too big for naive approaches like Base64 encoding or stuffing the whole byte array into a JSON payload—here are some solid, battle-tested solutions I've used in similar scenarios:

方案1：流式传输二进制数据（避免全量内存加载） #

Your original code loads the entire file into a byte[], which is risky for large files (OOM issues) and might be causing the data corruption you're seeing. Instead, stream the data directly from GCS to the target app without loading everything into memory.

修改GCS下载代码（返回InputStream而非byte[]） #

public InputStream downloadFileStreamFromGCS(String objectName, String bucketName) throws IOException { 
    val storage = StorageOptions.getDefaultInstance().getService(); 
    val blob = storage.get(BlobId.of(bucketName, objectName)); 
    return blob.reader();
}

发送方（Spring Boot）用WebClient流式发送 #

WebClient is ideal for non-blocking streaming transfers:

WebClient webClient = WebClient.create("http://your-target-app-base-url");

// Stream the GCS input stream directly to the target app
Mono<Void> transferResult = webClient.post()
    .uri("/api/receive-binary-data")
    .contentType(MediaType.APPLICATION_OCTET_STREAM)
    .body(BodyInserters.fromResource(new InputStreamResource(downloadFileStreamFromGCS(objectName, bucketName))))
    .retrieve()
    .bodyToMono(Void.class);

transferResult.block(); // Use non-blocking .subscribe() if you prefer reactive patterns

接收方（Spring Boot）处理流式输入 #

@PostMapping("/api/receive-binary-data")
public ResponseEntity<Void> receiveBinaryData(@RequestBody InputStream inputStream) {
    // Process the stream directly (no need to load all bytes into memory)
    try (inputStream) {
        // Example: Save to local disk or process incrementally
        Files.copy(inputStream, Paths.get("/path/to/your/target/file"), StandardCopyOption.REPLACE_EXISTING);
    } catch (IOException e) {
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).build();
    }
    return ResponseEntity.ok().build();
}

方案2：通过GCS签名URL共享（无需直接传递数据） #

This is my favorite approach for large data—instead of sending the file between apps, let the receiver download it directly from GCS using a temporary signed URL. It's efficient, secure, and avoids bandwidth overhead between your services.

发送方生成GCS临时签名URL #

public String generateGcsSignedUrl(String objectName, String bucketName, int expirationMinutes) throws IOException {
    val storage = StorageOptions.getDefaultInstance().getService();
    BlobId blobId = BlobId.of(bucketName, objectName);
    BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();

    // Generate a URL valid for the specified number of minutes
    return storage.signUrl(
        blobInfo,
        Duration.ofMinutes(expirationMinutes),
        Storage.SignUrlOption.withV4Signature()
    ).toString();
}

发送方通知接收方（传递短URL而非大文件） #

// Send the signed URL via a simple JSON request
RestTemplate restTemplate = new RestTemplate();
Map<String, String> payload = Map.of("signedGcsUrl", generateGcsSignedUrl(objectName, bucketName, 60));
restTemplate.postForObject("http://your-target-app-base-url/api/notify-data-ready", payload, Void.class);

接收方下载并处理数据 #

@PostMapping("/api/notify-data-ready")
public ResponseEntity<Void> handleDataNotification(@RequestBody Map<String, String> payload) {
    String signedUrl = payload.get("signedGcsUrl");
    if (signedUrl == null) {
        return ResponseEntity.badRequest().build();
    }

    try (InputStream inputStream = new URL(signedUrl).openStream()) {
        // Process the file stream as needed
        Files.copy(inputStream, Paths.get("/path/to/processed/file"), StandardCopyOption.REPLACE_EXISTING);
    } catch (IOException e) {
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).build();
    }
    return ResponseEntity.ok().build();
}

方案3：使用Multipart/form-data传输 #

If you must transfer the file directly between apps, using multipart/form-data is a reliable alternative that avoids binary data corruption issues.

发送方发送Multipart请求 #

RestTemplate restTemplate = new RestTemplate();
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.MULTIPART_FORM_DATA);

MultiValueMap<String, Object> multipartBody = new LinkedMultiValueMap<>();
// Attach the GCS stream as a multipart file
multipartBody.add("dataFile", new InputStreamResource(downloadFileStreamFromGCS(objectName, bucketName)) {{
    setFilename("transfer-data.bin");
}});

HttpEntity<MultiValueMap<String, Object>> request = new HttpEntity<>(multipartBody, headers);
restTemplate.postForObject("http://your-target-app-base-url/api/receive-multipart", request, Void.class);

接收方处理Multipart文件 #

@PostMapping("/api/receive-multipart")
public ResponseEntity<Void> receiveMultipartFile(@RequestParam("dataFile") MultipartFile file) {
    try (InputStream inputStream = file.getInputStream()) {
        Files.copy(inputStream, Paths.get("/path/to/save/file"), StandardCopyOption.REPLACE_EXISTING);
    } catch (IOException e) {
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).build();
    }
    return ResponseEntity.ok().build();
}

总结 #

• 优先选择方案2（GCS签名URL）：It's the most efficient, secure, and reduces load on both your apps and network.
• If direct transfer is required: Use方案1（streaming binary）or方案3（multipart）—both avoid loading the entire file into memory and prevent data corruption issues.

内容的提问来源于stack exchange，提问作者Swarnim Pratap Singh