通过Flux<DataBuffer>向Azure Blob存储写入大文件时挂起问题求助
问题:REST下载大文件写入Azure Blob存储时出现挂起异常
我需要通过REST接口下载大文件并存储到Azure Blob存储,目前遇到了异常问题:
数据获取逻辑
我用以下代码从REST接口获取数据流:
var flux = this.webClient .get() .uri(urlToAssert) .accept(MediaType.APPLICATION_OCTET_STREAM) .exchangeToFlux(clientResponse -> clientResponse.body(BodyExtractors.toDataBuffers()));
本地文件写入验证
将上述数据流写入本地文件系统时,50MB、1GB、5GB的文件都能正常完成:
try (OutputStream outputStream = new FileOutputStream(targetFile.resolve("testData.zip").toAbsolutePath().toString(), false)) { DataBufferUtils.write(flux, outputStream).map(DataBufferUtils::release).blockLast(Duration.ofHours(22)); } catch (Exception e) { throw new IllegalStateException(e); }
由此可确认流处理逻辑本身是正常的。
Azure Blob写入异常
但使用Azure Blob存储的OutputStream写入时,50MB文件可正常完成,1GB文件虽然最终会出现在Blob存储中,但代码会在try块内挂起:
try (OutputStream outputStream = this.blob.getOutputStreamParallel(destination, this.blob.getParallelTransferOptions())) { DataBufferUtils.write(flux, outputStream).map(DataBufferUtils::release).blockLast(Duration.ofHours(22)); outputStream.flush(); } catch (IOException e) { throw new IllegalStateException(e); }
相关方法与配置
获取Blob输出流的方法定义:
public OutputStream getOutputStreamParallel(Path path, ParallelTransferOptions parallelTransferOptions) { var blobClientTarget = this.containerClient.getBlobClient(relativePathUnix(path)); return blobClientTarget.getBlockBlobClient().getBlobOutputStream(parallelTransferOptions, null, null, null, null); }
并行传输配置:
this.parallelTransferOptions = new ParallelTransferOptions() .setBlockSizeLong(40 * Constants.MB) .setMaxConcurrency(5) .setProgressListener(bytesTransferred -> { log.info("write bytes, bytes transferred '{}'", bytesTransferred); });
日志中显示存在异常情况(相关异常信息见截图),想请教这是我的代码问题,还是Azure Blob存储的故障?
内容的提问来源于stack exchange,提问作者user286974
相关产品推荐
相关产品推荐

