如何降低Amazon S3 AWS上传ProgressListener的通知频率与冗余度？

实现原理 #

AWS SDK的ProgressListener默认会在每完成8KB缓冲区数据传输后触发progressChanged事件，无需修改SDK底层逻辑，只需在监听器内部增加累计计数和阈值判断逻辑，即可实现自定义粒度的日志打印。

具体实现代码 #

修改后的Java参考实现 #

TransferManager tm = new TransferManager(new ProfileCredentialsProvider());        

PutObjectRequest request = new PutObjectRequest(
        existingBucketName, keyName, new File(filePath));

// 自定义打印阈值，这里设置为1MB=1048576字节，可自行调整
long printThreshold = 1048576L;
AtomicLong totalTransferred = new AtomicLong(0);
AtomicLong lastPrinted = new AtomicLong(0);

request.setGeneralProgressListener(new ProgressListener() {
    @Override
    public void progressChanged(ProgressEvent progressEvent) {
        long current = totalTransferred.addAndGet(progressEvent.getBytesTransferred());
        // 累计增量达到阈值才打印
        if (current - lastPrinted.get() >= printThreshold) {
            System.out.println("Transferred bytes: " + current);
            lastPrinted.set(current);
        }
    }
});       

Upload upload = tm.upload(request);

这里用AtomicLong是为了避免多线程场景下的计数错乱，如果你能确认监听器只会单线程回调也可以用普通long变量。

Scala适配实现 #

import com.amazonaws.event.{ProgressEvent, ProgressListener}
import com.amazonaws.services.s3.model.PutObjectRequest
import com.amazonaws.services.s3.transfer.TransferManager
import com.amazonaws.auth.profile.ProfileCredentialsProvider
import java.util.concurrent.atomic.AtomicLong
import java.io.File

val tm = new TransferManager(new ProfileCredentialsProvider())
val request = new PutObjectRequest(existingBucketName, keyName, new File(filePath))

val printThreshold = 1048576L // 1MB阈值，可自行调整
val totalTransferred = new AtomicLong(0)
val lastPrinted = new AtomicLong(0)

request.setGeneralProgressListener(new ProgressListener {
  override def progressChanged(progressEvent: ProgressEvent): Unit = {
    val current = totalTransferred.addAndGet(progressEvent.getBytesTransferred)
    if (current - lastPrinted.get() >= printThreshold) {
      println(s"Transferred bytes: $current")
      lastPrinted.set(current)
    }
  }
})

val upload = tm.upload(request)

扩展优化 #

如果需要按百分比打印进度，可以提前获取待上传文件的总大小，将阈值设置为总大小的对应百分比即可，比如设置每上传5%打印一次日志。

内容的提问来源于stack exchange，提问作者Indoordinosaur