AWS Java SDK触发BucketNotFound异常时S3源文件夹被删除问题

• Spring Boot应用集成AWS Java SDK实现S3跨桶文件拷贝，源文件夹、目标存储桶均存在时功能运行正常
• 边界测试时出现异常：将目标地址设置为不存在的存储桶，程序会按预期抛出NoSuchBucket异常，但异常抛出后观察到S3源文件夹“被删除”
• 已确认代码中无主动删除源文件夹的逻辑，日志无删除操作输出：未触发异常时源文件可完整保留，触发异常后1-2分钟内源文件夹“消失”；即使仅执行list类只读操作触发该异常，也会出现相同现象

相关代码与配置 #

Maven依赖 #

<dependency>
     <groupId>com.amazonaws</groupId>
     <artifactId>aws-java-sdk-s3</artifactId>
     <version>1.11.1034</version>
</dependency>

S3客户端配置 #

import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class AWSConfig {

    public static final Regions REGION = Regions.US_EAST_1;

    @Bean
    public AmazonS3 s3Client() {
        return AmazonS3ClientBuilder.standard()
            .withRegion(REGION)
            .build();
    }
}

文件拷贝逻辑 #

public String copyFile(S3ObjectSummary file, AmazonS3URI originUri, 
                        AmazonS3URI destinationUri, boolean deleteOrig) throws Exception {
        String originFile = "s3://"+file.getBucketName()+"/"+file.getKey();
        AmazonS3URI origUri = new AmazonS3URI(originFile);
        logger.info("Origin file="+originFile);
        String suffix = originFile.substring(originUri.toString().length());
        if (suffix.startsWith("/")) suffix = suffix.substring(1);
        String destPath = destinationUri.getKey().endsWith("/")?
                               destinationUri.getKey():destinationUri.getKey()+"/";
        String destFile = "s3://"+destinationUri.getBucket()+"/"+destPath+suffix;
        logger.info("Destination file="+destFile);
        
        try {
            // 校验源文件存在
            listFiles(originFile);
            // 校验目标桶存在，异常在此处抛出
            listFiles("s3://"+destinationUri.getBucket());
        } catch (Exception e) {
            logger.error("could not copy from origin "+originFile
                           +" to destination bucket s3://"+destinationUri.getBucket()
                           + " due to exception ",e);
            throw e;
        }
        
        AmazonS3URI destUri = new AmazonS3URI(destFile);
        s3Client.copyObject(origUri.getBucket(), origUri.getKey(), destUri.getBucket(), destUri.getKey());
        if (deleteOrig) { // 该参数固定为false
            deleteFile(originFile, file);
        }
        return destFile;
    }

列表查询方法 #

public List<String> listFiles (String file) throws IOException {
    List<String> fileList = new ArrayList<>();
    if (file.startsWith("s3")) {
        AmazonS3URI fileURI = new AmazonS3URI(file);
        fileList.addAll(s3Client.listObjects(fileURI.getBucket(),fileURI.getKey())
                .getObjectSummaries()
                .stream()
                .map(x->"s3://"+x.getBucketName()+"/"+x.getKey())
                .collect(Collectors.toList()));
    } else {
        fileList.addAll(Files.list(Paths.get(file))
                .map(x->x.toAbsolutePath().toString())
                .collect(Collectors.toList()));
    }
    return fileList;
}

异常日志片段 #

2022-06-28 17:20:00 [scheduling-1] INFO  c.c.s.c.m.e.s.ETLManagerScheduler - Found 164 output files in smc-results/results/deid, belonging to 1 flows.
2022-06-28 17:20:00 [scheduling-1] INFO  c.c.s.c.m.e.s.ETLManagerScheduler - copying over files from smc-results/results/deid to smc-resuts/results/...
2022-06-28 17:20:00 [scheduling-1] INFO  c.c.s.c.m.e.flow.ETLFlowManager - Origin file=s3://smc-results/results/deid...
2022-06-28 17:20:00 [scheduling-1] INFO  c.c.s.c.m.e.flow.ETLFlowManager - Destination file=s3://smc-resuts/results/...
2022-06-28 17:20:00 [scheduling-1] ERROR c.c.s.c.m.e.flow.ETLFlowManager - could not copy from origin s3://smc-results/results/deid/... to destination bucket s3://smc-resuts due to exception com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: NK8A8VRPVW5T0MWJ; S3 Extended Request ID: yWosaeT6c3z3h/JtI/2CCQUAm9mmMM69NW7vvgLAipKqAWqWvHtDmecoV7qc68fvS3HoWU2l2Bw=; Proxy: null)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5378)
    at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:927)
    at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:901)
    at ....listFiles(....java:259)
    at ...copyFile(....java:736)
    at ....copyFiles(....java:197)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
2022-06-28 17:20:00 [scheduling-1] WARN  c.c.s.c.m.e.s.ETLManagerScheduler - Could not list or move files from smc-results/results/deid to smc-resuts/results/... due to exception: com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: NK8A8VRPVW5T0MWJ; S3 Extended Request ID: yWosaeT6c3z3h/JtI/2CCQUAm9mmMM69NW7vvgLAipKqAWqWvHtDmecoV7qc68fvS3HoWU2l2Bw=; Proxy: null), S3 Extended Request ID: yWosaeT6c3z3h/JtI/2CCQUAm9mmMM69NW7vvgLAipKqAWqWvHtDmecoV7qc68fvS3HoWU2l2Bw=Wa

问题与S3 API、SDK的拷贝/列表逻辑无关，是三个因素叠加导致的误判：

• 目标桶名存在拼写错误：日志中目标桶为smc-resuts，比正确的源桶名smc-results少了一个字母l
• 权限配置过宽：应用使用的AK/SK持有CreateBucket权限，旧版AWS Java SDK V1的容错逻辑在访问不存在、且全局未被占用的桶名时，会隐式触发桶创建操作，不需要显式调用创建接口
• S3控制台的跳转与缓存延迟：异常触发后拼写错误的空桶smc-resuts被自动创建，控制台自动切到该空桶的对应路径，你看到的空目录根本不是原来的源路径smc-results/results/deid；1-2分钟的延迟是控制台缓存刷新时间，切回正确拼写的smc-results桶即可看到所有源文件完整存在，从未被删除。

• 第一时间登录S3控制台，手动选择正确拼写的smc-results存储桶，确认源文件完整性，删除误创建的空桶smc-resuts
• 替换存储桶存在性校验逻辑，不要用listObjects判断桶是否存在，改用专门的doesBucketExistV2方法，该方法不会触发隐式桶创建：

  if (!s3Client.doesBucketExistV2(destinationUri.getBucket())) {
      throw new IllegalArgumentException("目标存储桶不存在：" + destinationUri.getBucket());
  }
  

• 收窄应用AK的权限，移除s3:CreateBucket权限，从权限层面杜绝意外创建桶的可能
• 升级AWS Java SDK到V2版本，V2版本默认关闭所有隐式资源创建的容错逻辑，所有资源创建操作必须显式调用对应接口才会执行
• 不要依赖控制台路径导航判断文件是否存在，排查问题时直接精确指定桶名+对象前缀查询，避免控制台自动跳转导致的误判

内容的提问来源于stack exchange，提问作者Assafs