如何在AWS Batch(Fargate)中配置EFS挂载到Docker容器卷？

问题背景 #

我需要用AWS Batch（Fargate）运行容器化的VEBA预处理工作流：容器已推送到DockerHub，EFS上同步了S3的文件，但作业执行时触发EFS挂载失败错误：

ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: b'mount.nfs4: mounting 127.0.0.1:/mnt/efs/Fastq/ failed, reason given by server: No such file or directory' : unsuccessful EFS utils command execution; code: 32

本地Docker运行命令正常，现在需要修正作业定义，满足三个核心要求：从DockerHub拉取镜像、挂载EFS到容器、映射EFS路径到容器的三个卷目录（/volumes/input/、/volumes/output/、/volumes/database/）。

错误原因 #

1. EFS目标目录不存在：EFS不会自动创建目录，作业定义中指定的rootDirectory路径必须提前在EFS上创建
2. 缺失数据库卷配置：本地Docker挂载了数据库目录，但当前作业定义未配置该卷
3. 命令参数不全：本地命令包含数据库路径参数-x，当前作业定义的command中漏掉了该参数

修正后的作业定义 #

{
  "jobDefinitionName": "preprocess__35",
  "type": "container",
  "containerProperties": {
    "image": "docker.io/jolespin/veba_preprocess:1.2.0",
    "command": [
      "preprocess.py",
      "-1",
      "/volumes/input/35_R1.fq.gz",
      "-2",
      "/volumes/input/35_R2.fq.gz",
      "-n",
      "35",
      "-p",
      "16",
      "-o",
      "/volumes/output/preprocess",
      "-x",
      "/volumes/database/Contamination/chm13v2.0/chm13v2.0"
    ],
    "jobRoleArn": "arn:aws:iam::xxx:role/ecsTaskExecutionRole",
    "executionRoleArn": "arn:aws:iam::xxx:role/ecsTaskExecutionRole",
    "volumes": [
      {
        "name": "efs-input",
        "efsVolumeConfiguration": {
          "fileSystemId": "fs-xxx",
          "transitEncryption": "ENABLED",
          "rootDirectory": "/mnt/efs/Fastq"
        }
      },
      {
        "name": "efs-output",
        "efsVolumeConfiguration": {
          "fileSystemId": "fs-xxx",
          "transitEncryption": "ENABLED",
          "rootDirectory": "/mnt/efs/veba_output"
        }
      },
      {
        "name": "efs-database",
        "efsVolumeConfiguration": {
          "fileSystemId": "fs-xxx",
          "transitEncryption": "ENABLED",
          "rootDirectory": "/mnt/efs/database"
        }
      }
    ],
    "mountPoints": [
      {
        "sourceVolume": "efs-input",
        "containerPath": "/volumes/input",
        "readOnly": true
      },
      {
        "sourceVolume": "efs-output",
        "containerPath": "/volumes/output",
        "readOnly": false
      },
      {
        "sourceVolume": "efs-database",
        "containerPath": "/volumes/database",
        "readOnly": true
      }
    ],
    "environment": [],
    "ulimits": [],
    "resourceRequirements": [
      {
        "value": "16.0",
        "type": "VCPU"
      },
      {
        "value": "122880",
        "type": "MEMORY"
      }
    ],
    "networkConfiguration": {
      "assignPublicIp": "ENABLED"
    },
    "fargatePlatformConfiguration": {
      "platformVersion": "LATEST"
    },
    "ephemeralStorage": {
      "sizeInGiB": 40
    }
  },
  "tags": {
    "Name": "preprocess__35"
  },
  "platformCapabilities": [
    "FARGATE"
  ]
}

关键操作步骤 #

1. 提前创建EFS目录
   先将EFS挂载到任意EC2实例，创建所需目录并同步数据：

   # 挂载EFS到EC2实例（替换为你的EFS端点和挂载路径）
   sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-xxx.efs.us-east-1.amazonaws.com:/ /mnt/efs-temp
   # 创建目标目录
   sudo mkdir -p /mnt/efs-temp/Fastq /mnt/efs-temp/veba_output /mnt/efs-temp/database
   # 同步数据到对应目录
   sudo cp /local/path/35_R*.fq.gz /mnt/efs-temp/Fastq/
   sudo cp -r /local/path/Contamination /mnt/efs-temp/database/
   

2. 权限与网络检查

   • 确保ECS任务执行角色（ecsTaskExecutionRole）拥有EFS的访问权限（可通过IAM策略添加elasticfilesystem:ClientMount、elasticfilesystem:ClientWrite权限）
   • 确保EFS安全组允许Fargate任务所在安全组访问NFS端口（2049）

内容的提问来源于stack exchange，提问作者O.rka