多容器ECS任务挂载EBS卷失败问题求助

目标 #

在包含多个容器的ECS任务中使用单个EBS卷实现集中存储

已尝试操作 #

• 创建包含两个容器的任务定义，运行时总有一个容器启动失败，container_1报错：

CannotStartContainerError: Error response from daemon: error while mounting volume '': VolumeDriver.Mount: docker-legacy: Mount: ebs_test: failed: no device name returned

• 第二个容器可成功启动
• 移除container_2后，仅含container_1的任务能正常启动，EBS卷正常出现在/mnt目录：

root@d2dc53cd2a2b:/# ls /mnt/
ebs_test

• 移除container_1后，container_2也能正常在/mnt目录获取EBS卷：

bash-4.2# ls /mnt/
ebs_test

相关代码 #

locals {
  volume_name = "ebs_test"
}

resource "aws_ebs_volume" "ebs_test" {
  availability_zone = "us-west-2a"
  size = 51

  tags = {
    Name = local.volume_name
  }
}

resource "aws_ecs_task_definition" "ebs_test_task" {
  family = "ebs_test_task"
  requires_compatibilities = ["EC2"]
  execution_role_arn = aws_iam_role.ecs_task_execution_role.arn
  task_role_arn = aws_iam_role.execution_role.arn
  placement_constraints {
    type = "memberOf"
    expression = "ec2InstanceId == specific_id"
  }

  volume {
    name = local.volume_name
    docker_volume_configuration {
      scope = "shared"
      autoprovision = true
      driver = "rexray/ebs"
    }
  }

  volume {
    name = "efs_test"
    efs_volume_configuration {
      file_system_id = module.efs_mount.file_system_id
      transit_encryption = "ENABLED"
      authorization_config {
        access_point_id = aws_efs_access_point.efs_access_point.id
      }
    }
  }

  container_definitions = jsonencode([
    {
      name = "Container_1"
      memoryReservation = 512
      essential = true
      image = "<ACCOUNT_ID>.dkr.ecr.us-west-2.amazonaws.com/container_1:latest"
      portMappings = [
        {
          hostPort = 80,
          containerPort = 80
        }
      ],
      mountPoints = [
        {
          containerPath = "/efs_test"
          sourceVolume = "efs_test"
        },
        {
          containerPath = "/mnt/${local.volume_name}"
          sourceVolume = local.volume_name
        }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          awslogs-group = "task_name"
          awslogs-region = data.aws_region.current.name
          awslogs-stream-prefix = "container_1"
        }
      }
    },
    {
      name = "Container_2"
      memoryReservation = 512
      image = "<ACCOUNT_ID>.dkr.ecr.us-west-2.amazonaws.com/container_2:latest"
      portMappings = [
        {
          hostPort = 81,
          containerPort = 80
        }
      ],
      mountPoints = [
        {
          containerPath = "/efs_test"
          sourceVolume = "efs_test"
        },
        {
          containerPath = "/mnt/${local.volume_name}"
          sourceVolume = local.volume_name
        }
      ]
      essential = true
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          awslogs-group = "task_name"
          awslogs-region = data.aws_region.current.name
          awslogs-stream-prefix = "container_2"
        }
      }
    },
  ])
}

更新发现 #

将每个容器放在单独任务中顺序启动可规避问题，但用线程快速调用run_task时，仍有一个容器出现相同错误，推测可能是EBS卷挂载超时或被占用导致？

解决方案 #

1. 调整RexRay挂载超时配置 #

RexRay默认挂载超时可能不足以应对EBS卷的挂载延迟，在ECS实例的RexRay配置文件（通常为/etc/rexray/config.yml）中添加以下配置：

ebs:
  mountTimeout: 120
  unmountTimeout: 60

修改后重启RexRay服务：

sudo systemctl restart rexray

2. 配置容器启动依赖 #

ECS任务启动时会并行尝试为多个容器挂载卷，而EBS挂载是独占操作。可以给容器添加启动依赖，让第二个容器在第一个容器完成挂载后再启动：
修改任务定义的容器部分，给Container_2添加dependsOn：

{
  "name": "Container_2",
  // 其他原有配置
  "essential": true,
  "dependsOn": [
    {
      "containerName": "Container_1",
      "condition": "STARTED"
    }
  ]
}

3. 使用ECS原生EBS卷挂载（替代RexRay） #

AWS ECS支持原生EBS卷挂载（适用于EC2启动类型），稳定性更高。修改任务定义中的卷配置：

volume {
  name = local.volume_name
  ebs_volume_configuration {
    volume_id = aws_ebs_volume.ebs_test.id
    delete_on_termination = false
  }
}

4. 检查ECS实例权限 #

确保ECS实例的IAM角色拥有ec2:AttachVolume、ec2:DetachVolume、ec2:DescribeVolumes权限，避免因权限不足导致挂载失败。

内容的提问来源于stack exchange，提问作者Cogito Ergo Sum