AWS Batch多任务运行时ECS容器OOM问题排查与求助

问题场景 #

使用AWS Batch + ECS + EC2架构，Serverless模板如下：

Description: >
Setup
  AWS Batch Compute Enviornment
  AWS Batch Job Definition
  AWS Batch Queue

Parameters: 
ComputeEnvironmentName:
Description: The batch compute enviornment name to use
Type: String
PlatformVpcStackName:
Description: VPC stack name for export subnets
Type: String
JobDefinitionName:
Description: Batch Job Definition name
Type: String
ECRRepositoryName:
Description: ECR Repository Name
Type: String
JobQueueName:
Description: Batch Job Queue Name
Type: String
EnvironmentName:
Description: Environment name e.g. dev, staging, prod
Type: String
Default: dev
ComponentName:
Description: A name where this resource is created for
Type: String
PartName:
Description: The name of the component's part
Type: String

Resources:
ASGLaunchTemplate:
Type: AWS::EC2::LaunchTemplate
Properties:
  LaunchTemplateData:
    MetadataOptions:
      HttpEndpoint: enabled
      HttpTokens: required
  TagSpecifications:
    - ResourceType: launch-template
      Tags:
      - Key: STAGE
        Value: !Ref EnvironmentName
      - Key: COMPONENT_NAME
        Value: !Ref ComponentName
      - Key: PART_NAME
        Value: !Ref PartName
      - Key: StackID
        Value: !Ref 'AWS::StackId'

tttCommonParserBatchComputeEnvironment:
Type: 'AWS::Batch::ComputeEnvironment'
Properties:
  ComputeEnvironmentName: !Ref ComputeEnvironmentName
  ComputeResources:
    MaxvCpus: 2
    MinvCpus: 0
    InstanceRole: !GetAtt tttHandlerEcsInstanceProfile.Arn
    InstanceTypes:
      - a1.medium
    DesiredvCpus: 2
    LaunchTemplate:
      LaunchTemplateId: !Ref ASGLaunchTemplate
    Tags:
      'STAGE' : !Ref EnvironmentName
      'COMPONENT_NAME' : !Ref ComponentName
      'PART_NAME' : !Ref PartName
      'StackID' : !Ref 'AWS::StackId'
  State: ENABLED
  Type: MANAGED
  Tags:
    'STAGE' : !Ref EnvironmentName
    'COMPONENT_NAME' : !Ref ComponentName
    'PART_NAME' : !Ref PartName
    'StackID' : !Ref 'AWS::StackId'

BatchJobDefinition:
Type: AWS::Batch::JobDefinition
Properties:
  Type: container
  JobDefinitionName: !Ref JobDefinitionName
  RetryStrategy: 
    Attempts: 1
  Timeout:
    AttemptDurationSeconds: 300
  ContainerProperties:
    JobRoleArn: !Ref tttHandlerJobServiceRole
    Image: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/testing
    Vcpus: 1
    Memory: 256
  Tags: {
    "STAGE" :  !Ref EnvironmentName,
    "COMPONENT_NAME" : !Ref ComponentName,
    "PART_NAME" : !Ref PartName,
    "StackID" : !Ref 'AWS::StackId'
    }

BatchJobQueue: 
Type: "AWS::Batch::JobQueue"
Properties:
  JobQueueName: !Ref JobQueueName
  Priority: 1
  State: ENABLED
  ComputeEnvironmentOrder:
    - Order: 1
      ComputeEnvironment: !Ref tttCommonParserBatchComputeEnvironment
  Tags:
    'STAGE' : !Ref EnvironmentName
    'COMPONENT_NAME' : !Ref ComponentName
    'PART_NAME' : !Ref PartName
    'StackID' : !Ref 'AWS::StackId'

tttCommonParserEcsInstanceRole:
Type: AWS::IAM::Role
Properties:
  RoleName: !Join ["-", [!Ref EnvironmentName, !Ref ComponentName, !Ref PartName, "ttt-common-parser-ecsInstanceRole"] ]
  AssumeRolePolicyDocument:
    Statement:
      - Action: ['sts:AssumeRole']
        Effect: Allow
        Principal:
          Service: [ec2.amazonaws.com]
    Version: '2012-10-17'
  Path: /        
  ManagedPolicyArns:
    - "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
  Tags:
    - Key: STAGE
      Value: !Ref EnvironmentName
    - Key: COMPONENT_NAME
      Value: !Ref ComponentName
    - Key: PART_NAME
      Value: !Ref PartName
    - Key: StackID
      Value: !Ref 'AWS::StackId'

tttHandlerEcsInstanceProfile:
  Type: "AWS::IAM::InstanceProfile"
  Properties:
    Path: "/"
    Roles:
      - !Ref tttCommonParserEcsInstanceRole

tttHandlerJobServiceRole:
Type: AWS::IAM::Role
Properties:
  RoleName: !Join ["-", [!Ref EnvironmentName, !Ref ComponentName, !Ref PartName, "ttt-common-parser-jobServiceRole"] ]
  AssumeRolePolicyDocument:
    Statement:
      - Action: ['sts:AssumeRole']
        Effect: Allow
        Principal:
          Service: [ecs-tasks.amazonaws.com]
    Version: '2012-10-17'
  Path: /        

运行单个批处理任务正常，但同时运行2-3个任务时，部分任务报错：AWS ECS: OutOFMemoryError: Container killed due to memory usage。排查后确认：

• 单个任务实际占用136MB内存
• 多任务运行时容器被复用，第二个任务启动时容器已占用136MB，总内存达到272MB，超过了256MB的容器内存限制
• EC2主机内存充足（a1.medium实例拥有2GB内存）

解决方案 #

方案1：禁用容器复用，每个任务独立实例化容器 #

AWS Batch在EC2计算环境下默认会复用同一JobDefinition的容器以提升效率，要禁用该行为，需修改BatchJobDefinition的配置，确保每个任务启动全新容器：

1. 在ContainerProperties中添加readonlyRootFilesystem: true：设置容器根文件系统为只读，阻止容器复用（复用容器需要可写层存储任务状态）
2. 确保任务执行完成后容器自动退出：检查容器镜像的启动命令，确保任务完成后执行exit操作，避免容器处于挂起状态被复用

修改后的BatchJobDefinition片段：

BatchJobDefinition:
Type: AWS::Batch::JobDefinition
Properties:
  Type: container
  JobDefinitionName: !Ref JobDefinitionName
  RetryStrategy: 
    Attempts: 1
  Timeout:
    AttemptDurationSeconds: 300
  ContainerProperties:
    JobRoleArn: !Ref tttHandlerJobServiceRole
    Image: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/testing
    Vcpus: 1
    Memory: 256
    readonlyRootFilesystem: true  # 添加该配置禁用容器复用
    Command: ["your-task-command", "&&", "exit"]  # 确保任务完成后退出
  Tags: {
    "STAGE" :  !Ref EnvironmentName,
    "COMPONENT_NAME" : !Ref ComponentName,
    "PART_NAME" : !Ref PartName,
    "StackID" : !Ref 'AWS::StackId'
    }

方案2：优化资源配置与动态扩容 #

如果需要保留容器复用的效率，可通过调整内存限制或开启EC2自动扩容来解决内存溢出问题：

2.1 调整容器内存限制

由于单个任务实际仅需136MB，两个任务叠加后需272MB，可直接将容器内存限制从256MB提升至300MB，满足多任务复用的内存需求：

BatchJobDefinition:
# ... 其他配置
  ContainerProperties:
    # ... 其他配置
    Memory: 300  # 调整内存限制

2.2 开启EC2自动扩容

通过Batch托管的计算环境自动扩容EC2实例，让多任务分散到不同实例运行，避免单实例上的容器复用内存叠加：

1. 修改tttCommonParserBatchComputeEnvironment的ComputeResources配置，提升MaxvCpus上限（当前为2，可调整为4或更高）
2. 确保DesiredvCpus设置为0，让Batch根据任务负载自动调整实例数量

修改后的计算环境片段：

tttCommonParserBatchComputeEnvironment:
Type: 'AWS::Batch::ComputeEnvironment'
Properties:
  ComputeEnvironmentName: !Ref ComputeEnvironmentName
  ComputeResources:
    MaxvCpus: 4  # 提升最大vCPU数量，允许扩容更多实例
    MinvCpus: 0
    InstanceRole: !GetAtt tttHandlerEcsInstanceProfile.Arn
    InstanceTypes:
      - a1.medium
    DesiredvCpus: 0  # 让Batch自动调整实例数量
    LaunchTemplate:
      LaunchTemplateId: !Ref ASGLaunchTemplate
    Tags:
      'STAGE' : !Ref EnvironmentName
      'COMPONENT_NAME' : !Ref ComponentName
      'PART_NAME' : !Ref PartName
      'StackID' : !Ref 'AWS::StackId'
  State: ENABLED
  Type: MANAGED
  Tags:
    'STAGE' : !Ref EnvironmentName
    'COMPONENT_NAME' : !Ref ComponentName
    'PART_NAME' : !Ref PartName
    'StackID' : !Ref 'AWS::StackId'

此外，还可以通过CloudWatch监控ECS实例的内存使用率，设置告警触发ASG扩容，进一步优化动态资源调整。

内容的提问来源于stack exchange，提问作者Shadab Faiz