如何让多个ECS任务在同一g4dn.xlarge EC2实例上并行运行
问题详情
在配置为1块GPU、4核CPU、16GB内存的g4dn.xlarge型号EC2实例上,尝试运行多个ECS任务时遇到以下问题:
- 采用GPU共享配置实现多任务共用GPU,但启动多个任务时,第二个任务会一直处于配置中状态,直至第一个任务执行完成
- CloudWatch监控数据显示,每个任务运行期间CPU利用率均低于50%
当前使用的CDK代码
const taskDefinition = new TaskDefinition(this, 'TaskDefinition', { compatibility: Compatibility.EC2 }) const container = taskDefinition.addContainer('Container', { image: ContainerImage.fromEcrRepository(<image>), entryPoint: ["python", "src/script.py"], workingDirectory: "/root/repo", startTimeout: Duration.minutes(5), stopTimeout: Duration.minutes(60), memoryReservationMiB: 8192, logging: LogDriver.awsLogs({ logGroup: logGroup, streamPrefix: 'prefix', }), }) const startUpScript = UserData.forLinux() // 用于实现任务共享GPU的配置脚本 startUpScript.addCommands( `(grep -q ^OPTIONS=\\"--default-runtime /etc/sysconfig/docker && echo '/etc/sysconfig/docker needs no changes') || (sed -i 's/^OPTIONS="/OPTIONS="--default-runtime nvidia /' /etc/sysconfig/docker && echo '/etc/sysconfig/docker updated to have nvidia runtime as default' && systemctl restart docker && echo 'Restarted docker')` ) const launchTemplate = new LaunchTemplate(this, 'LaunchTemplate', { machineImage: EcsOptimizedImage.amazonLinux2( AmiHardwareType.GPU ), detailedMonitoring: false, instanceType: InstanceType.of(InstanceClass.G4DN, InstanceSize.XLARGE), userData: startUpScript, role: <launchTemplateRole>, }) const autoScalingGroup = new AutoScalingGroup(this, 'AutoScalingGroup', { vpc: vpc, minCapacity: 0, maxCapacity: 1, desiredCapacity: 0, launchTemplate: launchTemplate, }) const capacityProvider = new AsgCapacityProvider(this, 'AsgCapacityProvider', { autoScalingGroup: autoScalingGroup, }) cluster.addAsgCapacityProvider(capacityProvider)
补充说明
已为任务定义分配CPU和内存资源,但问题仍然存在。

内容的提问来源于stack exchange,提问作者Austin Ulfers
相关产品推荐
相关产品推荐

