如何在aws emr create-cluster命令中指定Spot实例类型？

要在aws emr create-cluster命令中指定Task节点使用Spot实例，需要在对应的Instance Group配置中添加"Market": "SPOT"参数，还可按需添加"SpotPrice"设置最高出价（不指定则默认按当前Spot价格计费，最高不超过按需实例价格）。

修改后的完整CLI命令 #

aws emr create-cluster \
    --name hm-amazon-emr-cluster \
    --release-label emr-6.12.0 \
    --applications=Name=Trino \
    --instance-groups '[{"InstanceCount":1,"InstanceGroupType":"MASTER","Name":"Primary","InstanceType":"r5.xlarge","EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"VolumeType":"gp2","SizeInGB":32},"VolumesPerInstance":2}]}},{"InstanceCount":1,"InstanceGroupType":"CORE","Name":"Core","InstanceType":"r5.xlarge","EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"VolumeType":"gp2","SizeInGB":32},"VolumesPerInstance":2}],"EbsOptimized":true}},{"InstanceCount":15,"InstanceGroupType":"TASK","Name":"Task","InstanceType":"r5.xlarge","Market": "SPOT", "SpotPrice": "0.25", "EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"VolumeType":"gp2","SizeInGB":32},"VolumesPerInstance":2}]}}]' \
    --log-uri s3n://hongbomiao-bucket/amazon-emr/logs/ \
    --service-role arn:aws:iam::8888888888888:role/service-role/AmazonEMR-ServiceRole-xxxxxxxxxxxxxxx \
    --scale-down-behavior TERMINATE_AT_TASK_COMPLETION \
    --auto-termination-policy '{"IdleTimeout":7200}' \
    --step-concurrency-level 10 \
    --region us-west-2

格式化后的Instance Groups配置（突出Spot相关修改） #

[
  {
    "InstanceCount": 1,
    "InstanceGroupType": "MASTER",
    "Name": "Primary",
    "InstanceType": "r5.xlarge",
    "EbsConfiguration": {
      "EbsBlockDeviceConfigs": [
        {
          "VolumeSpecification": {
            "VolumeType": "gp2",
            "SizeInGB": 32
          },
          "VolumesPerInstance": 2
        }
      ]
    }
  },
  {
    "InstanceCount": 1,
    "InstanceGroupType": "CORE",
    "Name": "Core",
    "InstanceType": "r5.xlarge",
    "EbsConfiguration": {
      "EbsBlockDeviceConfigs": [
        {
          "VolumeSpecification": {
            "VolumeType": "gp2",
            "SizeInGB": 32
          },
          "VolumesPerInstance": 2
        }
      ],
      "EbsOptimized": true
    }
  },
  {
    "InstanceCount": 15,
    "InstanceGroupType": "TASK",
    "Name": "Task",
    "InstanceType": "r5.xlarge",
    "Market": "SPOT",
    "SpotPrice": "0.25",
    "EbsConfiguration": {
      "EbsBlockDeviceConfigs": [
        {
          "VolumeSpecification": {
            "VolumeType": "gp2",
            "SizeInGB": 32
          },
          "VolumesPerInstance": 2
        }
      ]
    }
  }
]

参数说明

• "Market": "SPOT"：必填参数，明确指定实例购买类型为Spot
• "SpotPrice"：可选参数，设置你愿意支付的最高小时价格；若不填写，AWS会按当前Spot市场价格收费，且不会超过对应按需实例的价格
• 控制台导出的克隆命令未包含Spot配置属于控制台的行为限制，需手动在CLI命令中补充该参数

内容的提问来源于stack exchange，提问作者Hongbo Miao