使用Flintrock在AWS启动Spark集群时的错误解决咨询

我按照教程尝试用Flintrock在AWS EC2实例上创建Spark集群，最终目标是在4个EC2实例（1主3从）上并行执行Spark操作，并在主节点汇总结果。

运行命令 flintrock launch cluster_name 时触发以下错误：

An error occurred (InvalidParameterCombination) when calling the RunInstances operation: The parameter iops is not supported for gp2 volumes. Operation aborted.

我的config.yaml配置内容如下：

services:
  spark:
    version: 3.1.2
    # git-commit: latest  # if not 'latest', provide a full commit SHA; e.g. d6dc12ef0146ae409834c78737c116050961f350
    # git-repository:  # optional; defaults to https://github.com/apache/spark
    # optional; defaults to download from a dynamically selected Apache mirror
    #   - can be http, https, or s3 URL
    #   - must contain a {v} template corresponding to the version
    #   - Spark must be pre-built
    #   - files must be named according to the release pattern shown here: https://dist.apache.org/repos/dist/release/spark/
    # download-source: "https://www.example.com/files/spark/{v}/"
    # download-source: "s3://some-bucket/spark/{v}/"
    # executor-instances: 1
  hdfs:
    version: 3.3.0
    # optional; defaults to download from a dynamically selected Apache mirror
    #   - can be http, https, or s3 URL
    #   - must contain a {v} template corresponding to the version
    #   - files must be named according to the release pattern shown here: https://dist.apache.org/repos/dist/release/hadoop/common/
    # download-source: "https://www.example.com/files/hadoop/{v}/"
    # download-source: "http://www-us.apache.org/dist/hadoop/common/hadoop-{v}/"
    # download-source: "s3://some-bucket/hadoop/{v}/"

provider: ec2

providers:
  ec2:
    key-name: spark_cluster
    identity-file: /media/sf_linuxvm/spark_cluster.pem
    instance-type: t2.micro
    region: us-east-1
    # availability-zone: <name>
    ami: ami-0230bd60aa48260c6
    user: ec2-user
    # ami: ami-61bbf104  # CentOS 7, us-east-1
    # user: centos
    # spot-price: <price>
    # spot-request-duration: 7d  # duration a spot request is valid, supports d/h/m/s (e.g. 4d 3h 2m 1s)
    # vpc-id: <id>
    # subnet-id: <id>
    # placement-group: <name>
    # security-groups:
    #   - group-name1
    #   - group-name2
    # instance-profile-name:
    # tags:
    #   - key1,value1
    #   - key2, value2  # leading/trailing spaces are trimmed
    #   - key3,  # value will be empty
    # min-root-ebs-size-gb: <size-gb>
    tenancy: default  # default | dedicated
    ebs-optimized: no  # yes | no
    instance-initiated-shutdown-behavior: terminate  # terminate | stop
    # user-data: /path/to/userdata/script
    # authorize-access-from:
    #   - 10.0.0.42/32
    #   - sg-xyz4654564xyz

launch:
  num-slaves: 3
  # install-hdfs: True
  install-spark: True
  java-version: 8

debug: false

我已经在网上和Stack Overflow搜索过该错误的解决方案，但不确定如何应用到我的具体场景中。

这个错误的核心原因是：Flintrock默认尝试给gp2类型的EBS卷配置IOPS参数，但gp2卷不支持手动设置IOPS（只有gp3卷支持该操作）。以下是两种可行的解决方法：

方法1：明确指定使用gp3卷 #

在providers.ec2配置块中添加root-ebs-volume-type参数，将根卷类型设置为gp3：

providers:
  ec2:
    # 保留原有的其他配置项
    root-ebs-volume-type: gp3

修改后重新执行flintrock launch cluster_name命令即可。

方法2：升级Flintrock到最新版本 #

部分旧版本的Flintrock存在默认给gp2卷配置IOPS的bug，如果你不想切换卷类型，可以尝试升级Flintrock到最新稳定版，新版本已经修复了这个问题：

pip install --upgrade flintrock

额外提示 #

t2.micro实例属于AWS免费套餐资源，但单实例资源有限（1核1G内存），运行Spark集群可能会出现性能瓶颈或资源不足的情况。如果后续执行Spark任务时遇到卡顿或内存溢出问题，建议将实例类型升级为t2.small或更高配置的机型。

内容的提问来源于stack exchange，提问作者Eric Mariasis