使用boto3创建EMR集群遇服务角色EC2权限不足问题咨询

现象 #

• 通过AWS Web控制台和以下CLI命令可正常创建EMR集群：

aws emr create-cluster \
 --name "emr-cluster" \
 --log-uri "s3n://aws-logs-433212232334-us-east-1/elasticmapreduce/" \
 --release-label "emr-7.1.0" \
 --service-role "arn:aws:iam::433212232334:role/service-role/AmazonEMR-ServiceRole-20240418T133040" \
 --ec2-attributes '{"InstanceProfile":"AmazonEMR-InstanceProfile-20240418T133024","EmrManagedMasterSecurityGroup":"sg-07ced3d3223bet564","EmrManagedSlaveSecurityGroup":"sg-04b7742eed31ec137","KeyName":"emr-key-pair","AdditionalMasterSecurityGroups":[],"AdditionalSlaveSecurityGroups":[],"SubnetId":"subnet-0cf42a674f981a42a"}' \
 --tags 'for-use-with-amazon-emr-managed-policies=true' \
 --applications Name=Hadoop Name=Spark \
 --configurations '[{"Classification":"spark-hive-site","Properties":{"hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"}}]' \
 --instance-groups '[{"InstanceCount":1,"InstanceGroupType":"CORE","Name":"Core","InstanceType":"m5.xlarge","EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"VolumeType":"gp2","SizeInGB":32},"VolumesPerInstance":2}]}},{"InstanceCount":1,"InstanceGroupType":"MASTER","Name":"Primary","InstanceType":"m5.xlarge","EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"VolumeType":"gp2","SizeInGB":32},"VolumesPerInstance":2}]}}]' \
 --scale-down-behavior "TERMINATE_AT_TASK_COMPLETION" \
 --auto-termination-policy '{"IdleTimeout":3600}' \
 --region "us-east-1"

• 使用以下boto3代码创建集群时，集群因报错终止：

Service role arn:aws:iam::433212232334:role/service-role/AmazonEMR-ServiceRole-20240418T133040 has insufficient EC2 permissions

boto3代码：

import boto3

emr_client = boto3.client('emr')
response = emr_client.run_job_flow(
    Name='emr-cluster',
    LogUri='s3n://aws-logs-433212232334-us-east-1/elasticmapreduce/',
    ReleaseLabel='emr-7.1.0',
    ServiceRole="arn:aws:iam::433212232334:role/service-role/AmazonEMR-ServiceRole-20240418T133040",
    JobFlowRole="AmazonEMR-InstanceProfile-20240418T133024",
    ScaleDownBehavior='TERMINATE_AT_TASK_COMPLETION',
    EbsRootVolumeSize=15,
    AutoTerminationPolicy={
        'IdleTimeout': 3600
    },
    Applications=[
        {
            'Name': 'Spark'
        },
        {
            'Name': 'Hadoop'
        }
    ],
    Configurations=[
        {
            "Classification": "spark-hive-site",
            "Properties": {
                "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
            }
        }
    ],
    Instances={
        'KeepJobFlowAliveWhenNoSteps': True,
        'Ec2KeyName': 'emr-key-pair',
        'InstanceGroups': [
            {
                "InstanceCount": 1,
                "InstanceRole": "CORE",
                "Name": "Core",
                "InstanceType": "m5.xlarge",
                "EbsConfiguration": {
                    "EbsBlockDeviceConfigs": [
                        {
                            "VolumeSpecification": {
                                "VolumeType": "gp2",
                                "SizeInGB": 32
                            },
                            "VolumesPerInstance": 2
                        }
                    ]
                }
            },
            {
                "InstanceCount": 1,
                "InstanceRole": "MASTER",
                "Name": "Primary",
                "InstanceType": "m5.xlarge",
                "EbsConfiguration": {
                    "EbsBlockDeviceConfigs": [
                        {
                            "VolumeSpecification": {
                                "VolumeType": "gp2",
                                "SizeInGB": 32
                            },
                            "VolumesPerInstance": 2
                        }
                    ]
                }
            }
        ]
    }
)

原因 #

对比CLI命令和boto3代码，核心差异在于网络配置缺失：

• CLI命令通过--ec2-attributes指定了SubnetId、EmrManagedMasterSecurityGroup、EmrManagedSlaveSecurityGroup，EMR服务角色只需要对这些明确指定的资源有操作权限即可。
• boto3代码的Instances参数中未配置上述网络项，EMR会尝试使用默认VPC的资源（默认子网、默认安全组），但你的服务角色没有访问默认VPC资源的权限，因此触发"EC2权限不足"的错误。

解决方案 #

修改boto3代码，在Instances字典中添加缺失的网络配置，与CLI中的参数保持一致：

Instances={
    'KeepJobFlowAliveWhenNoSteps': True,
    'Ec2KeyName': 'emr-key-pair',
    'Ec2SubnetId': 'subnet-0cf42a674f981a42a',  # 添加子网ID
    'EmrManagedMasterSecurityGroup': 'sg-07ced3d3223bet564',  # 添加主节点安全组
    'EmrManagedSlaveSecurityGroup': 'sg-04b7742eed31ec137',  # 添加核心节点安全组
    'InstanceGroups': [
        {
            "InstanceCount": 1,
            "InstanceRole": "CORE",
            "Name": "Core",
            "InstanceType": "m5.xlarge",
            "EbsConfiguration": {
                "EbsBlockDeviceConfigs": [
                    {
                        "VolumeSpecification": {
                            "VolumeType": "gp2",
                            "SizeInGB": 32
                        },
                        "VolumesPerInstance": 2
                    }
                ]
            }
        },
        {
            "InstanceCount": 1,
            "InstanceRole": "MASTER",
            "Name": "Primary",
            "InstanceType": "m5.xlarge",
            "EbsConfiguration": {
                "EbsBlockDeviceConfigs": [
                    {
                        "VolumeSpecification": {
                            "VolumeType": "gp2",
                            "SizeInGB": 32
                        },
                        "VolumesPerInstance": 2
                    }
                ]
            }
        }
    ]
}

内容的提问来源于stack exchange，提问作者DumbCoder