AWS EMR Jupyter Notebook调用start_notebook_execution时集群ID为null的问题

我在AWS EMR Jupyter Notebook中调用boto3的start_notebook_execution接口时，收到错误提示：The EMR cluster Id null is not available，但我已经验证过：

• 传入的集群ID并非null、None或空字符串
• 目标集群处于运行状态（我正从该集群的Notebook执行操作）

该代码在其他集群和Notebook上可正常运行，因此问题大概率不在代码本身。在出现此错误前，我还遇到过一系列权限问题，添加对应IAM策略后才出现当前的集群ID为null的错误。

此前遇到的权限错误 #

错误1：iam:PassRole权限不足 #

An error was encountered:
An error occurred (AccessDeniedException) when calling the StartNotebookExecution operation: User: arn:aws:sts::<account>:assumed-role/EMR_EC2_DefaultRole/i-<roleid> is not authorized to perform: iam:PassRole on resource: arn:aws:iam::<account>:role/EMR_Notebooks_DefaultRole because no identity-based policy allows the iam:PassRole action
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the StartNotebookExecution operation: User: arn:aws:sts::<account>:assumed-role/EMR_EC2_DefaultRole/i-<roleid> is not authorized to perform: iam:PassRole on resource: arn:aws:iam::<account>:role/EMR_Notebooks_DefaultRole because no identity-based policy allows the iam:PassRole action

错误2：elasticmapreduce:StartNotebookExecution权限不足（notebook-execution资源） #

An error was encountered:
An error occurred (AccessDeniedException) when calling the StartNotebookExecution operation: User: arn:aws:sts::<account>:assumed-role/EMR_EC2_DefaultRole/i-<roleid> is not authorized to perform: elasticmapreduce:StartNotebookExecution on resource: arn:aws:elasticmapreduce:us-east-1:<account>:notebook-execution/* because no identity-based policy allows the elasticmapreduce:StartNotebookExecution action
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the StartNotebookExecution operation: User: arn:aws:sts::<account>:assumed-role/EMR_EC2_DefaultRole/i-<roleid> is not authorized to perform: elasticmapreduce:StartNotebookExecution on resource: arn:aws:elasticmapreduce:us-east-1:<account>:notebook-execution/* because no identity-based policy allows the elasticmapreduce:StartNotebookExecution action

错误3：elasticmapreduce:StartNotebookExecution权限不足（editor资源） #

An error was encountered:
An error occurred (AccessDeniedException) when calling the StartNotebookExecution operation: User: arn:aws:sts::<account>:assumed-role/EMR_EC2_DefaultRole/i-<roleid> is not authorized to perform: elasticmapreduce:StartNotebookExecution on resource: arn:aws:elasticmapreduce:us-east-1:<account>:editor/e-<editor> because no identity-based policy allows the elasticmapreduce:StartNotebookExecution action
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the StartNotebookExecution operation: User: arn:aws:sts::<account>:assumed-role/EMR_EC2_DefaultRole/i-<roleid> is not authorized to perform: elasticmapreduce:StartNotebookExecution on resource: arn:aws:elasticmapreduce:us-east-1:<account>:editor/e-<editor> because no identity-based policy allows the elasticmapreduce:StartNotebookExecution action

当前已添加的内联IAM策略 #

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "elasticmapreduce:StartNotebookExecution",
            "Resource": [
                "arn:aws:elasticmapreduce:<region>:<account>:editor/*",
                "arn:aws:elasticmapreduce:<region>:<account>:notebook-execution/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::<account>:role/EMR_Notebooks_DefaultRole"
        }
    ]
}

排查与解决建议 #

1. 检查EMR_Notebooks_DefaultRole的信任策略
   确保该角色允许EMR_EC2_DefaultRole传递给它，需在EMR_Notebooks_DefaultRole的信任策略中添加以下语句：

   {
       "Effect": "Allow",
       "Principal": {
           "AWS": "arn:aws:iam::<account>:role/EMR_EC2_DefaultRole"
       },
       "Action": "sts:AssumeRole"
   }
   

   否则即使拥有iam:PassRole权限，目标角色不允许被当前角色传递，会导致参数传递异常，最终出现集群ID为null的错误。

2. 补充集群资源的权限
   当前策略仅包含editor和notebook-execution资源，但StartNotebookExecution操作还需要对目标EMR集群的权限。需在策略中添加：

   {
       "Effect": "Allow",
       "Action": "elasticmapreduce:StartNotebookExecution",
       "Resource": "arn:aws:elasticmapreduce:<region>:<account>:cluster/<cluster-id>"
   }
   

   若需要允许所有集群，可替换为arn:aws:elasticmapreduce:<region>:<account>:cluster/*。

3. 验证参数传递的完整性
   尽管确认传入的集群ID不为空，仍建议在调用start_notebook_execution前打印所有传入参数（包括ClusterId、ServiceRole等必填项），确认参数未被意外覆盖或丢失：

   import boto3
   emr = boto3.client('emr')
   params = {
       'EditorId': 'your-editor-id',
       'NotebookExecutionName': 'test-execution',
       'ClusterId': 'your-cluster-id',
       'ServiceRole': 'EMR_Notebooks_DefaultRole'
   }
   print(params)  # 确认ClusterId存在且正确
   response = emr.start_notebook_execution(**params)
   

4. 检查EMR集群的标签和权限边界
   确认目标集群未设置限制访问的标签，且当前角色的权限边界未禁止访问该集群。

5. 刷新IAM角色权限
   IAM权限存在缓存，建议重启Notebook实例，或等待5-10分钟让权限生效后再测试。

内容的提问来源于stack exchange，提问作者mwarrior