本地JupyterLab+SageMaker扩展作业无输出问题排查求助

我配置了带有SageMaker扩展的本地JupyterLab实例，用于在AWS中运行作业。作业已启动，S3输入目录中存在正确的.ipynb文件，但未生成任何带“output”前缀的文件，无结果可下载。已按照官方手册完成所有配置，且双重检查了IAM角色与权限，但仍无法解决问题，需要排查方向及相关日志的查看方式。

详细配置与排查过程 #

• 创建IAM用户lab并配置LabPolicy内联策略：

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EventBridgeSchedule",
            "Effect": "Allow",
            "Action": [
                "events:TagResource",
                "events:DeleteRule",
                "events:PutTargets",
                "events:DescribeRule",
                "events:EnableRule",
                "events:PutRule",
                "events:RemoveTargets",
                "events:DisableRule"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/sagemaker:is-scheduling-notebook-job": "true"
                }
            }
        },
        {
            "Sid": "IAMPassRoleToNotebookJob",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::*:role/SagemakerJupyterScheduler*",
            "Condition": {
                "StringLike": {
                    "iam:PassedToService": [
                        "sagemaker.amazonaws.com",
                        "events.amazonaws.com"
                    ]
                }
            }
        },
        {
            "Sid": "IAMListRoles",
            "Effect": "Allow",
            "Action": "iam:ListRoles",
            "Resource": "*"
        },
        {
            "Sid": "S3ArtifactsAccess",
            "Effect": "Allow",
            "Action": [
                "s3:PutEncryptionConfiguration",
                "s3:CreateBucket",
                "s3:PutBucketVersioning",
                "s3:ListBucket",
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetEncryptionConfiguration",
                "s3:DeleteObject",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::sagemaker-automated-execution-*"
            ]
        },
        {
            "Sid": "S3DriverAccess",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::sagemakerheadlessexecution-*"
            ]
        },
        {
            "Sid": "SagemakerJobs",
            "Effect": "Allow",
            "Action": [
                "sagemaker:DescribeTrainingJob",
                "sagemaker:StopTrainingJob",
                "sagemaker:DescribePipeline",
                "sagemaker:CreateTrainingJob",
                "sagemaker:DeletePipeline",
                "sagemaker:CreatePipeline"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/sagemaker:is-scheduling-notebook-job": "true"
                }
            }
        },
        {
            "Sid": "AllowSearch",
            "Effect": "Allow",
            "Action": "sagemaker:Search",
            "Resource": "*"
        },
        {
            "Sid": "SagemakerTags",
            "Effect": "Allow",
            "Action": [
                "sagemaker:ListTags",
                "sagemaker:AddTags"
            ],
            "Resource": [
                "arn:aws:sagemaker:*:*:pipeline/*",
                "arn:aws:sagemaker:*:*:space/*",
                "arn:aws:sagemaker:*:*:training-job/*",
                "arn:aws:sagemaker:*:*:user-profile/*"
            ]
        },
        {
            "Sid": "ECRImage",
            "Effect": "Allow",
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchGetImage"
            ],
            "Resource": "*"
        }
    ]
}

• 创建IAM角色SagemakerJupyterSchedulerRole，替换信任策略：

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "sagemaker.amazonaws.com",
                    "events.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

• 附加AmazonSageMakerFullAccess权限，并创建附加SagemakerJupyterSchedulerExecutionPolicy执行策略：

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::sagemaker-automated-execution-xxxxxxxxxxxx-us-east-1/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:CreateBucket",
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:ListAllMyBuckets",
                "s3:GetBucketCors",
                "s3:PutBucketCors"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketAcl",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::sagemaker-automated-execution-xxxxxxxxxxxx-us-east-1"
            ]
        }
    ]
}

• 作业配置：

Image: arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-base-python-38
Kernel: python3
Role ARN: arn:aws:iam::xxxxxxxxxxxx:role/SagemakerJupyterSchedulerRole
Input: s3://sagemaker-automated-execution-xxxxxxxxxxxx-us-east-1/
Output: s3://sagemaker-automated-execution-xxxxxxxxxxxx-us-east-1/

• 执行后仅在S3桶中看到输入文件：

❯ aws s3 ls s3://sagemaker-automated-execution-xxxxxxxxxxxx-us-east-1/ --recursive
2024-04-18 16:57:19       4536 helloworld2ipynb-helloworld2-45e83409-2024-04-18-16-57-17/input/hello-world2.ipynb
2024-04-18 16:49:00       4536 helloworld2ipynb-helloworld2-4d179d7f-2024-04-18-16-48-57/input/hello-world2.ipynb
2024-04-15 11:31:10       4536 helloworld2ipynb-helloworld2-ae714726-2024-04-15-11-31-08/input/hello-world2.ipynb
2024-04-15 10:47:33       2631 helloworld2ipynb-helloworld2-b516481a-2024-04-15-10-47-31/input/hello-world2.ipynb

• 测试发现SageMaker未实际运行作业，修改resource-metadata.json添加{"AppType": "JupyterLab"}后，作业启动但报错：

[FATAL tini (8)] exec amazon_sagemaker_scheduler failed: No such file or directory

• 最终定位问题：
  • 未遵循手册的严格版本要求，使用了第三方分支版本；
  • us-east-1区域S3桶的LocationConstraint为null，影响扩展逻辑导致问题：

(base) lab4:~$ aws s3api get-bucket-location --bucket sagemaker-automated-execution-xxxxxxxxxxxx-us-east-2
{
    "LocationConstraint": "us-east-2"
}
(base) lab4:~$ aws s3api get-bucket-location --bucket sagemaker-automated-execution-xxxxxxxxxxxx-us-east-1
{
    "LocationConstraint": null
}

内容的提问来源于stack exchange，提问作者danilabagroff