SageMaker创建处理作业时ContainerArguments参数语法错误排查

问题场景 #

执行以下SageMaker CLI创建处理任务的命令：

aws sagemaker create-processing-job --processing-job-name serialize-test \
        --processing-resources 'ClusterConfig={InstanceCount=1,InstanceType="ml.m5.large",VolumeSizeInGB=5}' \
        --app-specification ImageUri="awsid.dkr.ecr.us-east-1.amazonaws.com/custom-container-name",ContainerEntrypoint=["python",$SAGEMAKER_PROGRAM],ContainerArguments=["--data","s3://sagemaker-us-east-1-awsid/data/003.wav"] \
        --role-arn arn:aws:iam::awsid:role/rolename \
        --processing-output-config 'Outputs=[{OutputName="output1",S3Output={S3Uri="s3://sagemaker-us-east-1-awsid/output/",LocalPath="/opt/ml/processing/output1/",S3UploadMode="Continuous"}}]'

云日志报错：

2024-02-23T15:38:40.881-05:00   unknown option --data
2024-02-23T15:38:40.881-05:00   usage: python [option] ... [-c cmd | -m mod | file | -] [arg] ...
2024-02-23T15:38:40.881-05:00   Try `python -h' for more information.

使用的Dockerfile：

# SageMaker PyTorch image
FROM 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:2.0.0-cpu-py310

ENV PATH="/opt/ml/code:${PATH}"

# this environment variable is used by the SageMaker PyTorch container to determine our user code directory.
ENV SAGEMAKER_SUBMIT_DIRECTORY /opt/ml/code

# /opt/ml and all subdirectories are utilized by SageMaker, use the /code subdirectory to store your user code.
COPY serialize.py /opt/ml/code/serialize.py

ENV SAGEMAKER_PROGRAM serialize.py

serialize.py代码：

import argparse
import pickle
from sagemaker.serializers import DataSerializer


def main(args):
    data = args.data
    # Read the audio file
    with open(data, "rb") as f:
        audio_data = f.read()

    # Serialize the audio data
    serializer = DataSerializer(content_type='audio/wav')
    serialized_audio = serializer.serialize(audio_data)

    with open('output.pkl', 'wb') as f:
        pickle.dump(serialized_audio, f)


def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--data", type=str)
    args = parser.parse_args()

    return args


if __name__ == "__main__":
    args = parse_args()
    main(args)

疑问：问题是否出在--app-specification的ContainerArguments语法？具体错误是什么？

错误原因 #

核心问题有两点：

1. 环境变量解析失败：在CLI命令中，$SAGEMAKER_PROGRAM作为环境变量，直接写在--app-specification的参数里时，会被当成字符串字面量传递，而非解析为镜像中设置的serialize.py。此时Python会尝试执行名为$SAGEMAKER_PROGRAM的文件，显然不存在，进而把后续的--data当成Python自身的命令行参数，触发报错。
2. 不必要的Entrypoint指定：基于SageMaker官方PyTorch镜像的容器，会自动读取SAGEMAKER_PROGRAM环境变量，加载并执行对应的脚本，无需手动指定ContainerEntrypoint为python $SAGEMAKER_PROGRAM。手动指定反而破坏了容器的默认启动逻辑，导致参数传递路径错误。

修正方案 #

移除手动指定的ContainerEntrypoint，仅保留ContainerArguments，让容器使用默认启动逻辑运行脚本并接收参数。修正后的命令如下：

aws sagemaker create-processing-job --processing-job-name serialize-test \
        --processing-resources 'ClusterConfig={InstanceCount=1,InstanceType="ml.m5.large",VolumeSizeInGB=5}' \
        --app-specification ImageUri="awsid.dkr.ecr.us-east-1.amazonaws.com/custom-container-name",ContainerArguments=["--data","s3://sagemaker-us-east-1-awsid/data/003.wav"] \
        --role-arn arn:aws:iam::awsid:role/rolename \
        --processing-output-config 'Outputs=[{OutputName="output1",S3Output={S3Uri="s3://sagemaker-us-east-1-awsid/output/",LocalPath="/opt/ml/processing/output1/",S3UploadMode="Continuous"}}]'

额外说明 #

如果确实需要手动指定ContainerEntrypoint，必须直接写死脚本路径（而非依赖环境变量），确保参数正确传递给脚本：

--app-specification ImageUri="awsid.dkr.ecr.us-east-1.amazonaws.com/custom-container-name",ContainerEntrypoint=["python","/opt/ml/code/serialize.py"],ContainerArguments=["--data","s3://sagemaker-us-east-1-awsid/data/003.wav"]

但推荐使用第一种方案，依赖官方容器的默认逻辑，避免手动维护路径带来的潜在问题。

内容的提问来源于stack exchange，提问作者matsuo_basho