求助：AWS EventBridge触发SageMaker+StepFunctions ML工作流失败

问题背景 #

已在SageMaker Notebook中通过Python SDK创建并成功执行单步骤ML预处理工作流，但通过EventBridge触发同一状态机时执行失败，需实现每周自动触发的自动化方案。

可能的失败原因 #

• 权限缺失：EventBridge触发角色无states:StartExecution权限，或StepFunctions执行角色缺少SageMaker/S3操作权限
• 输入参数不匹配：EventBridge传递的输入格式与状态机定义的schema不符，或PreprocessingJobName生成逻辑有问题
• 环境依赖：状态机定义中依赖Notebook本地变量（如input_data_path），未完成动态配置

解决方案步骤 #

1. 修复权限配置 #

调整StepFunctions执行角色权限

确保workflow_execution_role关联的IAM策略包含以下权限：

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sagemaker:CreateProcessingJob",
                "sagemaker:DescribeProcessingJob",
                "sagemaker:StopProcessingJob"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::your-account-id:role/your-sagemaker-execution-role"
        }
    ]
}

给EventBridge添加触发权限

允许EventBridge调用StepFunctions状态机：

import boto3

sf_client = boto3.client('stepfunctions')
state_machine_arn = branching_workflow.state_machine_arn

sf_client.add_permission(
    StateMachineArn=state_machine_arn,
    Action='states:StartExecution',
    Principal='events.amazonaws.com',
    SourceArn='arn:aws:events:your-region:your-account-id:rule/your-rule-name'
)

2. 优化状态机输入参数 #

避免依赖外部传入PreprocessingJobName，改用StepFunctions内置函数自动生成唯一名称，修改ProcessingStep的job_name配置：

from stepfunctions.states import StatesFormat, StatesUUID

processing_step = ProcessingStep(
    "my-processing-step",
    processor=script_processor,
    job_name=StatesFormat('preprocess-{}'.format(StatesUUID())),    
    inputs=inputs,
    outputs=outputs,
    container_arguments=["--train-test-split-ratio", "0.2"],
    container_entrypoint=["python3", "/opt/ml/processing/input/code/preprocessing.py"],
)

这样无需在EventBridge触发时传递额外输入，简化触发逻辑。

3. 通过Python SDK创建EventBridge每周触发规则 #

import boto3
import uuid

eventbridge = boto3.client('events')
state_machine_arn = branching_workflow.state_machine_arn

# 创建每周日凌晨触发的规则（可调整cron表达式）
rule_name = f"weekly-ml-preprocessing-{uuid.uuid4().hex[:8]}"
rule_arn = eventbridge.put_rule(
    Name=rule_name,
    ScheduleExpression='cron(0 0 ? * SUN *)',
    State='ENABLED',
    Description='Weekly trigger for SageMaker preprocessing workflow'
)['RuleArn']

# 绑定目标为StepFunctions状态机
eventbridge.put_targets(
    Rule=rule_name,
    Targets=[
        {
            'Id': '1',
            'Arn': state_machine_arn,
            'Input': '{}'  # 无需额外输入时传空对象
        }
    ]
)

# 给规则添加权限
sf_client.add_permission(
    StateMachineArn=state_machine_arn,
    Action='states:StartExecution',
    Principal='events.amazonaws.com',
    SourceArn=rule_arn
)

4. 故障排查 #

• 查看StepFunctions执行日志：进入失败的执行记录，检查Execution event history中的错误信息（如权限拒绝、参数错误）
• 测试EventBridge规则：在控制台手动触发规则，验证输入参数是否符合状态机schema

内容的提问来源于stack exchange，提问作者stackword_0