AWS Step Functions Wait状态输出配置问题及并行Glue作业需求

核心是确保Wait状态不会覆盖启动Glue作业返回的关键元数据（JobRunId、JobName等），具体操作：

• 调整Wait状态的ResultPath/OutputPath配置：
  默认Wait状态会用自身输出覆盖整个输入，导致后续Get status状态拿不到正确的JobRunId。修改Wait状态配置，将Wait输出存到子字段，保留原有输入：

  {
    "Type": "Wait",
    "Seconds": 60,
    "ResultPath": "$.waitInfo",
    "Next": "Get Glue Job Status"
  }
  

• 确认Get status状态的参数读取路径：
  确保Get Glue Job Status状态从原有输入字段读取参数，而非被Wait覆盖的内容：

  {
    "Type": "Task",
    "Resource": "arn:aws:states:::glue:get-job-run",
    "Parameters": {
      "JobName.$": "$.JobName",
      "RunId.$": "$.JobRunId"
    },
    "Next": "Check Job Status"
  }
  

• 替代方案：用Pass状态传递元数据
  如果Wait状态配置不好调整，可在Wait之后加一个Pass状态，显式传递需要的作业参数：

  {
    "Type": "Pass",
    "Parameters": {
      "JobName.$": "$.JobName",
      "JobRunId.$": "$.JobRunId"
    },
    "Next": "Get Glue Job Status"
  }
  

通过错误捕获和并行分支的容错配置实现：

• 给Parallel状态添加全局Catch规则：
  捕获并行分支中Glue作业的失败异常，允许其他分支继续执行，最终收集所有结果：

  {
    "Type": "Parallel",
    "Branches": [
      {
        "StartAt": "Start Glue Job 1",
        "States": { /* 作业1的启动-轮询流程 */ }
      },
      {
        "StartAt": "Start Glue Job 2",
        "States": { /* 作业2的启动-轮询流程 */ }
      }
    ],
    "Catch": [
      {
        "ErrorEquals": ["Glue.JobFailedException", "Glue.EntityNotFoundException"],
        "ResultPath": "$.failedBranch",
        "Next": "Collect Results"
      }
    ],
    "Next": "Collect Results"
  }
  

• 给单个分支配置独立容错：
  在每个并行分支的末尾添加Catch规则，处理该分支的失败，确保分支不会中断：

  {
    "StartAt": "Start Glue Job",
    "States": {
      "Start Glue Job": { /* 启动作业配置 */ },
      "Get Job Status": { /* 查询状态配置 */ },
      "Check Status": {
        "Type": "Choice",
        "Choices": [
          { "Variable": "$.JobRun.State", "StringEquals": "SUCCEEDED", "Next": "Mark Job Success" },
          { "Variable": "$.JobRun.State", "StringEquals": "FAILED", "Next": "Mark Job Failed" }
        ]
      },
      "Mark Job Failed": {
        "Type": "Pass",
        "Parameters": { "status": "FAILED", "jobName": "$.JobName" },
        "End": true
      },
      "Mark Job Success": {
        "Type": "Pass",
        "Parameters": { "status": "SUCCEEDED", "jobName": "$.JobName" },
        "End": true
      }
    }
  }
  

• 用Map状态实现批量并行容错：
  如果是批量启动相同类型的Glue作业，用Map状态并设置容错参数：

  {
    "Type": "Map",
    "ItemsPath": "$.jobList",
    "MaxConcurrency": 5,
    "ToleratedFailureCount": 2, // 允许最多2个作业失败
    "Iterator": {
      "StartAt": "Start Glue Job",
      "States": { /* 单个作业的启动-轮询流程 */ }
    },
    "Next": "Process Results"
  }
  

内容的提问来源于stack exchange，提问作者Guillaume