Eventbridge Pipes对接DynamoDB流与事件总线失败排查求助

问题背景 #

尝试通过Terraform创建从DynamoDB流读取数据并推送至自定义事件总线的EventBridge Pipe，但Pipe持续失败，需排查问题。

配置代码 #

data "aws_iam_policy_document" "pipe_policy" {
  statement {
    sid    = ""
    effect = "Allow"
    principals {
      identifiers = ["pipes.amazonaws.com"]
      type        = "Service"
    }
    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "eventbridge_dynamodb_role" {
  name = "eventbridge_dynamodb_role"

  managed_policy_arns = [
    "arn:aws:iam::aws:policy/AmazonDynamoDBReadOnlyAccess",
    "arn:aws:iam::aws:policy/AmazonEventBridgeFullAccess"
  ]

  assume_role_policy = data.aws_iam_policy_document.pipe_policy.json
}

# 创建自定义事件总线
resource "aws_cloudwatch_event_bus" "example_event_bus" {
  name = "example_event_bus"
}

resource "awscc_pipes_pipe" "pipe" {
  name     = "pipe-customer-request"
  role_arn = aws_iam_role.eventbridge_dynamodb_role.arn

  source = aws_dynamodb_table.example_table.stream_arn

  source_parameters = {
    dynamo_db_stream_parameters = {
      starting_position = "LATEST"
      batch_size = 1
    }
  }

  target = aws_cloudwatch_event_bus.example_event_bus.arn
}

监控面板 #

具体排查步骤 #

1. 验证IAM权限细节 #

虽使用了托管策略，仍需确认：

• AmazonDynamoDBReadOnlyAccess是否包含DynamoDB流核心权限：dynamodb:GetRecords、dynamodb:GetShardIterator、dynamodb:DescribeStream
• AmazonEventBridgeFullAccess是否允许对目标事件总线执行events:PutEvents动作，无资源级限制
• 使用IAM策略模拟器，以pipes.amazonaws.com为服务主体，模拟执行上述权限，验证是否通过

2. 启用并查看Pipe日志 #

默认Pipe可能未开启日志，需补充配置：

1. 创建CloudWatch日志组：

resource "aws_cloudwatch_log_group" "pipe_logs" {
  name              = "/aws/eventbridge/pipes/pipe-customer-request"
  retention_in_days = 7
}

2. 在awscc_pipes_pipe资源中添加日志配置：

log_configuration = {
  cloudwatch_logs_log_destination = aws_cloudwatch_log_group.pipe_logs.arn
  level                           = "INFO"
}

3. 确保Pipe角色拥有logs:CreateLogStream、logs:PutLogEvents权限（可通过自定义策略补充）

完成后在Pipe控制台的日志标签页查看具体错误信息，比如权限被拒、数据格式异常等。

3. 检查DynamoDB流有效性 #

• 确认目标DynamoDB表已开启流功能，流视图类型（如NEW_IMAGE）符合需求
• 验证Terraform引用的stream_arn是否正确，可通过AWS控制台查看表的流ARN对比确认

4. 确认事件总线配置 #

• 检查自定义事件总线是否成功创建，ARN与Terraform配置一致
• 若事件总线设置了资源策略，需确保策略允许Pipe角色向其发送事件，示例策略：

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT_ID:role/eventbridge_dynamodb_role"
      },
      "Action": "events:PutEvents",
      "Resource": "arn:aws:events:REGION:ACCOUNT_ID:event-bus/example_event_bus"
    }
  ]
}

内容的提问来源于stack exchange，提问作者Kaizad Wadia