无需Lambda：AWS DMS同步Aurora Postgres到Kinesis时如何省略无关字段？

可以直接通过AWS DMS的表映射配置实现，无需额外部署Lambda等服务。具体是在现有表映射规则中添加转换规则（transformation），重写输出的JSON结构，只保留data部分内容。

修改后的完整表映射配置 #

{
    "rules": [
        {
            "rule-type": "selection",
            "rule-id": "1",
            "rule-name": "1",
            "rule-action": "include",
            "object-locator": {
                "schema-name": "public",
                "table-name": "%"
            }
        },
        {
            "rule-type": "object-mapping",
            "rule-id": "2",
            "rule-name": "DefaultMapToKinesis",
            "rule-action": "map-record-to-record",
            "object-locator": {
                "schema-name": "public",
                "table-name": "testing"
            }
        },
        {
            "rule-type": "transformation",
            "rule-id": "3",
            "rule-name": "ExtractDataOnly",
            "rule-action": "modify-record",
            "object-locator": {
                "schema-name": "public",
                "table-name": "testing"
            },
            "transformation": {
                "type": "extract-field",
                "source-path": "$.data",
                "target-path": "$"
            }
        }
    ]
}

配置说明 #

• 新增的transformation规则指定针对public.testing表生效
• extract-field类型的转换会将原记录的$.data路径下的内容提取出来，替换整个输出记录（target-path: "$"）
• 配置生效后，Kinesis Data Stream中收到的消息将仅保留data部分的结构，例如：

  {
      "id": "5",
      "value": 1111.22
  }
  

扩展说明 #

如果需要对public schema下所有表都应用该过滤规则，只需将转换规则中的object-locator.table-name改为%即可。

内容的提问来源于stack exchange，提问作者Vegan Vegeta