AWS Firehose JSON键映射配置：解决DMS数据转Parquet报错

问题场景 #

数据通过AWS DMS传输至Kinesis Data Stream，再由Kinesis Firehose转换为Parquet格式存储到S3桶。已通过Glue爬虫创建DMS数据表，但DMS添加的元数据字段（如record-type、partition-key-type）包含“-”，导致Firehose转换Parquet时触发报错：

expected at the position 30 of ‘struct’ but ‘-’ is found.

由于AWS控制台无法配置JSON键映射，需通过AWS CLI完成配置。

完整配置片段 #

以下是Firehose的DataFormatConversionConfiguration完整配置，核心通过OpenXJsonSerDe的ColumnToJsonKeyMappings将带“-”的JSON字段映射为合法的Parquet列名：

"S3BackupMode": "Disabled",
"DataFormatConversionConfiguration": {
    "SchemaConfiguration": {
        "RoleARN": "arn:aws:iam::123456789012:role/sample-firehose-delivery-role",
        "DatabaseName": "sample-db",
        "TableName": "sample-table",
        "Region": "us-east-1",
        "VersionId": "LATEST"
    },
    "InputFormatConfiguration": {
        "Deserializer": {
            "OpenXJsonSerDe": {
                "ColumnToJsonKeyMappings": {
                    "record_type": "record-type",
                    "partition_key_type": "partition-key-type",
                    "schema_name": "schema-name",
                    "table_name": "table-name",
                    "transaction_id": "transaction-id"
                }
            }
        }
    },
    "OutputFormatConfiguration": {
        "Serializer": {
            "ParquetSerDe": {}
        }
    },
    "Enabled": true
}

CLI执行命令 #

使用以下AWS CLI命令更新Firehose流配置，需替换所有占位符为实际资源信息：

aws firehose update-delivery-stream --delivery-stream-name YOUR_STREAM_NAME --delivery-stream-configuration '{"S3DestinationConfiguration": {"RoleARN": "YOUR_ROLE_ARN", "BucketARN": "YOUR_S3_BUCKET_ARN", "S3BackupMode": "Disabled", "DataFormatConversionConfiguration": { "SchemaConfiguration": {"RoleARN": "arn:aws:iam::123456789012:role/sample-firehose-delivery-role", "DatabaseName": "sample-db", "TableName": "sample-table", "Region": "us-east-1", "VersionId": "LATEST"}, "InputFormatConfiguration": {"Deserializer": {"OpenXJsonSerDe": {"ColumnToJsonKeyMappings": {"record_type": "record-type","partition_key_type": "partition-key-type","schema_name":"schema-name","table_name":"table-name","transaction_id":"transaction-id"}}}}, "OutputFormatConfiguration": {"Serializer": {"ParquetSerDe": {}}}, "Enabled": true}}}'

需替换的占位说明 #

• YOUR_STREAM_NAME：目标Kinesis Firehose流的名称
• YOUR_ROLE_ARN：Firehose用于访问S3和Glue的IAM角色ARN
• YOUR_S3_BUCKET_ARN：存储Parquet文件的目标S3桶ARN
• SchemaConfiguration中的RoleARN、DatabaseName、TableName、Region：替换为你的Glue数据目录角色、数据库、表名及所属区域

内容的提问来源于stack exchange，提问作者Rita R