从Kinesis数据流解码Base64字符串出现乱码求助

问题场景 #

从Kinesis获取的Base64字符串，使用Node.js直接解码为UTF-8时出现乱码。AWS文档明确Kinesis使用Base64和UTF-8（octet-stream）规范，但常规解码方式无法得到正确内容。

原Node.js解码代码 #

const stream = "84mawgoBMRqXAQgAGpIBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNS4zMzk1MzBcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5OS9odW1pZGl0eVwiLCBcInZhbHVlXCI6IFwie1xcXCJyZWxhdGl2ZSBodW1pZGl0eVxcXCI6IDEzfVwifSIalgEIABqRASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDUuMzQxNTE5XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTEvY2FzaW5ndGVtcGVyYXR1cmVcIiwgXCJ2YWx1ZVwiOiBcIntcXFwiY2Vsc2l1c1xcXCI6IDEwfVwifSIalwEIABqSASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDUuMzQzNTM4XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTMvaHVtaWRpdHlcIiwgXCJ2YWx1ZVwiOiBcIntcXFwicmVsYXRpdmUgaHVtaWRpdHlcXFwiOiA5Mn1cIn0iGpMBCAAajgEie1wiZGF0ZXRpbWVcIjogXCIyMDIyLTEwLTA4IDEzOjU3OjA1LjM0NTc4NVwiLCBcInRvcGljXCI6IFwiZ3JlZW5mb3JnZS91dWlkL3V0aWxpdHk5L2NvMlwiLCBcInZhbHVlXCI6IFwie1xcXCJwYXJ0cyBwZXIgbWlsbGlvblxcXCI6IDQ0MX1cIn0iGpcBCAAakgEie1wiZGF0ZXRpbWVcIjogXCIyMDIyLTEwLTA4IDEzOjU3OjA1LjM1MjY1M1wiLCBcInRvcGljXCI6IFwiZ3JlZW5mb3JnZS91dWlkL3V0aWxpdHk5L2FpcnByZXNzdXJlXCIsIFwidmFsdWVcIjogXCJ7XFxcImhlY3RvIHBhc2NhbHNcXFwiOiA1NzF9XCJ9IhqXAQgAGpIBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNS4zNTU5NzVcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5MS9odW1pZGl0eVwiLCBcInZhbHVlXCI6IFwie1xcXCJyZWxhdGl2ZSBodW1pZGl0eVxcXCI6IDY5fVwifSIapgEIABqhASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDUuMzU4MTA5XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTYvY2FzaW5ndGVtcGVyYXR1cmVcIiwgXCJ2YWx1ZVwiOiBcIntcXFwiY2Vsc2l1c1xcXCI6IDEyLjA5NzU4NTcxODcyNjkzOH1cIn0iGpQBCAAajwEie1wiZGF0ZXRpbWVcIjogXCIyMDIyLTEwLTA4IDEzOjU3OjA1LjM2NjczN1wiLCBcInRvcGljXCI6IFwiZ3JlZW5mb3JnZS91dWlkL3V0aWxpdHkzL2NvMlwiLCBcInZhbHVlXCI6IFwie1xcXCJwYXJ0cyBwZXIgbWlsbGlvblxcXCI6IDE1NDR9XCJ9IhqYAQgAGpMBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNS4zNzE5MjZcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5Ni9haXJwcmVzc3VyZVwiLCBcInZhbHVlXCI6IFwie1xcXCJoZWN0byBwYXNjYWxzXFxcIjogMTEwMH1cIn0iGqYBCAAaoQEie1wiZGF0ZXRpbWVcIjogXCIyMDIyLTEwLTA4IDEzOjU3OjA1LjM3NDQ1M1wiLCBcInRvcGljXCI6IFwiZ3JlZW5mb3JnZS91dWlkL3V0aWxpdHk5L2Nhc2luZ3RlbXBlcmF0dXJlXCIsIFwidmFsdWVcIjogXCJ7XFxcImNlbHNpdXNcXFwiOiAwLjk0MTg2MTc4NDk5MTgxNzh9XCJ9IhqiAQgAGp0BIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNS4zNzk0MzRcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5Ni9haXJ0ZW1wZXJhdHVyZVwiLCBcInZhbHVlXCI6IFwie1xcXCJjZWxzaXVzXFxcIjogMTYuMTM3OTM2NjY0Njg4MTl9XCJ9IhqXAQgAGpIBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNS4zODY3ODVcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5MS9haXJwcmVzc3VyZVwiLCBcInZhbHVlXCI6IFwie1xcXCJoZWN0byBwYXNjYWxzXFxcIjogOTg1fVwifSIamAEIABqTASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDYuMzM2MDI5XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTMvYWlycHJlc3N1cmVcIiwgXCJ2YWx1ZVwiOiBcIntcXFwiaGVjdG8gcGFzY2Fsc1xcXCI6IDEwMzh9XCJ9IhqWAQgAGpEBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNi4zNDY3MDFcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5MS9jYXNpbmd0ZW1wZXJhdHVyZVwiLCBcInZhbHVlXCI6IFwie1xcXCJjZWxzaXVzXFxcIjogMTB9XCJ9IhqXAQgAGpIBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNi4zNTA2NzhcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5My9odW1pZGl0eVwiLCBcInZhbHVlXCI6IFwie1xcXCJyZWxhdGl2ZSBodW1pZGl0eVxcXCI6IDk0fVwifSIakwEIABqOASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDYuMzUzODU2XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTkvY28yXCIsIFwidmFsdWVcIjogXCJ7XFxcInBhcnRzIHBlciBtaWxsaW9uXFxcIjogNDM3fVwifSIamwEIABqWASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDYuMzU3MjUwXCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTkvaGlnaGxldmVsZmxvYXRzd2l0Y2hcIiwgXCJ2YWx1ZVwiOiBcIntcXFwiZW5hYmxlZFxcXCI6IHRydWV9XCJ9IhqmAQgAGqEBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNi4zNzEyMDRcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5Ni9jYXNpbmd0ZW1wZXJhdHVyZVwiLCBcInZhbHVlXCI6IFwie1xcXCJjZWxzaXVzXFxcIjogMTIuMDk3NTg1NzE4NzI2OTM4fVwifSIalAEIABqPASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDYuMzkwMjg3XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTYvY28yXCIsIFwidmFsdWVcIjogXCJ7XFxcInBhcnRzIHBlciBtaWxsaW9uXFxcIjogMTUzNH1cIn0iGqIBCAAanQEie1wiZGF0ZXRpbWVcIjogXCIyMDIyLTEwLTA4IDEzOjU3OjA2LjQwMjcyM1wiLCBcInRvcGljXCI6IFwiZ3JlZW5mb3JnZS91dWlkL3V0aWxpdHkzL2FpcnRlbXBlcmF0dXJlXCIsIFwidmFsdWVcIjogXCJ7XFxcImNlbHNpdXNcXFwiOiA1Ny4xMjgyMTE3MTk1MzEwN31cIn0io+JZCmdplVwO/VbJVxs5Dg=="

var payload = Buffer.from(stream, 'base64').toString('utf8');
console.log('Decoded payload: %s', payload);

Python端Kafka生产者代码 #

kafka_producer = KafkaProducer( bootstrap_servers= "10.0.0.129:9092",
                                value_serializer=lambda v: json.dumps(v).encode('utf-8'))
data = {
                "datetime": str(datetime.now()),
                "topic": str(msg.topic),
                "value": str(msg.payload.decode()),
                "environment": "test-v1"
        }
kafka_producer.send("greenforge-events", data)

问题原因 #

获取到的Base64字符串是多条Kinesis记录的聚合格式，而非单条JSON数据。Kinesis会自动聚合多条记录，每条记录前添加4字节的大端字节序长度前缀。直接将整个Buffer解码为UTF-8时，这些非UTF-8的长度字节会被解析为乱码。

修复方案 #

解析聚合记录，循环读取每条记录的长度前缀，再提取对应长度的内容进行解码：

const stream = "84mawgoBMRqXAQgAGpIBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNS4zMzk1MzBcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5OS9odW1pZGl0eVwiLCBcInZhbHVlXCI6IFwie1xcXCJyZWxhdGl2ZSBodW1pZGl0eVxcXCI6IDEzfVwifSIalgEIABqRASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDUuMzQxNTE5XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTEvY2FzaW5ndGVtcGVyYXR1cmVcIiwgXCJ2YWx1ZVwiOiBcIntcXFwiY2Vsc2l1c1xcXCI6IDEwfVwifSIalwEIABqSASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDUuMzQzNTM4XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTMvaHVtaWRpdHlcIiwgXCJ2YWx1ZVwiOiBcIntcXFwicmVsYXRpdmUgaHVtaWRpdHlcXFwiOiA5Mn1cIn0iGpMBCAAajgEie1wiZGF0ZXRpbWVcIjogXCIyMDIyLTEwLTA4IDEzOjU3OjA1LjM0NTc4NVwiLCBcInRvcGljXCI6IFwiZ3JlZW5mb3JnZS91dWlkL3V0aWxpdHk5L2NvMlwiLCBcInZhbHVlXCI6IFwie1xcXCJwYXJ0cyBwZXIgbWlsbGlvblxcXCI6IDQ0MX1cIn0iGpcBCAAakgEie1wiZGF0ZXRpbWVcIjogXCIyMDIyLTEwLTA4IDEzOjU3OjA1LjM1MjY1M1wiLCBcInRvcGljXCI6IFwiZ3JlZW5mb3JnZS91dWlkL3V0aWxpdHk5L2FpcnByZXNzdXJlXCIsIFwidmFsdWVcIjogXCJ7XFxcImhlY3RvIHBhc2NhbHNcXFwiOiA1NzF9XCJ9IhqXAQgAGpIBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNS4zNTU5NzVcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5MS9odW1pZGl0eVwiLCBcInZhbHVlXCI6IFwie1xcXCJyZWxhdGl2ZSBodW1pZGl0eVxcXCI6IDY5fVwifSIapgEIABqhASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDUuMzU4MTA5XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTYvY2FzaW5ndGVtcGVyYXR1cmVcIiwgXCJ2YWx1ZVwiOiBcIntcXFwiY2Vsc2l1c1xcXCI6IDEyLjA5NzU4NTcxODcyNjkzOH1cIn