通过Lambda+Firehose写入Redshift遇Redshift.DataLoadFailed错误排查

Hey there, let's break down what's going wrong here and how to fix it quickly.

The Root Cause #

Your issue boils down to missing newline characters (\n) at the end of each record when sending data to Firehose. Right now, your Lambda is passing each record as a raw string without a trailing newline—so Firehose just shoves all the records together into one giant line in S3. When Redshift tries to load this, it can't tell where one record ends and the next begins, hence the "Delimiter not found" error.

The Fix #

Update your Lambda code to append a newline character to each record's Data field. This tells Firehose (and eventually Redshift) where each individual record stops:

firehose.putRecordBatch({
  DeliveryStreamName: 'raincover-nea-rainfall',
  Records: chunks[0].map(item => {
    console.log(item) // Example: "S118","2018-05-21T00:10:00+08:00","0"
    // Add newline to the end of each record
    return { Data: item + '\n' }
  })
}, (err, data) => {
  if (err) {
    console.log('FAILED TO ADD CHUNK', err.message)
    return //reject(err)
  }
  console.log('CHUNK ADDED')
  console.log(data) //resolve(data)
})

Quick Checks to Confirm #

• Double-check your Firehose Redshift destination settings: make sure the row delimiter is set to \n (this is the default, but it's worth verifying if someone changed it)
• After deploying the fix, check the S3 bucket again—your data should now be split into individual lines like you expected:

  "S77","2018-05-21T00:05:00+08:00","0"
  "S109","2018-05-21T00:05:00+08:00","0"
  "S117","2018-05-21T00:05:00+08:00","0"
  

Once you make this change, Redshift will be able to parse each record correctly and your load errors should disappear.

内容的提问来源于stack exchange，提问作者Jiew Meng