Fluentd(td-agent) base64解码过滤器v0.2.0解码异常求助

I’ve run into similar headaches with strict vs lenient Base64 decoding across different tools, so let’s break this down and fix it:

Root Cause: Non-Base64 Characters in Your Input #

Bash’s base64 command is surprisingly forgiving—it automatically ignores any characters that aren’t part of the standard Base64 alphabet (A-Z, a-z, 0-9, +, /, =), like spaces, newlines, or stray symbols. However, the fluent-plugin-filter-base64-decode v0.2.0 uses a strict decoding mode that doesn’t skip these non-compliant characters. This mismatch is almost certainly causing the garbage at the end of your decoded output.

Looking at your message field, I spotted a space embedded in the Base64 string:

dTE3PTt b3JlZj1odHRwcyUzQSUyRiUyRnd3dy5hdWRpYmxlLmNvbSUyRmVwJTJGZnJlZXRyaWFsJTNGc291cmNlX2NvZGUlM0RXQVBHQjEzNzEyMDcxODAwNEY/CTE=

That space is invisible to bash’s decoder but throws off the plugin’s strict parsing.

Solutions to Resolve the Issue #

1. Clean the Base64 String Before Decoding #

Add a record_transformer filter to strip non-Base64 characters from your message field before passing it to the base64-decode plugin. Insert this into your td-agent config ahead of the base64-decode block:

<filter your_input_tag>
  @type record_transformer
  enable_ruby true
  <record>
    message ${record["message"].gsub(/[^A-Za-z0-9\+\/=]/, '')}
  </record>
</filter>

This Ruby regex removes any character not in the Base64 alphabet, ensuring the decoder gets a fully compliant string to work with.

2. Upgrade the Plugin (If Available) #

The v0.2.0 version of the plugin might have a bug around lenient decoding. Check for updates and upgrade with:

td-agent-gem update fluent-plugin-filter-base64-decode

If a newer version fixes the strictness issue, that’s a quick win—if not, stick with the cleaning step above.

3. Bypass the Plugin with Custom Ruby Decoding #

If the plugin still misbehaves, you can skip it entirely and use Ruby’s built-in Base64 module directly in a record_transformer. This will match bash’s lenient behavior:

<filter your_input_tag>
  @type record_transformer
  enable_ruby true
  <record>
    message ${Base64.decode64(record["message"].gsub(/[^A-Za-z0-9\+\/=]/, ''))}
  </record>
</filter>

This combines the cleaning and decoding steps into one, using Ruby’s decoder which behaves consistently with bash’s base64 command.

Test the Fix #

After applying any of these changes, send a test log with your problematic string and verify the decoded output no longer has garbage at the end. Use td-agent-log-test to validate your config locally before deploying to production.

内容的提问来源于stack exchange，提问作者LeonidG