如何用Fluentd内置过滤器替换日志中id、pw/pass参数的敏感值？

Absolutely! You don't need a custom plugin for this—Fluentd's filter_record_transformer with Ruby's gsub functionality is perfect for masking those sensitive id, pw, and pass parameters in your URLs. Here's exactly how to set it up:

Step-by-Step Configuration #

First, let's assume your logs contain the URLs in a field named message (adjust this to match your actual field name if it's different). You'll use the record_transformer filter with Ruby-enabled string manipulation to replace the sensitive values:

<filter your_input_tag>
  @type record_transformer
  enable_ruby true  # Required to use Ruby's gsub method
  <record>
    message ${record["message"].gsub(/id=[^&]+/, 'id=*').gsub(/pw=[^&]+/, 'pw=****').gsub(/pass=[^&]+/, 'pass=****')}
  </record>
</filter>

Breakdown of the Logic: #

• enable_ruby true: This enables Ruby's string methods (like gsub) within the filter, which is essential for the regex-based replacements.
• Regex Patterns:
  • /id=[^&]+/: Matches id= followed by any characters until the next & (so it only targets the id parameter's value, not other parts of the URL).
  • /pw=[^&]+/ and /pass=[^&]+/: Do the same for the pw and pass parameters, replacing their values with ****.
• Chained gsub Calls: We chain multiple gsub methods to handle all three parameter types in one pass.

Why Grep Filter Isn't the Right Fit #

The grep filter is designed to filter in or out entire log records based on patterns, not modify the content of individual fields. So it won't help you replace sensitive values within URLs—stick with record_transformer for this use case.

Additional Tips #

• If your URLs are stored in a different field (e.g., request_url), just replace record["message"] with record["request_url"] in the configuration.
• Test your regex patterns first! You can use tools like Ruby's irb to verify that gsub works as expected with your actual URL formats.
• If you have other sensitive parameters to mask, add another .gsub(/param_name=[^&]+/, 'param_name=****') to the chain.

内容的提问来源于stack exchange，提问作者monouser