Fluentd输出至S3失败：核对凭证后仍遇Access Denied错误

尝试通过Fluentd将docker-compose部署的应用日志上传至AWS S3存储桶时遇到Access Denied错误，已核对存储桶名称、区域及密钥信息无误，求排查方向。

错误日志 #

fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-s3-1.3.0/lib/fluent/plugin/out_s3.rb:456:in `rescue in check_apikeys'    
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-s3-1.3.0/lib/fluent/plugin/out_s3.rb:451:in `check_apikeys'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-s3-1.3.0/lib/fluent/plugin/out_s3.rb:240:in `start'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:203:in `block in start'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:191:in `each'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:191:in `block in lifecycle'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:178:in `each'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:178:in `lifecycle'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:202:in `start'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:248:in `start'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:147:in `run'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:720:in `block in run_worker'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:971:in `main_process'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:711:in `run_worker'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:376:in `<top (required)>'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/bin/fluentd:23:in `load'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/bin/fluentd:23:in `<main>'
fluentd          | 2024-07-04 09:01:48 +0000 [error]: #0 unexpected error error_class=RuntimeError error="can't call S3 API. Please check your credentials or s3_region configuration. error = #<Aws::S3::Errors::AccessDenied: Access Denied>"
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 suppressed same stacktrace
fluentd          | 2024-07-04 09:01:48 +0000 [error]: Worker 0 finished unexpectedly with status 1

当前Fluentd S3输出配置 #

<match pattern>
  @type s3

  aws_key_id x
  aws_sec_key x
  s3_bucket x
  s3_region x

  path logs/${tag}/%Y/%m/%d/
  s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}

  # if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
  # need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
  <buffer tag,time>
    @type file
    path /var/log/fluent/s3
    timekey 3600 # 1 hour partition
    timekey_wait 10m
    timekey_use_utc true # use utc
  </buffer>
  <format>
    @type json
  </format>
</match>

• IAM权限不足：密钥正确不代表权限足够，检查IAM用户策略是否包含s3:PutObject、s3:ListBucket等必要权限，且权限范围覆盖目标存储桶及指定路径。
• 存储桶策略限制：存储桶自身的Bucket Policy可能设置了访问限制，比如拒绝非HTTPS请求、限制特定IP段访问，或者明确拒绝当前IAM用户的访问。
• 密钥被环境变量覆盖：如果docker-compose中通过环境变量传递AWS密钥，容器内的AWS_ACCESS_KEY_ID、AWS_SECRET_ACCESS_KEY可能覆盖配置文件中的值，需确认这些环境变量的正确性。
• 区域配置不匹配：部分S3区域需要指定正确的端点，或者插件对区域格式要求严格（比如是否带-1后缀），尝试添加s3_endpoint参数明确指定，或核对区域代码是否符合AWS标准。
• 缓冲区目录权限问题：Fluentd的缓冲区目录/var/log/fluent/s3如果没有读写权限，会导致日志无法写入缓冲区，进而触发API调用失败，检查容器内该目录是否为fluentd用户所有且权限正确。
• 插件版本兼容性：当前使用的fluent-plugin-s3 1.3.0与fluentd 1.14.6可能存在兼容性问题，尝试升级插件到最新稳定版，或降级到已知兼容的版本。
• 临时凭证过期：如果使用IAM角色临时凭证而非永久密钥，可能存在凭证过期的情况，确认是否使用了正确的永久密钥，或临时凭证是否在有效期内。
• 存储桶名称大小写问题：S3存储桶名称大小写敏感，确保配置中的s3_bucket值与实际存储桶名称完全一致，包括大小写。

内容的提问来源于stack exchange，提问作者Garou