RHEL EC2实例上Amazon CloudWatch Agent无法推送日志的问题

问题背景 #

在RHEL EC2实例上通过Ansible角色安装CloudWatch Agent后，执行systemctl status amazon-cloudwatch-agent.service显示服务运行正常，但日志始终未出现在CloudWatch中。查看amazon-cloudwatch-agent.log，报错如下：

[outputs.cloudwatchlogs] Retried 114 time, going to sleep 46.304092322s before retrying.
[outputs.cloudwatchlogs] Aws error received when sending logs to <log-group>/<log-stream>: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>

    status code: 404, request id: 
[outputs.cloudwatchlogs] Retried 110 time, going to sleep 53.955265612s before retrying.

但使用AWS CLI执行以下命令能正常推送日志：

aws logs put-log-event --log-group-name <log-group> --log-stream-name <log-stream> --log-events timestamp=<xxxxxxxxxxxx>,message="testing aws cli"

原本以为实例无法获取AWS凭证，但CLI正常工作说明并非如此。已在common-config.toml中设置代理服务器，尝试将配置文件中的profile设为AmazonCloudWatchAgent，但因公司规定不能在实例上存放凭证文件，未指定凭证文件，配置如下：

# This common-config is used to configure items used for both ssm and cloudwatch access


## Configuration for shared credential.
## Default credential strategy will be used if it is absent here:
##  Instance role is used for EC2 case by default.
##  AmazonCloudWatchAgent profile is used for onPremise case by default.
[credentials]
   shared_credential_profile = "AmazonCloudWatchAgent"
#   shared_credential_file = "{file_name}"


## Configuration for proxy.
## System-wide environment-variable will be read if it is absent here.
## i.e. HTTP_PROXY/http_proxy; HTTPS_PROXY/https_proxy; NO_PROXY/no_proxy
## Note: system-wide environment-variable is not accessible when using ssm run-command.
## Absent in both here and environment-variable means no proxy will be used.
  [proxy]
     http_proxy = "http://xx.xx.xx.xx"
     https_proxy = "http://xx.xx.xx.xx"
#    no_proxy = "{domain}"

# [ssl]
#    ca_bundle_path = "{ca_bundle_file_path}"

修改配置后出现新错误：

[outputs.cloudwatchlogs] Retried 4 time, going to sleep 1.70156025s before retrying.
[outputs.cloudwatchlogs] Aws error received when sending logs to <log-group>/<log-stream>: SharedCredsLoad: failed to load shared credentials file
caused by: FailedRead: unable to open file
caused by: open /root/.aws/credentials: no such file or directory

想确认是否是CloudWatch配置遗漏导致这些错误？

问题分析与解决步骤 #

1. 核心问题：代理配置阻断实例元数据访问 #

CLI正常但Agent失败的关键差异在于代理的作用范围：AWS CLI默认会跳过对EC2实例元数据服务（IMDS）的代理请求，而CloudWatch Agent在配置全局代理后，会将所有请求（包括获取角色凭证的IMDS请求）都发往代理服务器，导致IMDS请求返回404；后续指定shared_credential_profile后，Agent又会尝试读取本地凭证文件，违反公司规定且无文件可读取。

2. 修复操作 #

步骤1：移除自定义凭证配置

注释或删除common-config.toml中的[credentials]段，让Agent默认使用EC2实例角色获取凭证：

# [credentials]
#    shared_credential_profile = "AmazonCloudWatchAgent"
#    shared_credential_file = "{file_name}"

步骤2：在代理配置中添加IMDS地址到no_proxy

EC2实例元数据地址为169.254.169.254，必须排除代理，让Agent直接访问IMDS：

[proxy]
   http_proxy = "http://xx.xx.xx.xx"
   https_proxy = "http://xx.xx.xx.xx"
   no_proxy = "169.254.169.254"

步骤3：重启CloudWatch Agent服务

systemctl restart amazon-cloudwatch-agent.service

3. 验证 #

• 查看amazon-cloudwatch-agent.log，确认不再出现EC2MetadataError和凭证相关错误
• 登录CloudWatch控制台，检查对应日志流是否开始接收日志

内容的提问来源于stack exchange，提问作者Jess_D_