EC2实例运行Logstash向Amazon ES转发日志遇插件问题

Hey there, let's work through this issue with the logstash-output-amazon_es plugin on your EC2 instance running the default Linux AMI. I’ve tackled similar production setups before, so here are the most critical checks and fixes to get this working properly:

1. Verify IAM Permissions & Role Setup #

• Your EC2 instance’s IAM role must have explicit permissions to interact with your AWS ES domain. At minimum, it needs es:ESHttpPut and es:ESHttpPost actions, targeted at your ES domain’s ARN (e.g., arn:aws:es:us-east-1:1234567890:domain/your-es-domain).
• Test if the role works directly from the EC2 instance: run curl -X GET https://your-es-endpoint/_cat/indices?v --aws-sigv4 aws:amazon:es (you’ll need the AWS CLI or curl with sigv4 support). If this returns a permission error, your IAM policy is missing required access.

2. Validate Plugin Configuration Syntax #

Double-check your amazon_es output block—small syntax errors are a common culprit. A correct production-ready config should look like this:

output {
  amazon_es {
    hosts => ["your-es-domain-endpoint"]
    region => "us-east-1" # Replace with your ES region
    index => "application-logs-%{+YYYY.MM.dd}" # Rotate logs daily
    # Leave access/secret keys blank if using EC2 IAM role (recommended)
    aws_access_key_id => ""
    aws_secret_access_key => ""
  }
}

Note: Never hardcode access keys in production—let the plugin pull temporary credentials from the EC2 instance metadata service automatically.

3. Check Version Compatibility #

The logstash-output-amazon_es plugin has strict version matching with Logstash. For example:

• Logstash 7.x requires plugin versions 6.x+
• Logstash 6.x needs plugin versions 5.x or lower
  To confirm your installed plugin version, run:

bin/logstash-plugin list --verbose logstash-output-amazon_es

If versions don’t align, reinstall the compatible plugin version:

bin/logstash-plugin install logstash-output-amazon_es --version 5.1.1

4. Audit Network & ES Access Policies #

• Ensure your EC2 security group allows outbound HTTPS (port 443) traffic to your ES domain’s VPC or public endpoint.
• Update your AWS ES domain’s access policy to explicitly allow your EC2 IAM role:

  {
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::1234567890:role/your-ec2-iam-role"
    },
    "Action": ["es:ESHttp*"],
    "Resource": "arn:aws:es:us-east-1:1234567890:domain/your-es-domain/*"
  }
  

5. Debug with Logstash Logs #

Dig into Logstash’s debug logs to pinpoint exact errors:

• Default log location: /var/log/logstash/logstash-plain.log
• Start Logstash in debug mode for more detail:

  bin/logstash -f your-config-file.conf --debug
  

Look for keywords like SignatureDoesNotMatch (IAM issue), ConnectionTimeout (network issue), or InvalidIndexNameException (index formatting error).

6. Confirm Instance Metadata Service (IMDS) Access #

If using an IAM role, your EC2 instance must reach the IMDS to fetch temporary credentials. Test this with:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/

If this returns your IAM role name, IMDS is working. If not, check if IMDS is disabled on your instance.

内容的提问来源于stack exchange，提问作者pmurphy86