Elasticsearch：从非服务器局域网设备收集日志的配置疑问

Hey there, let's break down the key considerations, underlying logic, and authentication handling for your cross-LAN Elastic Stack setup—since you're looking to collect home network logs rather than the remote server's own logs.

一、跨网连接的底层逻辑 #

First, let's clarify what's happening when your local laptop's FileBeat tries to talk to your Digital Ocean-hosted Logstash:

• Your home network sits behind a NAT (Network Address Translation) gateway (your router), so devices inside use private IPs that aren't directly reachable from the public internet. But since your Elastic Stack server is on a public cloud, your laptop can initiate outgoing connections to it as long as the server's target ports are accessible.
• The full connection flow is: Home Laptop (FileBeat) → Home Router (NAT outbound) → Public Internet → Digital Ocean Server (Firewall/Network) → Logstash. For this to work, the server must accept incoming traffic on the Logstash port, and your home network must allow outbound traffic to that port.
• When you run the setup command, FileBeat is trying to connect directly to Elasticsearch (not Logstash) to create index templates, dashboards, and other core resources—this is exactly why you saw the "couldn't connect to any configured elasticsearch hosts" error: you commented out the Elasticsearch output in your config, so FileBeat had no target for setup requests.

二、跨网配置的通用要点 #

Here are the non-negotiable checks and adjustments you need to make:

• Validate network reachability first
  • Before tweaking configs, confirm your laptop can reach the server's Logstash port. Use tools like nc (netcat) or telnet to test:

    nc -zv <your-server-public-ip> <logstash-port>
    

  • On your Digital Ocean server, ensure the firewall (e.g., ufw) allows incoming traffic on the Logstash port (default 5044 for FileBeat). Add a rule with:

    sudo ufw allow 5044/tcp
    

  • Double-check that your home router doesn't block outbound traffic to that port (most home networks allow this by default, but it's worth verifying if you have strict firewall rules).
• Adjust FileBeat config for cross-LAN use
  • For log shipping to Logstash: Make sure output.logstash.hosts uses your server's public IP address (not a private IP—those only work within the server's local cloud network).
  • For the setup command: You need to temporarily re-enable (or specify via command line) the Elasticsearch output with the server's public IP and authentication details—since setup requires direct access to Elasticsearch, not Logstash.
• Home device log collection prep
  • Ensure your home network devices (router, smart devices, etc.) are configured to send logs to your laptop's FileBeat instance (e.g., enable syslog on your router and point it to your laptop's private IP and the port FileBeat is listening on for syslog input).
  • Configure FileBeat's input section to match the log type from your home devices (e.g., type: syslog with the correct port).

三、服务器认证的处理方式 #

Security is critical when sending logs over the public internet—here's how to handle authentication at each layer:

• FileBeat ↔ Logstash Authentication
  • Use TLS/SSL to encrypt traffic and verify identities:
    1. Generate a self-signed SSL certificate on your Elastic Stack server (or use a trusted CA if you have one).
    2. Configure Logstash's beats input to use SSL, pointing to your certificate and key files:

       input {
         beats {
           port => 5044
           ssl => true
           ssl_certificate => "/path/to/cert.pem"
           ssl_key => "/path/to/key.pem"
         }
       }
       

    3. Update FileBeat's output.logstash config to trust the server's certificate:

       output.logstash:
         hosts: ["<server-public-ip>:5044"]
         ssl.certificate_authorities: ["/path/to/ca-cert.pem"]
       

  • Optional: Add API key or username/password authentication in Logstash for an extra layer of security.
• FileBeat ↔ ElasticStack Authentication (for setup)
  • If your Elasticsearch instance has security enabled (default in recent Elastic Stack versions), you need to provide credentials when running the setup command. You can either:
    • Re-enable the output.elasticsearch section in your FileBeat config with username and password fields, or
    • Pass credentials via the command line:

      filebeat setup -E output.elasticsearch.hosts=["https://<server-public-ip>:9200"] -E output.elasticsearch.username="elastic" -E output.elasticsearch.password="<your-elastic-password>"
      

• Server-side Access Restriction
  • Limit incoming traffic to your Elastic Stack ports (5044 for Logstash, 9200 for Elasticsearch) to only your home network's public IP address. This can be done via Digital Ocean's cloud firewall or your server's local ufw rules—this prevents random internet users from trying to connect to your services.

内容的提问来源于stack exchange，提问作者Addy