Logstash无法连接Kafka：Docker环境下Elastic Stack主题消费失败求助

Hey there, let's work through this Logstash-Kafka connection issue step by step—Docker networking quirks are usually the culprit here!

1. Fix the Docker Network Misconfiguration #

The biggest red flag in your config is bootstrap_servers => 'localhost:9092'. When running Logstash inside a Docker container, localhost points to the Logstash container itself, not your host machine where Kafka might be running (or your Kafka container).

To fix this:

• Replace localhost:9092 with the Docker container name or internal IP address of your Kafka instance. For example, if your Kafka container is named kafka, use bootstrap_servers => 'kafka:9092'.
• Double-check your Kafka's advertised.listeners configuration. Make sure it's set to allow connections from other Docker containers. A common working setup for Kafka in Docker looks like:

  LISTENERS=PLAINTEXT://0.0.0.0:9092,PLAINTEXT_INTERNAL://kafka:9093
  ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://kafka:9093
  

  This lets your host use localhost:9092 and Docker containers use kafka:9093 (adjust the port in your Logstash config if you use the internal listener).

2. Verify the Kafka Topic Exists #

Before troubleshooting connections, confirm the quickstart-data topic actually exists:

• If you're running commands on your host, use:

  kafka-topics.sh --list --bootstrap-server localhost:9092
  

• If Kafka is in a container, exec into it first:

  docker exec -it <kafka-container-name> kafka-topics.sh --list --bootstrap-server localhost:9092
  

If the topic doesn't exist, create it with:

kafka-topics.sh --create --topic quickstart-data --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

3. Check Logstash Logs for Specific Errors #

Grab the Logstash container logs to get precise error details—this will tell you if it's a connection timeout, unknown topic, or auth issue:

docker logs <logstash-container-name>

Look for lines like Connection refused (network issue) or Unknown topic or partition (topic doesn't exist).

4. Ensure Containers Are on the Same Docker Network #

Make sure your Logstash and Kafka containers are attached to the same Docker network. If not, they can't communicate:

• List existing networks: docker network ls
• Check which network your containers use: docker inspect <container-name> | grep Network
• If needed, create a shared network and reattach containers:

  docker network create kafka-elastic-network
  docker network connect kafka-elastic-network <kafka-container-name>
  docker network connect kafka-elastic-network <logstash-container-name>
  

Updated Logstash Config Example #

Here's a revised config with the network fix plus a stdout output to debug:

input {
  kafka {
    bootstrap_servers => 'kafka:9092' # Replace with your Kafka container name/IP
    topics => 'quickstart-data'
    # Add these if your Kafka uses SASL auth (adjust values as needed)
    # security_protocol => "SASL_PLAINTEXT"
    # sasl_mechanism => "PLAIN"
    # sasl_username => "kafka-user"
    # sasl_password => "kafka-pass"
  }
}

output {
  elasticsearch {
    hosts => ['elasticsearch']
    user => 'elastic'
    password => 'changeme'
  }
  # Print events to console for debugging
  stdout { codec => rubydebug }
}

Start with the network fix first—that's 90% of Docker-Kafka connection issues. If it still fails, the logs will point you to the next step!

内容的提问来源于stack exchange，提问作者Matt