Kafka报错：客户端无可用代理连接，求协助缩容问题排查

Hey Ryan, let's walk through troubleshooting this Kafka connection error you're hitting after scaling down your cluster from 3 to 1 broker. That "Kafka: client has run out of available brokers to talk to" error usually points to a disconnect between your clients (Filebeat/Logstash) and the remaining broker, or issues with the broker itself. Here's how to dig into it step by step:

1. First, verify the remaining Kafka broker is healthy and accessible #

Before checking clients, make sure your single broker is actually up and reachable:

• Check if the Kafka service is running:

  systemctl status kafka
  

  If it's not running, check the broker's log file (typically /var/log/kafka/server.log) for startup errors—common issues include misconfigured listeners or ZooKeeper connection problems.
• Validate the broker's listener configuration:
  Open server.properties and confirm listeners and advertised.listeners are set to an IP/hostname that Filebeat/Logstash servers can reach. Avoid using localhost here, as it won't work for remote clients. Example:

  listeners=PLAINTEXT://10.0.0.5:9092
  advertised.listeners=PLAINTEXT://10.0.0.5:9092
  

• Test network connectivity from a client server:
  Use telnet or nc to confirm the broker's port is open:

  nc -zv 10.0.0.5 9092
  

  If this fails, check firewalls, security groups, or network ACLs blocking traffic to port 9092.

2. Ensure all clients are using the updated broker list #

It's easy to miss a client configuration when scaling down—double-check Filebeat and Logstash:

• Filebeat configuration:
  Open your filebeat.yml (or relevant output config file) and update the output.kafka.hosts field to only include your remaining broker:

  output.kafka:
    hosts: ["10.0.0.5:9092"]
    # ... other configs
  

  Restart Filebeat after making changes:

  systemctl restart filebeat
  

  Check Filebeat's logs (/var/log/filebeat/filebeat.log) for any lingering errors about connecting to old broker IPs.
• Logstash configuration:
  Find your Kafka input config (usually in /etc/logstash/conf.d/) and update bootstrap_servers to the single broker:

  input {
    kafka {
      bootstrap_servers => "10.0.0.5:9092"
      topic_id => "your-rails-logs-topic"
      # ... other configs
    }
  }
  

  Restart Logstash and check its logs (/var/log/logstash/logstash-plain.log) for connection issues.

3. Clean up stale cluster metadata (if using ZooKeeper) #

If your Kafka cluster uses ZooKeeper (common in older versions), it might still have references to the removed brokers:

• Connect to your ZooKeeper instance:

  zkCli.sh -server <zk-ip>:2181
  

• List registered broker IDs:

  ls /brokers/ids
  

  You should only see the ID of your remaining broker. If old IDs are present, delete them:

  delete /brokers/ids/<old-broker-id>
  

• Exit ZooKeeper and restart your Kafka broker to pick up the updated metadata.

4. Validate end-to-end message flow #

Once you've checked the above, test if messages can actually flow through the system:

• Send a test message to your target topic:

  kafka-console-producer.sh --broker-list 10.0.0.5:9092 --topic your-rails-logs-topic
  # Type a test message and press Enter
  

• Consume the message to confirm it's stored:

  kafka-console-consumer.sh --bootstrap-server 10.0.0.5:9092 --topic your-rails-logs-topic --from-beginning
  

  If this works, Kafka is functioning correctly—any remaining issues are likely with Filebeat/Logstash configuration.

内容的提问来源于stack exchange，提问作者Ryan Grush