Golang连接Kafka容器时出现‘Cant start consumers error’错误的求助

Hey there, let's troubleshoot why your Go application can't connect to your Kafka container. From the docker-compose snippet you shared, I can spot a few common configuration gaps that are likely causing this issue.

First, let's look at your Kafka environment variables—there are missing pieces that prevent external clients (like your Go app running outside the container) from properly connecting:

1. Fix the Kafka Container Configuration #

Kafka uses advertised listeners to tell clients how to reach it, and your current setup is incomplete. Here's what you need to add/modify:

• You haven't defined the security protocol mapping for your listeners, which Kafka needs to know how to handle each listener type.
• Your KAFKA_ADVERTISED_LISTENERS cuts off mid-way, and it's missing the external listener entry that your Go app will use.
• You should also specify which listener Kafka uses for inter-broker communication (even for a single broker, this is good practice).

Here's the corrected Kafka service block for your docker-compose:

kafka:
  image: confluentinc/cp-kafka:latest
  container_name: kafka
  depends_on:
    - zookeeper
  ports:
    - "9092:9092"
  environment:
    KAFKA_BROKER_ID: 1
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_LISTENERS: INTERNAL://kafka:9092,EXTERNAL://0.0.0.0:9092
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
    KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL://localhost:9092
    KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

2. Adjust Your Go Code Connection String #

Since your Go app is running outside the Docker network, you need to connect to localhost:9092 (not the internal kafka:9092 hostname, which only works inside the Docker network). Here's a quick example using the popular Sarama library:

package main

import (
  "log"
  "github.com/Shopify/sarama"
)

func main() {
  // Configure the Sarama client to match your Kafka version
  config := sarama.NewConfig()
  config.Version = sarama.V2_8_0_0 // Use the version matching your cp-kafka image
  config.Consumer.Return.Errors = true

  // Create a consumer using localhost:9092
  consumer, err := sarama.NewConsumer([]string{"localhost:9092"}, config)
  if err != nil {
    log.Fatalf("Cant start consumers error: %v", err)
  }
  defer consumer.Close()

  // Rest of your consumer logic here...
}

3. Additional Checks to Rule Out Other Issues #

• Wait for Zookeeper and Kafka to fully start: Kafka takes a few seconds to initialize after Zookeeper is up. You can add healthchecks to your docker-compose to ensure services are ready before your app tries to connect.
• Verify port availability: Make sure no other process is using port 9092 on your host machine. You can check with lsof -i :9092 (Linux/macOS) or netstat -ano | findstr :9092 (Windows).
• Check client-broker version compatibility: If you're using an older or newer Go Kafka client, make sure it supports the version of Kafka running in your container. Specifying the version in the client config (like in the example above) helps avoid compatibility issues.

Once you update these settings, restart your Kafka container and try running your Go app again—this should resolve the "Cant start consumers error" you're seeing.

备注：内容来源于stack exchange，提问作者Arjun Malhotra