Docker新手求助：搭建Kafka及消费者容器并实现本地连接与MongoDB存储

Hey there! As a Docker/Kafka newbie, you're already on the right track with your docker-compose setup—let's polish it up to make everything work smoothly, including getting your local machine connected to the Kafka container.

1. Fixed Docker Compose Configuration #

First, your original config was missing the MongoDB service and had a truncated Kafka environment variable. Here's a complete, working version tailored to your needs:

version: '3.8' # Using a newer, more compatible version
services:
  zookeeper:
    image: wurstmeister/zookeeper:latest
    ports:
      - "2181:2181"
    healthcheck:
      test: ["CMD", "zkServer.sh", "status"]
      interval: 10s
      timeout: 5s
      retries: 3

  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - "9092:9092" # For container-internal connections
      - "9093:9093" # For local machine connections
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "test:1:1" # Creates 'test' topic with 1 partition, 1 replica
      # Dual listeners to support both Docker network and local host
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_HOST://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    depends_on:
      zookeeper:
        condition: service_healthy

  mongodb:
    image: mongo:latest
    ports:
      - "27017:27017"
    volumes:
      - mongodb_data:/data/db # Persist MongoDB data across restarts
    environment:
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: secret

  consumer:
    image: springio/kafka-consumer # Assuming your Spring consumer image is built correctly
    depends_on:
      - kafka
      - mongodb
    environment:
      # Pass connection details to your Spring app
      SPRING_KAFKA_BOOTSTRAP_SERVERS: kafka:9092
      SPRING_DATA_MONGODB_URI: mongodb://admin:secret@mongodb:27017/kafka_messages?authSource=admin
      SPRING_KAFKA_CONSUMER_GROUP_ID: test-consumer-group
      SPRING_KAFKA_CONSUMER_AUTO_OFFSET_RESET: earliest

volumes:
  mongodb_data: # Named volume for persistent MongoDB storage

2. Key Config Breakdown #

• Zookeeper: Kafka relies on it to manage cluster state and topic metadata. The healthcheck ensures Kafka only starts once Zookeeper is fully ready.
• Kafka Listeners:
  • PLAINTEXT://kafka:9092: Lets your consumer container connect to Kafka using the Docker service name kafka (works within the Docker network).
  • PLAINTEXT_HOST://localhost:9093: Lets your local machine connect to Kafka via localhost:9093.
• MongoDB: Added a persistent volume to avoid data loss, plus root credentials (adjust these to your security needs).
• Consumer: Added environment variables to tell your Spring app how to connect to Kafka and MongoDB—make sure your consumer code uses these properties to listen to the test topic and write to the correct collection!

3. Connecting to Kafka from Your Local Machine #

To interact with Kafka from your local host (e.g., using a Kafka CLI or your own test app):

• Use the bootstrap server address localhost:9093
• For example, if you have the Kafka CLI tools installed locally, send a test message with:

  kafka-console-producer.sh --broker-list localhost:9093 --topic test
  

• Your Dockerized consumer will use kafka:9092 to connect, which works because Docker automatically resolves service names within its network.

4. Test the Stack Step-by-Step #

1. Start all services in detached mode:

   docker-compose up -d
   

2. Verify all containers are running:

   docker-compose ps
   

3. Send a test message using the local producer command above.
4. Check if the message was saved to MongoDB:

   docker exec -it <mongodb-container-name> mongosh -u admin -p secret
   # Then run these commands in the MongoDB shell:
   use kafka_messages
   db.messages.find() # Replace 'messages' with your consumer's target collection name
   

If your consumer isn't behaving as expected, check its logs with docker-compose logs consumer to debug connection or code issues!

内容的提问来源于stack exchange，提问作者user1028741