如何在Minikube部署Kafka集群并使用.NET客户端收发消息

Hey there! I get that setting up Kafka on Minikube with .NET can feel overwhelming when you're just starting out—let's skip the complicated stuff and get a working POC up and running in no time. Here's a straightforward, step-by-step plan tailored for beginners like you:

We'll use Bitnami's pre-configured Kafka/ZooKeeper images to avoid manual setup headaches. Create a file named kafka-minikube.yaml with this content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zookeeper
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
      - name: zookeeper
        image: bitnami/zookeeper:latest
        ports:
        - containerPort: 2181
        env:
        - name: ALLOW_ANONYMOUS_LOGIN
          value: "yes"
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper
spec:
  selector:
    app: zookeeper
  ports:
  - port: 2181
    targetPort: 2181
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
      - name: kafka
        image: bitnami/kafka:latest
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_BROKER_ID
          value: "1"
        - name: KAFKA_LISTENERS
          value: PLAINTEXT://0.0.0.0:9092
        - name: KAFKA_ADVERTISED_LISTENERS
          value: PLAINTEXT://kafka:9092
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zookeeper:2181
        - name: ALLOW_PLAINTEXT_LISTENER
          value: "yes"
---
apiVersion: v1
kind: Service
metadata:
  name: kafka
spec:
  selector:
    app: kafka
  ports:
  - port: 9092
    targetPort: 9092

Now deploy it to Minikube:

kubectl apply -f kafka-minikube.yaml

Wait for the pods to start running (check with kubectl get pods—you should see both zookeeper and kafka pods in "Running" state). Then forward the Kafka port to your local machine so your .NET app can reach it:

kubectl port-forward service/kafka 9092:9092

Keep this terminal window open—closing it will stop the port forwarding.

We'll use the official Confluent.Kafka NuGet package (the most widely used .NET Kafka client) for both producer and consumer.

Producer App #

1. Create a new console project:

dotnet new console -n KafkaDotNetProducer

2. Add the Confluent.Kafka package:

cd KafkaDotNetProducer
dotnet add package Confluent.Kafka

3. Replace the contents of Program.cs with this simple producer code:

using Confluent.Kafka;

var producerConfig = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    ClientId = "dotnet-test-producer"
};

using var producer = new ProducerBuilder<Null, string>(producerConfig).Build();

try
{
    Console.WriteLine("Sending test messages to Kafka...");
    for (int i = 1; i <= 5; i++)
    {
        var message = new Message<Null, string> { Value = $"Test message #{i} from .NET!" };
        var deliveryResult = await producer.ProduceAsync("test-topic", message);
        Console.WriteLine($"Delivered message to {deliveryResult.TopicPartitionOffset}");
    }
}
catch (ProduceException<Null, string> ex)
{
    Console.WriteLine($"Delivery failed: {ex.Error.Reason}");
}

Consumer App #

1. Create another console project:

dotnet new console -n KafkaDotNetConsumer

2. Add the Confluent.Kafka package:

cd KafkaDotNetConsumer
dotnet add package Confluent.Kafka

3. Replace Program.cs with this consumer code:

using Confluent.Kafka;

var consumerConfig = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "dotnet-test-consumer-group",
    AutoOffsetReset = AutoOffsetReset.Earliest // Start reading from the beginning of the topic
};

using var consumer = new ConsumerBuilder<Ignore, string>(consumerConfig).Build();
consumer.Subscribe("test-topic");

Console.WriteLine("Waiting for messages from Kafka...");
try
{
    while (true)
    {
        var consumeResult = consumer.Consume();
        Console.WriteLine($"Received message: {consumeResult.Message.Value}");
    }
}
catch (ConsumeException ex)
{
    Console.WriteLine($"Consumption error: {ex.Error.Reason}");
}
finally
{
    consumer.Close();
}

1. First, start the consumer:
   In the KafkaDotNetConsumer directory, run dotnet run—it will wait for messages.
2. Then, start the producer:
   Open a new terminal, navigate to KafkaDotNetProducer directory, run dotnet run.
3. You'll see the producer send messages, and the consumer instantly receive and print them. That's it—you've got a working Kafka <-> .NET messaging flow!

• This setup uses temporary storage for Kafka (good for POCs; add PersistentVolumeClaims later if you need data to survive pod restarts).
• The port forwarding step is crucial because Minikube runs in a separate VM—this lets your local .NET apps connect directly to the Kafka broker in the cluster.
• Confluent.Kafka has great docs, but this code sticks to the absolute basics to avoid overwhelming you.

内容的提问来源于stack exchange，提问作者Rudresh Bhatt