Kafka Broker宕机时生产者丢失消息问题技术问询

Hey there, let's break down what's going on with your Kafka producer setup and fix that message loss problem you're seeing.

Why no exceptions when the Broker goes down? #

Kafka's producer is built to be resilient by default, which is why you don't see immediate errors when your Broker goes offline:

• It first shoves messages into a local buffer (controlled by buffer.memory) and uses a background thread to handle sending them out.
• The default acks setting is 1—meaning it only waits for the Leader Broker to acknowledge receipt. When the Broker drops, the producer's retry mechanism kicks in: by default, retries is set to 2147483647 (basically infinite retries), so it'll keep trying to send without throwing an error right away.
• There's also a delivery.timeout.ms default of 5 minutes. During this window, the producer will keep retrying before it gives up on those messages.

Why are messages lost after restarting the Broker? #

The message loss boils down to how your producer is configured and the lack of redundancy in your single-node setup:

• While the Broker is down, messages pile up in the producer's local buffer. If that buffer fills up, or if the delivery.timeout.ms window expires, the producer will drop those messages by default (since enable.idempotence is off by default—no guarantee of exactly-once delivery, and no transaction safeguards).
• With a single Broker, there are no replica nodes to take over when it goes down. Any messages sent during the outage never get persisted to disk, so once the Broker comes back up, those in-flight messages are gone for good.

How to fix the message loss? #

Tweak these producer configurations to harden your setup against message loss:

• Enable idempotence: Set enable.idempotence=true. This automatically configures acks=all (waits for all in-sync replicas to confirm), sets a reasonable retry count, and limits in-flight requests to prevent duplicates. It guarantees exactly-once delivery for each message.
• Force acks=all: If you don't want to enable idempotence for some reason, manually set acks=all to ensure messages are persisted to disk (and all replicas, if you have them) before the producer gets an acknowledgment.
• Adjust retry and timeouts:
  • Bump up delivery.timeout.ms if you want the producer to keep retrying longer while waiting for the Broker to come back online.
  • Set linger.ms to a small value (like 5ms) to batch messages slightly—this improves efficiency without killing real-time performance.
• Add send callbacks: Attach a callback to your producer sends to track success/failure. This lets you log errors or implement custom retry logic if messages fail to send. Here's a quick code snippet:

  producer.send(new ProducerRecord<>("test", "message-key", "message-value"), (metadata, exception) -> {
      if (exception != null) {
          // Handle failure—log it, queue for retries, etc.
          System.err.println("Message failed to send: " + exception.getMessage());
      } else {
          // Confirm success
          System.out.printf("Message sent successfully to partition %d, offset %d%n", 
              metadata.partition(), metadata.offset());
      }
  });
  

• Use transactions (for strict guarantees): If you need rock-solid exactly-once semantics (e.g., financial systems), enable producer transactions by setting a transactional.id. This ensures messages are either all committed or rolled back, no partial losses.

Bonus tips #

• A single Broker is great for testing, but never use it in production. Spin up a 3-node cluster with replication enabled—this way, if one Broker goes down, replicas can still accept messages and prevent loss.
• Use monitoring tools like Prometheus + Grafana or Kafka's built-in JMX metrics to keep an eye on producer buffer usage, send success rates, and retry counts. This helps you catch issues before they lead to loss.

内容的提问来源于stack exchange，提问作者andrii