如何配置Kafka实现At-Least-Once投递？默认策略为At-Most-Once

Alright, let's walk through exactly how to set up Kafka to guarantee At-Least-Once delivery—you’re correct that the default setup leans toward At-Most-Once, so we need to adjust key settings on both the producer and consumer sides to shift this behavior.

Producer-Side Configuration #

These settings ensure your producer doesn’t consider a message "sent" until it’s safely persisted in the Kafka cluster, and handles transient failures without dropping messages:

• acks=all (or -1): This is the most critical setting. Instead of only waiting for the leader partition to acknowledge receipt (the default acks=1), the producer will wait for all in-sync replicas (ISR) to confirm they’ve stored the message. This prevents data loss if the leader fails before replicating the message to followers.
• retries=N (where N > 0): Set this to a positive number (e.g., retries=5) to enable automatic retries for transient errors like network blips or replica sync delays. Kafka will retry sending the message until it succeeds or hits the retry limit.
• retry.backoff.ms=1000 (adjust as needed): This controls the delay between retries to avoid overwhelming the cluster. A 1-second backoff is a safe starting point.
• enable.idempotence=true (optional but recommended): While At-Least-Once allows for duplicate messages (since retries can resend), enabling idempotence ensures the producer doesn’t create duplicates when retrying. Kafka uses a producer ID and sequence numbers to track sent messages, so the cluster will ignore duplicate sends automatically.
• max.in.flight.requests.per.connection=1 (or ≤5 if using idempotence): If you need strict message ordering, set this to 1. It limits the number of unacknowledged requests the producer can send at once, preventing retry messages from overtaking newer messages. If idempotence is enabled, the default is 5 which maintains ordering while keeping better throughput.

Consumer-Side Configuration #

The consumer side is where we ensure we only mark messages as processed after we’ve successfully handled them:

• enable.auto.commit=false: Disable automatic offset commits. The default enable.auto.commit=true commits offsets at regular intervals, which can lead to At-Most-Once behavior if the consumer crashes after committing but before processing the message.
• Manual offset commits: After successfully processing a batch of messages, explicitly commit the offset using:
  • commitSync(): Synchronously commits offsets, blocking until the commit is confirmed by the cluster. Use this for critical workflows where you need to ensure offsets are committed before moving on.
  • commitAsync(): Asynchronously commits offsets for better performance. Note that if the commit fails, it won’t retry automatically—you’ll need to handle failures in the callback if consistency is critical.
• auto.offset.reset=earliest (context-dependent): If your consumer starts with no committed offsets (e.g., first run or after a long outage), this setting tells it to start consuming from the beginning of the topic instead of skipping to the latest messages. Adjust this based on whether you can tolerate missing historical data.
• isolation.level=read_committed (optional, for transactional producers): If you’re using Kafka transactions to send messages, set this to ensure the consumer only reads messages that have been successfully committed. This avoids processing partial or rolled-back transactions.

Key Takeaway #

At-Least-Once delivery boils down to two core principles:

1. Producers don’t give up on sending a message until the cluster confirms it’s stored safely.
2. Consumers only commit offsets after they’ve finished processing the corresponding messages.

内容的提问来源于stack exchange，提问作者Amirrr