如何在Kafka中设置acknowledgement为false以避免消息消费后从主题删除？

Hey there! Let's break down your two questions step by step, since there's a common misunderstanding here about how Kafka handles message retention and acknowledgments.

1. How to set acknowledgement to false in Kafka #

First, it's important to clarify: "acknowledgement" can refer to two different settings depending on whether you're working with producers or consumers. Let's cover both scenarios:

For Producers #

If you’re referring to the producer’s acks configuration (which controls how many broker replicas confirm receipt before the producer considers a message successful), acks=false isn’t a valid option. The valid values are:

• acks=0: Producer sends messages without waiting for any broker acknowledgment at all.
• acks=1: Producer waits for only the leader broker to confirm receipt.
• acks=all (or -1): Producer waits for all in-sync replicas to confirm receipt.

For Consumers #

If you meant disabling automatic offset acknowledgment (so you control exactly when offsets are marked as "processed"), this is done by setting enable.auto.commit=false in your consumer configuration. Here’s how to implement this in common client libraries:

Java Client

Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "your-broker:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "your-consumer-group");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); // Disable auto-ack
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
// Add key/value deserializers
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

Python (confluent-kafka)

from confluent_kafka import Consumer

conf = {
    'bootstrap.servers': 'your-broker:9092',
    'group.id': 'your-consumer-group',
    'enable.auto.commit': False,  # Disable auto-commit
    'auto.offset.reset': 'earliest'
}

consumer = Consumer(conf)

2. Keeping messages in the topic after consumption (and how it relates to ack settings) #

First, a critical clarification: Kafka does NOT delete messages immediately after they’re consumed. Message deletion is controlled by the topic’s retention policies, not by whether the consumer acknowledged the message. Here’s how to ensure messages stay available long-term, plus how ack settings tie into re-consuming messages:

Configure Topic Retention Policies #

You can set these when creating a topic, or modify existing topics to adjust retention:

Create a topic with custom retention

Use Kafka’s command-line tool:

kafka-topics.sh --create \
  --bootstrap-server your-broker:9092 \
  --topic your-topic \
  --partitions 3 \
  --replication-factor 1 \
  --config retention.ms=864000000  # Retain messages for 10 days (in ms)
  --config retention.bytes=-1  # Disable size-based deletion (unlimited storage)

Modify an existing topic’s retention

kafka-configs.sh --alter \
  --bootstrap-server your-broker:9092 \
  --topic your-topic \
  --add-config retention.ms=864000000,retention.bytes=-1

How consumer ack settings enable re-consumption #

Disabling auto-commit (enable.auto.commit=false) lets you control when the consumer "marks" messages as processed. If you never commit offsets, the consumer will restart from the earliest available offset (or last committed offset) every time it launches, allowing you to re-read messages. You can also manually seek to specific offsets for targeted re-consumption:

Java example of manual offset seek

// Seek to the start of the topic
consumer.subscribe(Collections.singletonList("your-topic"));
consumer.seekToBeginning(consumer.assignment());

// Or seek to a specific offset for a single partition
TopicPartition partition = new TopicPartition("your-topic", 0);
consumer.seek(partition, 0); // Start from offset 0

Python example of manual offset seek

consumer.subscribe(['your-topic'])
// Seek to the start of all assigned partitions
for partition in consumer.assignment():
    consumer.seek(partition, 0)

Key Takeaway #

• Disabling auto-commit (enable.auto.commit=false) gives you control over when offsets are committed, but it doesn’t affect whether messages stay in the topic.
• To keep messages in the topic long-term, configure the topic’s retention.ms and retention.bytes settings. Kafka only deletes messages when these thresholds are exceeded, regardless of whether they’ve been consumed.

内容的提问来源于stack exchange，提问作者neb