Kafka 2.5日志段删除行为与《Kafka权威指南》旧版描述不符的疑问

Great question! You're absolutely right—Kafka did modify its log segment deletion mechanism after the 0.9.0.1 version referenced in Kafka: The Definitive Guide 1st edition. Let's break down what changed, why your 2.5 cluster behaves differently, and why you might not have spotted the change in docs at first:

Pre-0.10.x (Including 0.9.0.1) Behavior #

As the book describes:

• A log segment couldn't be deleted until it was closed
• Without log.segment.ms configured, segments only closed when they hit the log.segment.bytes size limit
• Retention policies like log.retention.ms only applied to closed segments

Post-0.10.x (Including Kafka 2.5) Behavior #

The Kafka team updated this logic to fix a critical pain point: low-throughput topics where segments never reached log.segment.bytes would retain old data forever. The key changes are:

• Segments no longer need to be closed to be eligible for deletion
• For active (unclosed) segments, Kafka checks the largest timestamp in the segment (largestTime in your logs) against the log.retention.ms threshold
• If the oldest data in the segment exceeds the retention window, it gets marked for deletion—even if the segment size is still under log.segment.bytes

What Your Logs Show #

Looking at your cluster's log entry:

[2020-12-24 15:51:17,808] INFO [Log partition=Topic-2, dir=/Folder/Kafka_data/kafka] Scheduling segments for deletion List(LogSegment(baseOffset=165828, size=895454171 , lastModifiedTime=1608220234000, largestTime=1608220234478)) (kafka.log.Log)

This segment is ~854MB (well under the default 1GB log.segment.bytes), but its largestTime timestamp is older than your 7-day (604800000ms) retention window. That's exactly why it's being deleted—this follows the newer, more flexible deletion logic.

Why the Change Isn't Obvious in Docs #

This adjustment was part of incremental improvements to log retention, not a flashy feature launch, so it might not be called out prominently in high-level docs. It was primarily driven by fixing edge cases for low-traffic topics, and it's now standard behavior in all modern Kafka versions.

内容的提问来源于stack exchange，提问作者Mayank Madhav