Apache Bookkeeper账本日志删除及Pulsar过期消息清理机制咨询

Great questions! Let's break them down clearly:

1. How to delete log entries from an Apache Bookkeeper ledger? #

The straight answer is: you can't delete individual log entries from an existing Bookkeeper ledger.

Bookkeeper is built on an append-only, immutable ledger model—this design choice ensures data consistency, durability, and simplifies replication across bookies. There's no native API to remove or modify specific entries once they're written.

If you need to "remove" certain entries, the common workaround is:

• Create a new Bookkeeper ledger
• Copy only the entries you want to retain into this new ledger
• Update your application to use the new ledger, and let the old one be garbage-collected once it's no longer referenced

2. How does Pulsar delete expired messages from Bookies when ledgers are immutable? #

Pulsar works around Bookkeeper's immutability with a smart combination of ledger management, cursor tracking, and garbage collection. Here's how it works:

• Ledger Rotation: Pulsar doesn't store all messages for a topic in one single ledger. It creates new ledgers automatically (when the current ledger reaches a size limit, or on a fixed schedule). This means expired messages are grouped in older, discrete ledgers rather than spread across a single large one.
• Cursor & Expiry Tracking: Each Pulsar subscription has a "cursor" that tracks the last message a consumer has acknowledged. Pulsar also respects message TTLs (Time-To-Live). When all subscriptions have consumed past a ledger, or every message in a ledger has exceeded its TTL, Pulsar marks that entire ledger as no longer needed.
• Bookkeeper Garbage Collection: Pulsar sends a signal to Bookkeeper indicating that a ledger is eligible for deletion. Bookkeeper's GC process then removes the entire ledger from the bookies. Since we're deleting full ledgers (not individual entries), this fits perfectly with Bookkeeper's immutable design.
• Tiered Storage (Optional): For large-scale deployments, Pulsar can first migrate older, infrequently accessed messages to cheaper tiered storage (like object storage). Once the migration is complete, the corresponding Bookkeeper ledger can be safely deleted, freeing up bookie disk space while preserving historical data if required.

内容的提问来源于stack exchange，提问作者Paul