Kafka Streams（2.7版本）精准一次语义相关咨询：exactly_once与exactly_once_beta的差异及测试方法

Hey there! Since you're working with Kafka 2.7 and have enabled exactly-once semantics in your Streams DSL app, let's break down your questions clearly and practically.

1. What's the difference between exactly_once and exactly_once_beta? #

The core distinction lies in maturity, feature support, and reliability:

• exactly_once_beta: This was the experimental, early-stage version of exactly-once semantics introduced in Kafka 0.11. It had critical limitations—for example, it didn't support all state store types, window operations had edge cases, and it wasn't fully robust across all failure scenarios. By Kafka 2.5, this beta flag was deprecated as the stable implementation became ready.
• exactly_once: This is the production-ready, stable version rolled out in Kafka 2.5. It fixes all gaps of the beta release, supports every standard Streams feature (including joins, windows, and all state store types), and leverages Kafka's transactional API to ensure atomicity of offset commits, processing, and output writes. For your Kafka 2.7 deployment, this is the only flag you should use—exactly_once_beta is effectively obsolete at this point.

2. How to test exactly-once semantics to ensure messages are processed only once? #

Testing exactly-once requires simulating real-world failures and verifying no duplicate processing occurs. Here are actionable steps:

• Simulate abrupt app crashes: Run your Streams app, send a test message to the input topic, then force-kill the app process mid-processing. Restart the app and check:
  • The output topic for duplicate records (your business logic should produce a unique output per input message).
  • Any state stores (e.g., key-value stores) to confirm state was updated only once—for example, a counter tied to the message ID should be incremented by 1, not 2.
• Trigger broker failures: If you have a multi-broker cluster, take down a broker while the app is processing messages. Once the broker recovers, confirm no duplicate processing happened.
• Add idempotent markers: Include a unique identifier (like the input message's key + a processing timestamp) in your output records. If exactly-once works, you'll never see duplicate markers in the output topic.
• Test producer retries: Configure your input producer to retry sending the same message (set retries > 0 and acks=all). Even with retries, your Streams app should process the message only once.

How to verify exactly-once semantics is working correctly in your app? #

Beyond testing, you can validate it's active and functioning as expected:

• Check startup logs: Look for log lines like Enabling exactly-once semantics using transactions in your Streams app logs. This confirms the app correctly picked up the processing.guarantee configuration.
• Inspect consumer group offsets: Use the kafka-consumer-groups.sh tool to check if your app's consumer group is committing offsets transactionally. Look for a transactional.id associated with the group—this is a telltale sign exactly-once is active.
• Monitor metrics: Kafka Streams exposes transaction-related metrics to track health:
  • transaction-success-rate: Should stay at 100% if transactions are completing successfully.
  • aborted-transactions-total: Should remain low (only increase on legitimate failures, not routine processing).
• Validate state consistency: Use the Kafka Streams interactive queries API to fetch state from your app's stores. For example, if you're counting messages, the count should match the number of unique input messages, even after multiple restarts or failures.

内容的提问来源于stack exchange，提问作者samik