Let me break down what's going on here, because you're right to be confused—your initial understanding of batch.size and linger.ms makes sense on paper, but there's a key detail about how Kafka handles oversized messages that you're missing.

First, Clarifying the Core Logic #

Your assumption that either batch.size being hit or linger.ms elapsing triggers a send is mostly correct—but the critical fine print is that batch.size refers to the cumulative size of records in a single partition's batch, not the size of an individual record.

Let's rephrase the official docs to highlight this:

Once the total accumulated records for a partition reach batch.size, the batch sends immediately regardless of linger.ms. If the batch stays under that size, the producer waits up to linger.ms for more records to join it.

So here's the problem with your test: when you send a single record that's larger than batch.size, Kafka can't fit it into any existing batch (since it's already bigger than the batch limit). It creates a brand new batch just for that one record—but why isn't that batch sent right away?

Why Your Oversized Message Is Waiting 60 Seconds #

Under normal circumstances, Kafka should send an oversized record immediately (as an "overflow batch") without honoring linger.ms. But there are a few possible reasons you're seeing the 60-second delay:

• Your Configuration Might Not Be Applied Correctly: Double-check that your producer is actually loading the settings you think it is. Looking at your YAML, you have linger.ms: 10 listed, but you mentioned testing with linger.ms=60000—make sure the test config is overriding the base config properly. Also, confirm that batch.size is being mapped correctly (some frameworks can mess up nested YAML to Kafka's dot-separated keys like batch.size).
• You're Miscalculating the Record's Actual Size: Kafka counts the entire serialized size of the record—including the key, value, message headers, and protocol overhead—not just the raw value content. It's possible your "oversized" message is actually under 16KB when you include all these parts. Use the serializedSize() method on your ProducerRecord to get the exact byte count.
• Async Sending Is Tricking Your Observation: If you're using the asynchronous send() method without waiting on the returned Future, you might think the producer is delaying when it actually sent the record immediately. Try switching to a synchronous send with producer.send(record).get() to block until the send completes and see if the delay persists.
• Edge Cases in Older Kafka Versions: If you're using a very old Kafka client (pre-0.11.x), there were some edge cases where overflow batches could be delayed by linger.ms. Upgrading to a newer client (preferably 2.0+) would fix this if that's the issue.

The Correct Full Logic Recap #

To make sure we're on the same page, here's the complete flow for batch sending:

• For each partition, the producer maintains an in-progress batch.
• When a new record arrives:
  1. If adding it to the current batch pushes the total size over batch.size, the current batch sends immediately. The new record starts a new batch—if that single record is already over batch.size, this new batch sends right away.
  2. If the batch stays under batch.size, the producer waits up to linger.ms for more records.
• Any pending batches also send immediately when you call producer.flush() or close the producer.

Steps to Debug Your Test #

1. Verify record size: Use record.serializedSize() to confirm your message is truly over 16KB.
2. Inspect active configs: Print out all producer properties at initialization to ensure batch.size=16384 and linger.ms=60000 are active.
3. Test synchronous sends: Use send(...).get() to eliminate async timing confusion.
4. Enable debug logs: Turn on Kafka producer DEBUG logging—look for lines like Batch for partition [topic]-0 is full, sending or Waking up to send batch after linger time to see exactly when batches are being triggered.

内容的提问来源于stack exchange，提问作者hsingh