Kafka Producer：发送消息后断开与保持连接的选型疑问

Great question—this is a common point of confusion when working with Kafka producers in persistent services like web applications. Let’s break down the tradeoffs, validate your intuition, and cover some key details you might have missed.

Short Answer #

For nearly all web app scenarios, you should maintain a single (or small pool of) Kafka Producer instance(s) for the entire lifecycle of your application, rather than disconnecting after every message send. Your intuition about message frequency influencing this is partially correct, but the scale of tradeoffs makes keeping connections the default best practice.

Why Keeping Connections Is Better (Most of the Time) #

Let’s unpack the real-world impact of each approach:

• Connection initialization overhead is non-trivial: Creating a Kafka Producer isn’t just opening a TCP connection. It involves broker handshake, fetching cluster metadata (broker lists, topic partitions, leader information), and setting up internal buffers and retry mechanisms. Even for low-frequency sends, repeating this process every time adds unnecessary latency and load to both your app and the Kafka cluster.

• Producers are designed for long-term use: Kafka’s Producer client is thread-safe, optimized for reuse, and includes built-in features like batch messaging, connection pooling, and automatic reconnection. These optimizations only work when you keep the instance alive—disconnecting after each send negates all of them. For example, batch messaging (enabled by default via linger.ms and batch.size) reduces the number of network roundtrips by grouping messages, which only works if the producer stays active between sends.

• TCP connection resource usage is minimal: A single idle Kafka producer connection uses negligible system resources (a few KB of memory, minimal CPU overhead). Modern servers can handle hundreds of such connections without breaking a sweat—this is a far smaller cost than the overhead of repeated reconnections.

Your "Frequency-Based" Intuition: Valid But Nuanced #

You’re right that message frequency plays a role, but the threshold for justifying disconnects is much higher than you might think:

• High-frequency sends: Definitely keep the connection alive—reconnecting here would cripple performance with constant setup/teardown overhead.
• Low-frequency sends: Even here, keeping the connection is still better for most cases. The Kafka client has a built-in connection.max.idle.ms configuration (default 9 minutes) that automatically closes idle connections after the specified timeout. This lets you reap the benefits of reuse when messages do come in, while automatically freeing up resources during long idle periods.

Key Details You Might Have Missed #

• Singleton pattern is critical: In web apps, never create a new Producer instance per request. Instead, initialize a single global instance (or use a dependency injection container to manage it) and reuse it across all requests. This avoids resource leaks and ensures you leverage the client’s optimizations.
• Automatic reconnection handles failures: The Producer client automatically detects broken connections and retries reconnection without manual intervention. If you disconnect manually, you’ll have to implement your own retry logic for connection failures, adding unnecessary complexity.
• Extreme edge cases where disconnects make sense: The only scenario where manually disconnecting after sends might be justified is if your app sends messages extremely infrequently (e.g., once every few hours/days) and runs on severely resource-constrained hardware. Even then, the built-in idle timeout usually handles this better than manual teardown.

Final Recommendation #

Stick with maintaining a long-lived Producer instance in your web app. Use the built-in idle connection timeout to manage resource usage during quiet periods, and rely on the client’s built-in optimizations for performance. Only consider manual disconnects if you’re in an extremely niche resource-constrained scenario.

内容的提问来源于stack exchange，提问作者Honinbo Shusaku