Apache Kafka疑问：生产者连Broker，消费者为何连ZooKeeper？新旧版本解析

Great question—this gets right to a key architectural evolution in Kafka that a lot of folks get confused about. Let's break this down clearly:

First, the old Kafka setup (pre-0.9) #

Producers never needed to touch ZooKeeper, and here's why: Their only job is to send messages to Kafka brokers. Brokers manage all topic metadata, partition assignments, and message storage, so producers just need a broker endpoint to push data to. No coordination or state storage needed from ZooKeeper here.

Old-style consumers (the "high-level consumer" API) did need ZooKeeper for two critical jobs:

• Offset storage: Exactly as you heard—consumer offsets (the last message a consumer processed) were stored in ZooKeeper as ZNodes. Every time a consumer finished a batch, it wrote its latest offset to ZooKeeper; when it restarted, it read that offset to pick up where it left off.
• Consumer group coordination: ZooKeeper tracked which consumers were in a group, assigned partitions to them, and handled rebalancing when consumers joined or left the group.

Why the shift to consumers connecting directly to brokers (Kafka 0.9+) #

Your hunch about offset storage is spot-on—it's the core reason for this change, but it's tied to a bigger architectural overhaul:

1. Offset storage moved to Kafka itself: Kafka created a dedicated internal compacted topic called __consumer_offsets to store all consumer group offsets. ZooKeeper was never designed for high-frequency write loads (offset updates happen constantly for active consumers), so shifting this to Kafka eliminated a major bottleneck.
2. Group coordination shifted to brokers: Brokers now run a "group coordinator" process that handles all consumer group tasks—membership tracking, partition assignment, rebalancing. Consumers communicate directly with this coordinator via broker endpoints, so ZooKeeper is no longer in the consumer's critical path.

Bonus: Why this is a better setup #

• Scalability: Kafka handles high-throughput, high-frequency offset updates far better than ZooKeeper ever could.
• Reliability: Reducing dependency on ZooKeeper lowers the risk of consumer failures tied to ZooKeeper outages or performance hits.
• Simpler ops: Consumers only need broker endpoints now—no need to expose ZooKeeper to client networks, making deployments cleaner.

Note: The old ZooKeeper-based consumer API is deprecated now, so you should always use the modern consumer that connects directly to brokers for new projects.

内容的提问来源于stack exchange，提问作者desheng