Kafka kept its metadata in ZooKeeper, so every controller failover meant re-reading the whole cluster's metadata before it could act. Confluent moved that metadata into a Kafka log and replaced the single controller with a quorum.

The Kafka controller design dates to 2012, and it barely changed for years. Each cluster has a single node acting as controller, elected by ZooKeeper watchers. It does everything a normal broker does, and on top of that it maintains the cluster’s metadata: broker IDs and racks, topic and partition information, leaders and in-sync replica lists, cluster-wide and per-topic configs, security credentials.

All of that persists in ZooKeeper as the source of truth, which is why most of ZooKeeper’s read and write traffic comes from the controller. Other brokers write directly too — a leader updating its ISR list, for instance. So the controller registers watchers for metadata changes, and when a watcher fires, a single-threaded loop handles it and propagates the new metadata out to every other broker.

Walk through a controlled shutdown. A broker that wants to stop asks the controller. The controller works out which partitions that broker hosts, picks new leaders for them, writes the updated ISR information to ZooKeeper, then propagates the new metadata to the remaining brokers — one at a time — via UpdateMetadata and LeaderAndISR requests. In the textbook example that is one partition. In practice a broker can host thousands, and the controller has to write to ZooKeeper for each one. Confluent’s estimate: this could take seconds or even more. Meanwhile clients hunting for the new leader by randomly consulting brokers may hit one that has not gotten the news yet, and their request times out.

Now take the ugly case: the controller crashes. The watcher fires, every broker races to register with ZooKeeper, and the winner becomes the new controller. Its first job is to fetch all the metadata back out of ZooKeeper — every topic partition across every path. That fetch is linear in the total number of partitions in the cluster, and until the bootstrap finishes the new controller cannot handle a single admin request. That is a long unavailability window, and it gets longer exactly as the cluster gets more valuable.

The recurring shape is the giveaway. Metadata change propagation between controller and brokers is linear in the number of partitions involved. Persisting that metadata to ZooKeeper is linear in the amount of metadata. Recovering a controller is linear in the size of the cluster. None of these are bugs — they are the architecture, working as designed, on a cluster bigger than the design anticipated.

Then came the reframing. At first glance Kafka was storing a snapshot of metadata across a set of ZooKeeper paths. But with all those watchers and path versions, what it was really tracking was a sequence of metadata change events — a metadata log. Behind ZooKeeper’s own APIs, the written data is maintained as a transaction log anyway. So why keep the log hidden inside ZooKeeper instead of storing it in Kafka, where it can be read directly? Logs are the one thing Kafka is really good at.

Make the metadata another internal Kafka topic and several problems dissolve at once. Chained metadata operations get ordered naturally by log offset and can be batched with asynchronous log I/O. Propagation stops being RPCs and becomes brokers replicating a changelog — so divergence stops being a category of bug, because every broker’s local view derives from the same log and is versioned by the offset it has reached. The control plane separates cleanly from the data path, with its own ports, queues, metrics, and threads. And a small group of brokers can synchronously replicate that log as a quorum of controllers rather than one lonely elected node.

KRaft’s one real deviation from the Raft paper is direction: it pulls rather than pushes. Voters fetch from the leader with an epoch and an offset, exactly like Kafka replicas already do, and that fetch doubles as the heartbeat that proves the leader is alive. A voter that diverged — say the old Epoch 2 leader holding records that never reached the majority — gets told the epoch only committed up to offset 6, truncates to there, and refetches. Reconciliation lands in one round trip instead of the ping-pong a pushing leader needs to find the right position.

Pull also defuses the disruptive-server problem. A voter removed from the quorum keeps fetching, so the leader can answer with an error code telling it to become an observer; in push-based Raft, the removed server just stops hearing from anyone and starts campaigning for an election it should not be in. There is a price — the new leader must call a separate begin-epoch API, and the leader must wait for the next fetch to advance its commit offsets — and Confluent calls those worthy tradeoffs. Reusing Kafka’s existing pull-based replication saved thousands of lines of code.