Kafka kept its metadata in a second distributed system, and that second system capped how many partitions Kafka could hold. KIP-500 moves the metadata into Kafka itself.

Apache Kafka stores its metadata — where partitions live, how topics are configured — outside of Kafka, in a separate Apache ZooKeeper cluster. In practice that means running at least three additional Java processes alongside the brokers, and often more. Confluent notes that Kafka clusters frequently end up with just as many ZooKeeper nodes as Kafka nodes.

The awkward part is what those two systems actually are. Kafka is a replicated distributed log with a pub/sub API on top. ZooKeeper is a replicated distributed log with a filesystem API on top. Two logs, two ways of doing network communication, security, monitoring, and configuration.

The duplication was annoying. The scalability limit was worse, and it hid inside a path that only runs at the worst possible moment.

Kafka elects a controller node to manage partition leaders and cluster metadata. When a cluster starts up, or when a new controller is elected after the old one goes away, that controller must load the full state of the cluster from ZooKeeper before it can proceed. Because of that, the more metadata a cluster accumulates, the longer every failover takes — and that directly limits the number of partitions Kafka can store.

The same shape shows up in ordinary work. Create or delete a topic today and the controller has to reload the entire list of topic names from ZooKeeper. Not because it wants to, but because ZooKeeper tells it that the set of topics changed without telling it which topics were added or removed.

Stated plainly: minimize the number of operations that take time proportional to the number of topics or partitions. No lengthy load when a new controller is elected. No second system to configure, secure, and monitor. And a public Kafka API for every operation that previously required talking to ZooKeeper directly — because several shipped admin tools still spoke to ZooKeeper, and one or two operations could not be done any other way.

Behind all of it sits the target: Confluent expects a single Kafka cluster to eventually support a million partitions or more. Metadata scalability is the thing standing between Kafka and that number.

KIP-500’s answer is to store Kafka’s metadata in Kafka. Metadata moves into a partition inside Kafka, and the controller becomes the leader of that partition. Metadata is treated as a log, which means brokers that need the latest updates read only the tail of it — the same way a consumer that only wants recent events reads the end of a topic rather than the whole thing. Brokers can also persist their metadata caches across restarts.

There is a bootstrapping problem in that design, and the KIP faces it directly: the controller depends on the metadata partition, so the metadata partition cannot depend on the controller for things like leader election. The nodes managing it run a self-managed Raft quorum instead. KIP-595 adapts Raft to feel native to Kafka — flipping Raft’s push-based replication to Kafka’s pull-based model, where followers connect to the leader rather than being pushed to, and renaming Raft’s “terms” to Kafka’s “epochs.”

Running without ZooKeeper — “KIP-500 mode” — starts out experimental. Most users stay on legacy mode with ZooKeeper, partly because KIP-500 mode will not support every feature at first, partly because Confluent wants confidence in it before making it the default, and partly because the upgrade path itself still needs perfecting.

That path is a bridge release: a version that can coexist with both pre- and post-KIP-500 Kafka, enabling zero-downtime upgrades. You upgrade to the bridge, then upgrade again to a release with no ZooKeeper at all. In a half-upgraded cluster the controller is always a post-KIP-500 broker, and it is the only node still talking to ZooKeeper, mirroring changes into it. Every other broker must therefore treat ZooKeeper as read-only — because a post-KIP-500 broker is no longer listening on ZooKeeper, a peer that announced a config or ACL change by writing to ZooKeeper directly would be announcing it to nobody.

The bill is real. The controller has to be split apart — the ZooKeeper-facing half separated from the general-purpose logic like replica set management. New controller APIs have to be written to replace what used to be a ZooKeeper write, such as AlterIsr (KIP-497), which lets a replica tell the controller its in-sync replica set changed without touching ZooKeeper at all. Old clients that send RPCs like IncrementalAlterConfigs to the wrong broker need a redirection system to forward them to the active controller.

What Confluent buys with that work is not raw speed. It is the removal of a whole category of things an operator has to know: one system to configure, one security model, one place where the truth lives, and no second cache to drift out of sync. Replacing ad hoc ZooKeeper access with documented RPCs also makes cross-version compatibility easier to maintain, and cuts the number of ZooKeeper writes a common operation like an ISR change requires.

The dependency was never the expensive part — the duplication was: two logs, two config surfaces, two views of who’s alive, and a scaling ceiling hidden in the one code path that only runs when something has already gone wrong.