Airbnb's biggest Kafka cluster stored its metadata in a shared Zookeeper ensemble that nobody owned, so the team moved it to a dedicated one with no downtime and no data loss.

Airbnb runs several production Kafka clusters. The largest is also the oldest — hundreds of brokers carrying over 1GB/s of incoming traffic, powering event logging and change data capture across the company’s data infrastructure.

Kafka doesn’t store its own cluster metadata. It leans on Zookeeper, a separate distributed coordination system, to hold it. And the Zookeeper this cluster leaned on was a legacy one — a multi-tenant ensemble shared between many different production use cases.

Airbnb’s account doesn’t describe an outage. What it describes is exposure. Because the Zookeeper cluster was shared, any incident on it would hit every dependent service at once — including Kafka, and therefore including the data pipelines built on top of Kafka.

The second problem was organizational rather than technical. The cluster lacked a clear owner. A shared dependency with no owner is a dependency nobody is on the hook for.

The root cause was the shape of the dependency itself. Kafka’s coordination layer sat inside a blast radius it didn’t control and couldn’t shrink, and no team was accountable for that blast radius.

There was, in principle, a way out of Zookeeper entirely. Kafka 2.8.0 had just shipped changes for KIP-500, the community’s proposal to remove Kafka’s Zookeeper dependency. But the feature was neither complete nor production-tested. Because of that, Airbnb judged it would take a few more releases before they’d trust running Kafka without Zookeeper at all — so the goal became a better Zookeeper, not no Zookeeper.

The approach was inspired by Yelp’s earlier write-up, modified to use Zookeeper observers so the source cluster never had to be touched. Call the old ensemble zk-source and the new, empty one zk-dest.

Phase one — configure the zk-dest hosts to join zk-source as observers, and let them replicate all of its data. Observers don’t vote on pending commits; they only learn about committed proposals from the leader. Because of that, they could be added without affecting quorum, and without editing a single config file or restarting a single host in zk-source. The leader accepts observers even though they aren’t in its server list.

Phase two — point every Kafka broker’s Zookeeper connection string at the zk-dest observers and roll-restart the brokers. Kafka is still reading zk-source’s data, just reaching it through the observers.

Phase three — sever the link. Insert iptables rules on the zk-dest hosts rejecting all Zookeeper TCP traffic to and from zk-source, confirm the Kafka subtree is fully caught up, then reconfigure zk-dest to list only its own hosts as participants. It forms its own quorum, holding a copy of the data it was replicating a moment earlier.

Zookeeper is unavailable to Kafka between the first and last actions of phase three. That window is the whole risk. Producers and consumers keep working through it — as long as every broker stays healthy — because they talk to brokers, not to Zookeeper. But the Kafka controller can’t detect a broker that dies while Zookeeper is out, so it won’t elect new partition leaders, and depending on the producer’s retry policy, messages to those partitions can be dropped and lost forever.

The other hazard was session expiry. Kafka brokers register ephemeral ZNodes under /brokers/ids that vanish when their Zookeeper session times out, and the source cluster’s session timeout was 40 seconds — a number Airbnb chose to work with rather than change, since changing it meant touching zk-source. Blocking traffic between the two clusters also prevented the zk-source leader from propagating session expirations into zk-dest, which is what keeps the controller from concluding, at the end of the migration, that a perfectly healthy broker has died.

Two more speed tricks bought margin inside that window. To verify the Kafka subtree was current, they compare zxids — Zookeeper’s ordered transaction ids — and to read them fast they modified Zookeeper’s bundled transaction log dump tool to print only the latest transactions and to use memory-mapped I/O. Dumping a full 500MB log took around 20 seconds; the modified tool took about one.

When the risky part of a migration is a short window of unavailability, the engineering work isn’t avoiding the window — it’s making everything inside it fast enough, and reversible enough, that the window never turns into an incident.