Airbnb runs an open-source distributed SQL database across three independent Kubernetes clusters, one per AWS availability zone, so a bad deploy only ever costs a third of the fleet.

Airbnb’s traditional approach, like many organizations, was deploying databases on costly, high-end standalone servers and scaling through sharding. As data demands grew, that strategy’s limits showed up as increasingly long and complex maintenance projects — the kind of debt that accumulates quietly until every schema change or capacity increase becomes its own effort.

Airbnb’s answer was to deploy an open-source, horizontally scalable distributed SQL database across three independent Kubernetes clusters, each living in a different AWS availability zone — an uncommon design given its complexity, but one that met their reliability and operability targets. Storage volumes attach via AWS EBS through Kubernetes Persistent Volume Claims, so a replaced node automatically reattaches to its existing volume rather than needing a full data copy.

Airbnb now runs several production clusters on this design, the largest handling 3M queries per second across 150 storage nodes, storing more than 300 TB of data spread across 4M internal shards — at 99.95% availability.

To make node replacement safe, Airbnb sorted replacement events into three categories — database-initiated (config changes, version upgrades), proactive infrastructure events (instance retirements, node upgrades), and unplanned failures (a node going unresponsive) — and built a custom Kubernetes operator to coordinate them. A custom check confirms every node is healthy before the operator deletes a pod for a database-initiated event, and an admission hook intercepts Kubernetes’ own pod eviction attempts, rejecting them but tagging the pod so the database operator can act on it safely instead, keeping the two kinds of replacement from colliding.

The multi-AZ split was tested for real: a faulty config deployment abruptly terminated all VMs of a specific type in a staging Kubernetes cluster, deleting most of that cluster’s query layer pods. Because the disruption was confined to a single Kubernetes cluster and AZ, two-thirds of the overall query layer nodes stayed operational and there was no impact. Separately, because EBS can hit tail-latency spikes as high as 1 second at p99, Airbnb added a storage read timeout that lets queries transparently retry against other nodes, along with cross-AZ replica reads and a stale-reads option for workloads that can tolerate it.

Airbnb is upfront that this setup adds complexity: three independently managed Kubernetes clusters instead of one, a custom operator to maintain, and overprovisioned capacity held in reserve specifically so an entire AZ, cluster, or all storage nodes in a zone can go down without taking traffic with it. They also chose self-managed Kubernetes over cloud-managed control planes specifically because it allows rolling back a bad control-plane upgrade — a rollback path cloud-managed Kubernetes doesn’t guarantee.

High availability for a stateful system isn’t a setting you turn on — it’s complexity you deliberately keep in three separate places so a single mistake can only ever reach one of them.