Zero downtime on a petabyte-scale store didn't come from one clean cutover — it came from making every table's migration reversible, one Kafka-backed stage at a time.
Mussel, Airbnb’s core key-value store bridging offline and online workloads, had reliably supported the company for years, first described in a 2022 blog post. But new requirements — real-time fraud checks, instant personalization, dynamic pricing, and growing data volume — needed a platform that combined real-time streaming with bulk ingestion while staying easy to operate.
Mussel v2 is a complete re-architecture: a stateless Dispatcher service on Kubernetes replaces v1’s tightly coupled, protocol-specific design, reducing scaling and node-replacement work from hours of manual scripting to minutes of automated rollout. Dynamic range sharding and presplitting replaced static hash partitioning, keeping p99 reads under 25ms even on tables past 100TB. Namespace-level tenancy, quota enforcement, and dashboards replaced opaque resource usage. In deployment, Mussel v2 can simultaneously ingest tens of terabytes in bulk, sustain more than 100,000 streaming writes per second on the same cluster, and hold p99 reads under 25ms.
Because Mussel v1 offered no table-level snapshots or change-data-capture streams — standard tools for this kind of migration elsewhere — Airbnb built a custom migration pipeline. Each table moved through four stages: Blue, where all traffic still ran on v1 while data was bootstrapped behind the scenes into v2; Shadowing (Green), where v2 processed reads and writes in parallel but only v1’s responses were used, letting the team validate correctness and performance risk-free; Reverse, where v2 took over active traffic with v1 on standby and automatic circuit breakers ready to fall back instantly if v2 showed elevated errors or lag; and Cutover, completed table by table, with Kafka serving as the durable intermediary throughout.
Migrating from v1’s eventually consistent model to v2’s strongly consistent one introduced new problems around write conflicts, requiring write deduplication, hotkey blocking, and lazy write repair — sometimes trading storage cost or read performance to get there. Shifting from hash-based to range-based partitioning meant inserting large consecutive data ranges could create hotspots in v2, requiring accurate sampling of v1’s data to presplit it into balanced shards ahead of ingestion. v2 also doesn’t push down range filters as effectively as v1, requiring client-side pagination for some prefix and range queries that used to be server-side.
Zero downtime on a petabyte-scale migration isn’t a property of the destination system — it’s a property of making every intermediate step independently reversible, one table at a time.
Source — read the original
A plain-language, AI-drafted and human-edited retelling of the article published on medium.com, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.