Etsy's ~1,000-shard MySQL cluster routed every query through a single index database — a full-site outage waiting to happen. Five years later it runs on Vitess.

Etsy has run a sharded MySQL architecture since around 2010: about 1,000 tables spread across about 1,000 shards, holding over 425 TB of data and taking roughly 1.7 million requests per second. Engineers reached that data through a proprietary ORM, and each sharded table designated a “shardifier id” — often shop_id or user_id, though over 30 options were in use — to decide which shard a record lived on.

The catch was how the ORM found a record’s shard. Shard mappings were assigned randomly at creation and stored in a single, unsharded “index” database. Every query to a sharded table first hit the index database to look up the mapping, then went to the right shard. Sharding itself was resilient — a downed shard cost only 1/1000 of traffic — but that resilience routed through one unsharded chokepoint.

Etsy adopted Vitess — open-source software that provides an abstraction layer for scaling, deploying, and managing large MySQL clusters — first adding it in 2018 as a pass-through, then porting the actual shard logic into Vitess “vindexes,” the constructs that define sharding strategies. The results they claim are operational, not flashy: scaling is no longer manual and takes days instead of months, the index database is gone as a single point of failure, and shard infrastructure is now hidden from developers writing data models and queries.

Crucially, database performance stayed largely the same as before, making the change invisible to end users. And one capability got dramatically faster: cross-shard “scatter” queries, where one model’s query time dropped from about 2 seconds to about 20 ms.

The obvious route — an out-of-the-box hash vindex that computes the shard algorithmically — would have required re-sharding all of Etsy’s data, because their existing mappings were random, not algorithmic. That was estimated to be manual and to likely take years, so instead the team wrote custom vindexes that ported the existing shard logic into Vitess without moving data. They changed the ORM’s shard-assignment algorithm to match Vitess’s hash algorithm, then stored all pre-existing mappings in a read-only SQLite database copied directly onto each Vitess server to avoid the latency of an external call. A custom “hybrid” vindex then split by a threshold id: newer ids use the hash vindex, older ones the SQLite lookup.

Rollout was deliberately incremental — one table at a time, using Etsy’s experimentation framework to ramp the percentage of traffic using vindexes for each table and to compare vindex routing against the old ORM routing, with a fast path back to 0% if anything looked wrong. Over a codebase with more than a decade of query patterns, that let them surface incompatibilities gradually rather than risk a large-scale outage.

The honest cost was scope and coordination: five years, roughly 2,500 pull requests and 6,000 queries. Vitess requires the shardifier id in a query’s WHERE clause to route it, but the old ORM never did, so many queries lacked it and had to be found and fixed. Transactions were trickier still — the ORM used to hit shards as separate databases, while Vitess presents them as one, creating different connections, and atomicity is only guaranteed per connection. That forced every table written together in a transaction to be ramped onto vindexes at the same time.

That constraint collided with reality: just 27 models — about 3% of tables, covering the most complex and critical data like receipts, listings, and transactions — accounted for a third of all database traffic and were tightly coupled by transactions. Despite ramping table-by-table to minimize risk, the team had to ramp those 27 models in unison, coordinating across the company for the single riskiest change.

You can dissolve a scary migration into hundreds of reversible steps, but transaction boundaries don’t divide — the tables that share a transaction have to cross together, no matter how carefully you ramped everything else.