Plenty of CPU. Plenty of I/O. No room. The bottleneck was the one you can't tune your way out of.
UDB is one of Meta’s main databases — the tier that manages data about Facebook’s social graph. It ran on MySQL with compressed InnoDB, and Meta’s own verdict on that setup is not grudging: MySQL with InnoDB was fast and reliable.
Several years earlier they had moved UDB’s hardware from Flashcache to pure Flash, which addressed some performance issues. That fix worked. It also quietly changed which resource ran out first.
On pure Flash, UDB was space-bound. The machines had excess CPU and excess random I/O capacity sitting idle — the database simply could not fit more data on the hardware it had. Meta wanted to support the same workload and the same data with less hardware, and this was the wall in the way.
That shape of problem is unusually unforgiving. You cannot cache your way out of it, and you cannot tune your way out of it, because the resource you have spare is not the resource you need.
Early experiments showed MyRocks cut space usage in half compared with compressed InnoDB, without significantly increasing CPU or I/O utilization — which is exactly the trade Meta had spare capacity to make. To match InnoDB where the database is much bigger than RAM, they implemented a clustered index, transactions (atomicity, row locks, MVCC, repeatable read and read committed, 2PC across binlog and RocksDB), crash-safe primaries and secondaries, SingleDelete for efficient secondary-index deletions, and fast data loading, index creation, and drop table.
Production OLTP migrations have to happen without stopping the service, without degrading latency or throughput, without returning wrong data, and without disrupting operations. MyRocks had a unique advantage in meeting that bar: it is still MySQL. Client protocols and schema definitions were unchanged, so no application change was needed (minor differences like Gap Lock aside). Existing MySQL administration tooling supported both engines. And crucially, the engines could replicate to each other in both directions.
Each replicaset had one primary and four secondaries across worldwide regions. Step one: build the first MyRocks secondary by dumping from InnoDB in the same region and bulk-loading into MyRocks — replication on the source InnoDB secondary was stopped during dump and load, with reads served by other InnoDB secondaries. Because MyRocks was optimized for loading and index creation, it could copy hundreds of GBs of InnoDB data per hour. After catch-up and verification, the InnoDB instance in that region was deleted.
Step two added a second MyRocks secondary, so that losing one wouldn’t force another dump-and-load — recovery could instead use a lightweight online binary copy (myrocks_hotbackup). Because deleting InnoDB instances kept pace with adding MyRocks ones, five instances per replicaset became three InnoDB and two MyRocks, and Meta migrated without increasing the number of physical servers in use. Step three promoted a MyRocks secondary to primary, the hardest step, since a primary serves concurrent writes to the same rows and poor concurrency handling causes errors and stalls. Step four copied MyRocks to all regions and removed InnoDB entirely.
Because MyRocks and RocksDB were newer, Meta ran continuous data consistency verification — during the migration and after, in production. They compared row counts and column checksums between a MyRocks table’s primary and secondary keys to catch corruption in either. They compared row counts and checksums across the InnoDB and MyRocks instances of the same replicaset, using GTID to start both transactions from a consistent snapshot. And they used MySQL’s Audit Plugin to capture real production queries and shadow-replay them against both engines, comparing results — later extended to primary traffic so concurrency bugs could be fixed before any primary went live.
Meta completed the UDB migration and successfully cut storage usage in half — the goal they set out with, alongside halving the number of servers used. The migration was completed quickly and without causing production problems.
Two operational hazards had to be handled on the way. MyRocks and RocksDB had limited support for direct I/O, so Meta switched to buffered I/O — and older Linux kernels had known issues where heavy buffered I/O caused swap and virtual-memory allocation stalls, so their kernel team fixed the vm allocation stalls in Linux 4.6 and upgraded to it before migrating. Separately, aggressively deleting the old InnoDB files caused long TRIM stalls on Flash, lasting seconds to tens of seconds. Rather than rm a 100GB file, they built a tool that split large files into ~64MB chunks and deleted each with short sleeps, throttled to around 128MB per second.
When the scarce resource changes — and hardware upgrades quietly change it — the fix isn’t a better configuration of the same engine, it’s a different engine that spends the resource you now have in surplus.
Source — read the original
A plain-language, AI-drafted and human-edited retelling of the article published on engineering.fb.com, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.