Dropbox rewrote Magic Pocket's storage engine in Rust and it worked so well that almost nobody learned it — and that turned out to be the problem.

Magic Pocket is Dropbox’s geo-distributed data storage system. It has run on close to a million simultaneous storage nodes and directly manages exabytes of on-disk storage. On a fleet that size and that expensive, the team defined “overhead” bluntly: overhead is anything that isn’t a hard drive.

Dropbox had already moved its backend stack from Python to Go. But the early Go codebase behind the storage engine carried a constant threat of OOM issues, and the CPU and RAM sitting in each storage node was cost the team wanted back. So a handful of engineers rewrote the storage engine in Rust, which gave them predictable utilization from a lack of runtime and tight control over memory allocation — C++ levels of control inside a safer, more ergonomic language.

Years later, the accounting looked strange. The Rust components were doing exactly what they were built to do. Magic Pocket has run without a single major incident. And yet innovation on those Rust components slowed for a period of time, and the team left holding them ended up with mixed feelings about the use of Rust to begin with.

Nobody had shipped a bad rewrite. The code was fine. Something else had failed.

The first clue is the split. Only a handful of engineers rewrote the storage engine in Rust; the rest of the team kept building the rest of the system in Go. That split was necessary while the Rust work was a small skunkworks project — but it persisted for years afterwards.

The second clue is that Rust is genuinely hard to onboard into. Developers can usually jump between Python, Go and Java and pattern-match their way to success without being fluent. The first time a developer runs afoul of the borrow checker, or has to reason about trait objects, they need actual dedicated learning time. The third clue is the quiet one: the Rust components worked well and didn’t require much maintenance — so nobody had a reason to go learn them.

Then the original authors left the team to work on Nucleus, the clean-slate rewrite of the Dropbox sync engine. Because the Rust components had never demanded attention, and because Rust is expensive to pick up casually, there weren’t many engineers left on the team who were experts in these very large systems.

On Nucleus, the next major Dropbox project in Rust, the team made sure the entire team was fluent and actively contributing to the codebase. Inside the Nucleus team, Rust was an unequivocal success: the codebase runs on over half a billion devices, manages trillions of files, and has an unblemished reliability record. A testing framework codenamed Trinity deterministically verified invariants across complex thread interleavings, which Rust’s futures support and the freedom to write an adversarial scheduler made possible.

The boundaries were harder. Large companies have hundreds of internal systems that talk to each other — monitoring, RPC frameworks, release process, authentication — and every one of them has to be ported to each new supported language. Nucleus’s prototype API server, Tomahawk, was originally written in Rust and the team took on that porting burden themselves. Eventually it was just too much work, and they rewrote Tomahawk in Go, the company’s de facto backend language. Sometimes the barrier to migration is just too high.

Every hop — Python to Go, then Go to Rust on the highest-profile projects — bought real performance, type safety and productivity. But as senior engineers moved to the exciting new stack, there was less cutting-edge work and less library investment left behind on the codebases they came from. A new language doesn’t just cost the work it takes to write; it costs the work it pulls away from everything else.

A rewrite that never breaks is the easiest kind of institutional knowledge to lose — reliability buys you silence, and silence is indistinguishable from nobody understanding the code.