Magic Pocket stores multiple exabytes across over half a million disks. Pocket Watch is the six-system verification layer that continuously proves the data is still there — and it runs more traffic than the customers do.

Magic Pocket is Dropbox’s in-house, multi-exabyte storage system, spread across more than 500,000 disks in production. It stores data using a variant of Reed-Solomon erasure coding, similar to Local Reconstruction Codes, which buys very high durability with low storage overhead. On paper the durability target is 27 nines — 99.9999999999999999999999999%.

The Dropbox team states the problem more bluntly: your disks are lying to you. Hard drives are a reliable technology, but when you have over half a million of them, they fail in every strange way imaginable — bad sectors, silent corruption, bit-flips, and fsyncs that don’t actually fsync. A drive can report that a write succeeded when nothing landed.

The intuitive defense is replication: keep several copies, and losing one disk costs you nothing. That handles random hardware death. It does not handle the failures that actually erase data — a software bug in the deleter, an operator error, a bad configuration change, or silent corruption that quietly rots a fragment while every copy still reports “present.”

So the engineering question is uncomfortable: at this scale, you cannot trust the storage layer to tell you the truth about itself, and you cannot pause the system to audit it. How do you continuously prove that hundreds of billions of blocks still exist, still match their checksums, and still live in the zones users were promised — without ever taking the store offline?

Pocket Watch is Dropbox’s answer — not one checker but six specialized verification systems, each aimed at a different way data can go wrong. The Disk Scrubber reads every bit on disk and validates it against checksums, sweeping each disk roughly every one to two weeks to catch silent corruption. The Metadata Scanner iterates the Block Index at around a million blocks per second, confirming each block actually exists on the storage nodes that should hold it, aiming for a full scan per zone every week. The Storage Watcher is a black-box end-to-end checker that samples 1% of written blocks and re-reads them after one minute, one hour, one day, one week, and one month.

The other three guard deletions and placement. The Trash Inspector verifies that every extent moved to trash was genuinely safe to remove before it’s purged; the Extent Referee polices filesystem transitions so nothing is unlinked without proper authorization; and the Cross-zone Verifier scans the File Journal to confirm each file physically lives in the zones the user requires. Because these systems run constantly against live data, more than 50% of the workload on Dropbox’s disks and databases is its own internal verification traffic — the system reads itself more than customers do.

Dropbox doesn’t trust Pocket Watch blindly either. Before any new code version ships, the verifiers run for at least a week on a staging cluster that is tens of petabytes in size, and a team member has to sign off — because a verification system with a bug is worse than none, since it can authorize the very deletions it’s supposed to prevent.

At exabyte scale, durability is not a number you set once; it’s a workload you run forever — because the only way to know your data is still there is to keep reading it back and refusing to trust anything, including the checker that does the reading.