Box moved 600 TB of file metadata off three on-premises HBase clusters onto two Bigtable clusters with zero downtime, by writing to both databases synchronously until it trusted the new one.

Box stored customer file metadata in HBase — the mapping from a file to the physical storage locations that file actually lives in. A service called Storage Service, running on Kubernetes, owns that metadata, and it is read on every single upload and download at Box. At the start of the migration, each HBase cluster held over 600 billion rows and 200 terabytes of data, taking roughly 15,000 writes and 20,000 reads per second, with headroom to serve millions of requests during analytical jobs.

The architecture was three fully replicated clusters in different geographic regions: two active for high availability, and a third to absorb routine maintenance. Each regional Storage Service wrote to its local cluster; those writes replicated outward. On reads, it hit the local cluster first and fell back to the others if replication lagged.

The constraint was absolute: secure file upload and download is core to Box’s content cloud, so the migration had to move over 600 TB of data with zero downtime. Bigtable is HBase-compatible and needs no maintenance downtime, which meant Box could drop from three clusters to two — one primary taking all requests, one replicated secondary for disaster recovery.

Which raised the awkward question the whole plan turned on: how do you merge three replicas into two?

In theory the metadata in all three HBase clusters was the same — writes were partitioned, replication was guaranteed. In practice it had drifted, and Storage Service had been quietly reconciling the inconsistencies at read time. So before choosing how to import, Box measured the drift.

They ran the Google-provided Multiverse Scan Job, a MapReduce job that scans HBase table snapshots in parallel to perform what amounts to a sort-merge-join across the three tables, comparing rows and cells. It scanned the entire table while comparing a random 10% of critical rows, took 160 Dataproc worker nodes, ran for four days, and dumped the differences into BigQuery for analysis. The inconsistencies fell into three buckets: rows missing from a cluster, rows present but missing columns, and rows present with differing non-critical columns.

Because the drift was mostly missing rows and columns rather than genuine conflicts, consolidating all three snapshots into a single Bigtable cluster would produce the most complete copy — and Bigtable’s own replication could carry it to the secondary. That settled the backfill. The live traffic was the other half.

Box chose synchronous modifications: every successful HBase write triggers the same Bigtable write, serialized, and if either step fails the whole request fails. That guarantees atomicity at the cost of making write latency the sum of both databases — a tradeoff they accepted, because writing to both in parallel would have introduced complicated logic into Storage Service. Storage Service also does a lot of check-and-modify operations, which could not be mirrored while Bigtable was still un-backfilled, so those deferred to HBase: modify only if the HBase check-and-modify succeeded.

The clusters were far too large to validate row by row, so Box did it three ways. Async read validation read from Bigtable on every customer read and logged the differences — noisy, because so many reads are immediately followed by an update, but it surfaced a real incompatibility: Bigtable regex scans behave differently from HBase, supporting only equals comparators and using RE2, which treats the any-character dot differently. They had to ship a Bigtable-specific regex and prove it returned what they expected. A Dataproc hash-comparison job over a 3% sample found a 0.1% mismatch, all traceable to optimistic modifications on certain columns, needing no re-import. A third, application-level job walked the filesystem and asked Storage Service to compare the two databases the way a customer would see them, and agreed.

Only then did reads flip to Bigtable, with synchronous dual writes to HBase left running as a rollback path until they were shut off and Bigtable became the source of truth. The wins Box reports: an 85% smaller storage footprint, autoscaling on CPU and storage that physical hardware never allowed, no more shuffling traffic around to patch, MapReduce jobs finishing in under 24 hours instead of days, and a secondary cluster running 25% lighter than the primary — where all three HBase clusters had been sized evenly. Bigtable also became queryable from BigQuery in real time, which HBase never was.

A migration is the only audit that actually runs — Box did not discover its replicas had drifted until it was forced to compare them, and the read path had been hiding that from everyone for years.