Meta's data ingestion system scrapes several petabytes of social graph data out of MySQL every day. The legacy pipeline design worked at small scale and started to buckle as landing-time requirements got stricter — so Meta migrated tens of thousands of jobs to a new architecture without a cutover.

Meta’s social graph runs on one of the largest MySQL deployments in the world, and every day its data ingestion system incrementally scrapes several petabytes out of that MySQL fleet into the data warehouse — the raw material behind analytics, reporting, and every downstream product from day-to-day decision-making to ML model training. The legacy system that did this scraping was built around customer-owned pipelines, an approach that, in Meta’s own words, “functioned effectively at a small scale.”

The replacement moves away from customer-owned pipelines entirely, toward what Meta describes as “a simpler self-managed data warehouse service that still operates efficiently at hyperscale.” By the time the migration finished, 100% of the workload — tens of thousands of individual ingestion jobs — had moved onto the new system, and the legacy system was fully deprecated.

Every migrated job had to clear three explicit bars before being trusted in production: no data quality difference between old and new (verified by comparing row counts and checksums), no landing-latency regression, and no resource-utilization regression. Critical tables carried extra, negotiated criteria agreed on with the teams depending on them.

The migration ran each job through a three-phase lifecycle. In the Shadow Phase, a new-system job consumed the same production source but wrote to a separate shadow table — a production-realistic test with an isolated place to inspect results, monitored continuously for row-count and checksum mismatches logged to Scuba, Meta’s real-time data management system.

Once a job ran reliably in production as a shadow, it entered the Reverse Shadow Phase: the shadow job’s output was switched to write the real production table, becoming the new production job, while the old production job kept running but now wrote to the shadow table instead. Both benefits ran at once — ongoing data-quality comparison continued, and rollback meant flipping which system counted as “production” rather than reconstructing anything. Migration Cleanup then removed the old system’s now-shadow job once no discrepancies remained.

None of this scaled without dedicated tooling. Because job status signals streamed continuously to Scuba, Meta built external tools that automatically promoted or demoted jobs between lifecycle stages based on whether they met the criteria — plus dashboards so engineers could track both the overall rollout and individual jobs. That automation, not just the three-phase design, is what made a tens-of-thousands-of-jobs migration tractable rather than a manually-tracked spreadsheet exercise.

A safe large-scale data migration isn’t one cutover — it’s two systems running side by side long enough that switching which one is “production” becomes a label flip instead of a leap of faith.