Team-by-team, Twitter's exabyte move to BigQuery crawled — two weeks per dataset and choking on a 77 Gbit/s proxy. An automated framework cut that to one hour.
Moving an exabyte isn't the hard part — onboarding every dataset by hand is. So they stopped doing it by hand.
Twitter’s data lived on-premise in HDFS, stored in Thrift format, and the goal was to get it into BigQuery on Google Cloud so employees could actually query it at scale. The first approach was the obvious one: let individual teams ingest their own data, writing Scalding jobs to pull datasets from the on-prem Hadoop filestore up into BigQuery.
That worked, in the sense that data moved. It did not work in the sense that it took roughly two weeks of engineering to onboard each new dataset.
AFTER
One hour per dataset, and an exabyte you can query
Twitter replaced the do-it-yourself pattern with an automated ingestion framework, and the numbers moved sharply. Onboarding a new dataset dropped from two weeks to one hour. Ingestion throughput increased by 100%. Data availability rose by more than 165% — all in just under six months. Maintenance dropped too, by leaning on managed services like Airflow instead of bespoke per-team jobs.
The payoff shows up in usage: Twitter employees now run over 10 million queries a month against almost an exabyte of data in BigQuery. The data didn’t just move — it became reachable.
THE BRIDGE
Dedicated copy clusters on a private road, mapped one-to-one
Two structural choices did the heavy lifting. Twitter and Google Cloud designed a resource-hierarchy architecture that maps HDFS to BigQuery one-to-one, built for scalability and security — so the shape of the on-prem world had a predictable, permission-aware home in the cloud rather than an ad hoc pile of tables.
To actually move the bytes, Twitter stood up dedicated copy clusters per data center whose only job was to copy data from the production clusters to Google Cloud Storage, over a private link to GCP rather than the congested shared proxy. Dedicated movers on a dedicated road, instead of every team elbowing through the same gate.
WHAT IT COST
Automation is a system you now own
The trade Twitter accepted is the one every automation buys: they replaced many teams’ throwaway scripts with a single framework, private-link plumbing, and per-data-center copy clusters that the platform team has to operate and maintain. The gains — an hour per dataset, doubled throughput, an exabyte serving over 10 million queries a month — are real and cited, but they’re the output of infrastructure that is now load-bearing. And it’s a hybrid arrangement: the source data still originates on-prem and is copied up, not abandoned.
When you’re moving an exabyte, the bottleneck is rarely the bytes — it’s the two-week manual step you’re about to run a thousand times, and the shared pipe everyone is fighting over.
The 2019 'Partly Cloudy' start scaled into a full exabyte push by 2022.
A plain-language, AI-drafted and human-edited retelling of the article published on blog.x.com, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.