THE MIGRATION LINE
INCIDENT // NETFLIX-2016 MIGRATED

On January 4, 2016, right before Netflix expanded into 130 new countries, its billing infrastructure went 100% AWS cloud-native — splitting one Oracle database into Cassandra for subscribers and MySQL for payments, because payment processing needed ACID transactions.

NETFLIX · 2016 · BILLING / CASSANDRA / MYSQL / AWS
System stress over time Breach at T+5
Jan 4, 2016 Cutover
130 new countries Global launch
since 1997 Billing history
Oracle GoldenGate Replication tool
BEFORE

One giant Oracle database, holding two very different jobs

Netflix’s billing infrastructure tracked open and paid billing periods, account credit, payment status, charge requests, and the date each member had paid through — plus feeding financial systems for revenue and tax reporting. All of it ran against a single, giant Oracle-based system in Netflix’s own data center, with integrations reaching into member accounts, payment processing, customer service, and DVD shipping.

The scale was not small: billions of rows of constantly changing data, accumulated since Netflix’s founding in 1997, growing every minute in one large shared database.

AFTER

Two data stores, chosen for two different guarantees

Netflix disintegrated the single Oracle system into a services-based architecture and split the data across two purpose-built stores. Subscriber data moved to Cassandra, giving it the horizontal, multi-region scale the APIs needed. Payment processing integration needed ACID transactions, so all relevant data moved to MySQL instead — open-source, running on Netflix-managed EC2 instances, with a multi-region master using DRBD copy and read replicas in different regions.

On January 4, 2016 — just before Netflix expanded into 130 new countries — billing infrastructure became 100% AWS cloud-native.

THE BRIDGE

Country by country, with the old system still running underneath

The migration ran incrementally, not as a single cutover. Netflix built proxies and redirectors to keep applications in the data center working, unimpacted, until they were ready to move. New country launches gave the team a chance to test the new Cloud infrastructure on clean data first, before extending it to large, legacy countries like the US, which carried both streaming and DVD billing.

Migrating existing members’ data required real downtime: subscription data moved from Oracle to Cassandra country by country, with retry capability for failed processing and optimistic customer-state handling so members weren’t penalized while processing was halted. For the payments side, Netflix used Oracle GoldenGate to replicate tables — and their ongoing incremental changes — across the heterogeneous Oracle-to-MySQL boundary while production kept running, over a period of months. Weeks before the final switch, they ran their test database continuously against MySQL, fixing issues there before validating against Oracle and releasing to production.

WHAT IT COST

An honest admission about what they'd do differently

Netflix’s own reflection is candid: “We underestimated testing automation needs. We did not have a good way to test end to end flows.” Being a SOX system added its own tax — all migration tooling needed to be SOX-compliant, with auditability of deployments and actions built in along the way, using Spinnaker, Atlas, and a comparator built to reconcile Cassandra data against Oracle by country.

Post-migration, Netflix reports being more efficient and lighter in software footprint, with applications able to scale horizontally to keep up with subscriber growth — and, in their words, this was “a major cross functional engineering effort” spanning core platform, security, database engineering, tooling, and big data platform teams.

When one dataset needs ACID guarantees and another needs multi-region scale, splitting them into two purpose-built stores can beat forcing both through the same database.

Source — read the original

https://netflixtechblog.com/netflix-billing-migration-to-aws-451fba085a4

A plain-language, AI-drafted and human-edited retelling of the article published on netflixtechblog.com, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.

← All systems