THE MIGRATION LINE
INCIDENT // GOV-UK-NOTIFY-2024MIGRATED

GOV.UK Notify moved a 400GB, 1.3-billion-row PostgreSQL database off a retiring platform and onto its own AWS account — and cut over live traffic in 11 seconds.

GOV.UK NOTIFY·2024·DATABASE / POSTGRESQL / AWS-DMS / PUBLIC-SECTOR·✓ Sourced — gds.blog.gov.uk
System stress over timeBreach at T+6
400GB / 1.3B rowsDatabase size
~6 hoursFull load time
40+Practice runs
~11 secondsActual downtime
BEFORE

A database with a landlord who's leaving

GOV.UK Notify’s PostgreSQL database ran on the GOV.UK Platform as a Service — about 400GB, spread across 1.3 billion rows, 85 tables, 185 indexes, and 120 foreign keys, taking on the order of 1,000 inserts or updates a second on a typical weekday, plus a similar volume of reads. It ran PostgreSQL 11.

Two clocks were running out at once: the PaaS itself was being retired, and PostgreSQL 11 was about to become unsupported. Neither was optional to fix — the team had to move the database, and it had to land somewhere newer.

AFTER

Same data, new account, PostgreSQL 15

The destination was a new RDS PostgreSQL database inside GOV.UK Notify’s own AWS account, upgraded straight to PostgreSQL 15. Getting there took under five minutes of downtime as the target — and the team beat that target by two orders of magnitude: the actual cutover took roughly 11 seconds.

THE BRIDGE

Full load, ten days of replication, then a DNS flip

The move ran on AWS Database Migration Service in two phases. First, a full load: DMS copied every table, row by row, up to a fixed point in time — about six hours for 1.3 billion rows. Foreign keys and indexes were deliberately left off the target until after the full load finished, so they wouldn’t slow down the bulk copy. Schema itself came across via pg_dump, and a VPC peering connection linked the DMS instance to the target database across the two AWS environments.

Second, change data capture: DMS read the source database’s transaction log for everything written since the full load began, and kept replaying it into the target. Initial catch-up took a couple of hours; after that, it just ran in the background — for about ten days — keeping the new database in lockstep with the old one while the team rehearsed the cutover more than 40 times.

On migration day itself — a Saturday evening, one of Notify’s quietest windows — a Python script ran the whole sequence: pg_terminate_backend killed every connection to the source database in under a second, a password change stopped anything from reconnecting, the script checked DMS’s status tables and wrote a test record to confirm replication had fully caught up, and then it flipped a Route53 DNS record for database.notifications.service.gov.uk — set to a 1-second TTL — from 100% weighted to the old database to 100% weighted to the new one. No application redeploy required.

WHAT IT COST

DMS worked, but they'd choose differently next time

The team is candid that AWS DMS wasn’t the smoothest tool for a PostgreSQL-to-PostgreSQL move — it “potentially added more complexity, and an unfamiliar replication process.” For a same-engine migration, they say they’d invest more time trying alternatives like pglogical next time. DMS did the job here, but the honest verdict is that it was the right tool given the timeline, not necessarily the ideal one for this exact case.

AFTERMATH

Next up: the applications themselves

With the database moved, the database layer was no longer coupled to the retiring PaaS. That set up the next phase of the migration: moving the applications themselves onto AWS Elastic Container Service.

A live database migration doesn’t get safe because the tool is fast — it gets safe because you rehearse the exact sequence until the failure modes are boring.

Source — read the original

A plain-language, AI-drafted and human-edited retelling of the article published on gds.blog.gov.uk, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.

← All systems