THE MIGRATION LINE
INCIDENT // ATLASSIAN-2020 MIGRATED

Confluence Cloud's real ceiling wasn't CPU or memory — it was database connections, and one busy tenant could exhaust them for everyone. Vertigo rebuilt the architecture around that fact.

ATLASSIAN · 2020 · AWS / MULTI-TENANT / ARCHITECTURE / SCALING
System stress over time Breach at T+3
~30K req/sec cc-proxy throughput
under 25 ms Latency
99.99% Success rate
6 tenant shards Regions
BASELINE

Every tenant drinking from the same well

Confluence Cloud ran on a single-tenant architecture, and Project Vertigo — begun in 2015 — set out to move it to a stateless, multi-tenant one built on AWS. The point of the rebuild was better scale, reliability, and performance, plus the ability to deploy frequently and seamlessly.

But the constraint that shaped everything wasn’t compute. It was the database connection pool.

REQUIREMENTS

What the new shape had to guarantee

The architecture needed to stop letting one customer’s load become another customer’s error. That means per-tenant limits on database connections — a fairness boundary the old design had no place to enforce.

It also had to be stateless, so nodes could be added, replaced, and deployed without ceremony, and it had to be regional, so tenants could be placed near their users rather than everyone sharing one location.

OPTIONS

You cannot stop the world to rebuild it

The hard part of a rewrite this size isn’t the target architecture — it’s that Confluence Cloud has to keep serving customers every day you’re building it. A flag-day cutover from monolith to new stack would put every tenant’s data and uptime on a single irreversible moment.

So the question became less “what should the new architecture be” and more “how do you run both architectures at once, and move traffic between them at a pace you can undo.”

THE CALL

Make the boundary a routing rule

Atlassian built two foundational services. cc-proxy, written with NginX and Lua, sits in front and routes traffic between the old monolith and the new stack — so which architecture serves a given request is a decision made per request, at the edge, not a property of the release. It handles around 30,000 requests per second, under 25 milliseconds, at a 99.99% success rate. Alongside it, cc-ledger was built from the ground up on asynchronous reactive streams — Spring WebFlux, DynamoDB Global Tables, and DAX caching — as a low-latency, high-throughput critical service.

The result is a multi-tenant estate spread over six regional tenant shards — US East and West, Ireland and Frankfurt, Singapore and Sydney — built from RDS PostgreSQL, EC2 web and worker nodes, SQS queues, and ElastiCache, with search and Smarts as separate platform services.

CONSEQUENCES

What Atlassian does and doesn't claim

It’s worth reading this account for what it withholds. Atlassian gives hard numbers for cc-proxy and describes the architecture in detail, but publishes no migration timeline, no customer-impact figures, no cost comparison, and no catalogue of what blew up along the way. The story is told from the architecture’s point of view, not the incident log’s.

What it does establish is the fix for the original defect: with per-tenant connection limits enforceable and the workers no longer competing for the web tier’s connections, the pool stops being a shared cliff that any one tenant can walk everybody off.

In a multi-tenant system, your real capacity limit is whichever resource your tenants share without a quota — find it and fence it, because your noisiest customer will find it for you.

Source — read the original

https://www.atlassian.com/engineering/scaling-rearchitecting-and-decomposing-confluence-cloud

A plain-language, AI-drafted and human-edited retelling of the article published on atlassian.com, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.

← All systems