THE MIGRATION LINE
INCIDENT // CANVA-2024 SHIPPED

Canva counts billions of content usages a month to pay creators. Counting them one database round trip at a time on MySQL was O(N) and took over a day — so they moved the counting into an OLAP warehouse.

CANVA · 2024 · DATABASE / OLAP / SNOWFLAKE / ELT
System stress over time Breach at T+3
billions of usages / month Scale
doubles every 18 months Growth
>1 day → minutes Aggregation
>1 day → <1 hour Pipeline latency
SITUATION

Counting billions of usages, one at a time

Canva launched its Creators program three years ago, and usage of creator content has doubled every 18 months. Today Canva pays creators based on billions of content usages each month — not just templates, but images, videos, and more. Getting that count wrong isn’t a rounding error; it’s someone’s income, so accuracy is the top requirement, alongside scaling with exponential growth and staying operable as volume climbs.

The first architecture used the stack the team knew best: MySQL, with the counting split across worker services. A deduplication worker scanned a table of usages and tagged each record with an event type; an aggregation worker then scanned that table and incremented counters in yet another table.

COMPLICATION

Every record cost a round trip, and storage kept doubling

The deduplication scan was single-threaded and sequential, walking records with a pointer. That was easy to reason about during incidents, but it didn’t scale: processing each usage record took at least one database round trip — one read of the event, one write to increment a counter — so a full scan was O(N) queries for N records. Batching helped constant factors but not the big-O; O(N/C) is still O(N). Multi-threading would have added complexity without fixing the fundamental cost.

Storage was the other wall. MySQL RDS doesn’t horizontally scale on its own, so the team doubled the instance size every time they needed more room — roughly every 8 to 10 months. In one stretch, free storage fell by almost 500 GB, half the total, within six months, and the instance grew to several terabytes.

QUESTION

How do you count N things without N round trips?

The obvious first move was to fix storage: the team shifted the raw usage events in the collection stage over to DynamoDB, which eased the growth pressure. The plan was to move the rest of the pipeline there too.

They stopped and reconsidered. Moving everything to DynamoDB would solve storage scalability, but the real bottleneck — the per-record database round trip — would still be there, because the counting logic would still be walking records one at a time. So the question sharpened: not “where do we store the data,” but “how do we count it without touching each record individually?”

ANSWER

Do the counting inside an OLAP warehouse

The answer was to stop counting incrementally and instead compute the whole month end to end, in a database built for exactly that. Canva moved from OLTP databases, optimized for transactions, to an OLAP database optimized for large analytical queries — Snowflake, which they already ran as their primary warehouse. And they moved from scheduled worker services to an ELT pipeline: Extract and Load the usage data into the warehouse using the data platform’s replication pipeline, then Transform it with DBT jobs written as SQL queries, materializing intermediate results as SQL views.

The deduplication and aggregation that used to be thousands of lines of worker code became SQL — filtering duplicates, then summing counts with GROUP BY. Because OLAP separates storage from compute, the aggregation runs mostly in memory on independently scalable compute, so Canva can now aggregate billions of usage records in a few minutes instead of over a day, and total pipeline latency dropped from over a day to under an hour.

TAKEAWAY

Simpler, with new seams to mind

The move wasn’t free of trade-offs. The DBT transformation service runs standalone with its own release schedule, so schema changes have to stay compatible across two deploy cadences, and it carries its own CI/CD cost. Unloading results back out of Snowflake into the low-latency operational databases needed a scheduled worker using S3 and SQS, with careful rate-limiting so the ingestion didn’t throttle the target RDS’s CPU. Canva is clear-eyed that they added infrastructure complexity even as they cut code.

But the ledger is lopsided in their favor: over 50% less data stored, thousands of lines of dedup and aggregation code deleted, and logic that they describe as “surprisingly simpler” once rewritten in SQL. Their own summary of the lesson is that simplicity drives reliability — reducing code and data complexity beat piling on cleverer fixes.

When a count won’t scale, look at the access pattern before the storage engine — moving the computation to where the data already lives beat moving the data to faster storage.

Source — read the original

https://www.canva.dev/blog/engineering/scaling-to-count-billions/

A plain-language, AI-drafted and human-edited retelling of the article published on canva.dev, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.

← All systems