THE MIGRATION LINE
INCIDENT // FIGMA-2024MIGRATED

Figma's teams were burning engineering hours working around ECS, so the infrastructure team moved most core services to EKS in under 12 months by refusing to change anything else.

FIGMA·2024·INFRA / KUBERNETES / CONTAINERS / AWS·✓ Sourced — figma.com
System stress over timeBreach at T+3
Q1 2023 → Jan 2024Timeline
3, all serving trafficProduction clusters
1/3 of requestsCoreDNS incident blast radius
BASELINE

Containerized, orchestrated, and quietly stuck

By early 2023, Figma had already done the hard part: every service ran in a container, orchestrated by AWS Elastic Container Service. ECS is a good off-the-shelf way to run containerized workloads, and it had worked. Figma also isn’t a microservices company and doesn’t plan to become one — a handful of powerful core services provide the modularization, and new products are usually built by adding logic inside them rather than spinning up new ones.

That mattered, because it changed the size of the question. Nobody was proposing to migrate thousands of services. The team was asking whether the next generation of Figma’s compute platform could be built on ECS at all, or whether they were iterating toward a local maximum.

IMPACT

The cost of working around your orchestrator

The bill showed up as engineering hours. Service-owning teams with complex needs kept building workarounds. The worst example was etcd, a strongly consistent consensus data store: ECS has no StatefulSets, the Kubernetes primitive that gives pods persistent identity, so Figma wrapped custom code into the etcd container’s startup to dynamically update cluster membership. That code was fragile and hard to maintain. On Kubernetes, StatefulSets are simply how you run etcd.

The paper cuts accumulated. Teams wanting open source software like Temporal couldn’t use Helm charts and would have had to hand-port every service into Terraform. Gracefully draining one misbehaving EC2 machine was awkward on ECS and trivial on EKS, where you cordon the node and let the API server move pods off while respecting shutdown routines. Nothing in the fleet auto-scaled, so Figma paid to keep services provisioned for peak load through nights and weekends when traffic is far lower. And registering or removing NLB targets took several minutes, which slowed emergency deploys and pushed up mean time to remediation during incidents.

CAUSE

Outside the ecosystem, every feature is a build

Each of those problems had a solution. The problem was where the solutions lived.

THE FIX

Swap the orchestrator. Change nothing else.

The decision was EKS, and the discipline was scope. The safest scoping for a major migration limits the change to the core system you’re swapping out and keeps the abstraction identical for everyone using the platform: same deploys, same tooling, same way services run. EKS functionality is largely a superset of ECS functionality, so this was mostly achievable without extra work.

Two tempting improvements were kept out. An expensive, overly complex logging pipeline that routed everything through CloudWatch and a transforming lambda was left alone rather than risk porting its logic into a Vector sidecar mid-migration. Pod-level autoscaling was deferred for the same reason. Both became fast-follows, shipped alongside the remaining service migrations.

ROLLOUT

Load test, then shift DNS a slice at a time

They plotted the plan through Q1 2023, built a staging environment and migrated a single service in Q2, spent Q3 productionalizing and load testing, and cut traffic over through Q4 and into January. Load testing meant running a Hello World service at the pod count of Figma’s largest services, which surfaced sizing problems in the supporting platform components — Kyverno, their cluster security assertion tool, will slow down new pod startup if it isn’t sized for the fleet.

Traffic moved via weighted DNS entries, shifting incrementally from each ECS service to its EKS equivalent, with the same dial available to revert. They also put a real service on the new stack before the staging environment was even finished, which de-risked the end-to-end story faster than staging alone could have.

AFTERMATH

A destroyed CoreDNS, and a third of an outage

By January 2024, a majority of Figma’s highest-priority services were on EKS — including the monolith holding core business logic and the service handling multiplayer. It was done with only minor incidents and little customer impact. Then an operator accidentally destroyed and recreated CoreDNS on one production cluster. Previously that would have been a full outage. Because of the three-cluster setup, impact was limited to a third of requests, and most downstream services retried and eventually succeeded.

The one real complaint came from their own engineers. The debugging and emergency-operations tooling wasn’t user friendly: three clusters meant typing cluster names into every command, and new fine-grained Kubernetes RBAC roles — more secure, properly least-privilege — meant users had to know which role they needed. The team paused other work and updated the tooling to infer the right cluster and role automatically, because nobody should be hunting for a role at 3 a.m. during an incident.

A migration finishes on time when the new system is allowed to be different in exactly one way — and every additional improvement you smuggle in is a second-order effect waiting to eat your quarter.

Source — read the original

A plain-language, AI-drafted and human-edited retelling of the article published on figma.com, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.

← All systems