Splitting the monolith bought DoorDash developer velocity and independent deploys — and a new class of failure where one slow database took the app down for three hours.

In 2020, DoorDash — the largest food delivery marketplace in the US, with over 30 million users by 2022 — migrated from a Python 2 monolith to a microservices architecture. The reasons were the usual good ones: smaller teams that could deploy independently, the freedom to use different tech stacks for different classes of service, and room to scale the engineering organization itself.

It worked. It also introduced new failures that did not exist with the monolithic architecture.

In May 2022, DoorDash performed routine database maintenance. That temporarily increased read/write latency for one service. Because that service got slower, upstream services that depended on it got slower too. Because those calls now took longer, they started timing out, and the error rate climbed. Because the error rate climbed, it tripped a circuit breaker — one that was misconfigured. The result was an outage in the app that lasted three hours.

Nothing in that chain was exotic. Every link was a service behaving exactly as designed, reacting to a neighbor that had gotten slightly slower.

DoorDash’s engineers catalogued how trouble travels through a distributed system. It spreads vertically as a cascading failure: a problem in one component, like a database, propagates to every service that depends on it, even services that are otherwise perfectly healthy. It spreads through retry storms: retries are a sensible response to the random, transient failures that distributed systems produce constantly, but while a downstream service is already degraded, retries amplify the work and degrade it further. And it spreads horizontally as a death spiral: when one node of a three-node service dies, its traffic lands on the other two, driving up their CPU and memory until one of them dies of resource saturation — and then all the traffic lands on the last machine standing.

The countermeasures each attack one of those loops. Load shedding lets a degraded service drop the requests engineers have marked unimportant — a logging call matters less than a user action — to maximize goodput during periods of stress, which is preferable to rate limiting every request blindly. DoorDash instrumented its servers with an adaptive concurrency limit using Netflix’s concurrency-limit library: when a machine takes longer to respond, the library lowers the maximum number of concurrent requests so each one gets more compute, and it can read request priorities from headers.

Auto-scaling is the counterintuitive one. Adding machines to a struggling service looks obvious, but DoorDash recommends teams do not use reactive auto-scaling, because it can temporarily reduce cluster capacity and make things worse: new machines need time to warm up — filling caches, compiling code, running startup tasks — and that warm-up steals resources from the warmed-up nodes still serving requests. They favor predictive auto-scaling instead, expanding the cluster based on expected traffic levels through the day, before the load arrives.

None of these come free. Load shedding is very difficult to configure and properly test, and a misconfigured load shedder can increase latency and cause outages on its own; the right parameters differ by workload, resources, and time of day, and auto-scaling changes the utilization level at which shedding should begin. The circuit breaker’s error threshold is extremely difficult to determine — 50% is a common rule of thumb online, but the right number depends entirely on the downstream service and its availability requirements. The three-hour outage in May 2022 was tripped by a circuit breaker that was misconfigured.

Breaking up a monolith doesn’t remove failure, it redistributes it — and the resilience patterns you add to catch the new failures are themselves systems that will page you when they’re tuned wrong.