Failing traffic out of a bad AWS region used to take Netflix close to 50 minutes, mostly waiting for new instances to boot. Project Nimble keeps fully booted, invisible standby capacity running at all times, so a failover becomes flipping a switch instead of provisioning from zero.

Netflix serves more than 117 million customers who together watch 140 million hours of content a day, and one of its main defenses against a bad AWS region is failing traffic out of it fast. Before Nimble, that failover took close to 50 minutes: about 5 minutes just to decide to push the button, since the mutations involved could destabilize an otherwise healthy region; 3 to 5 minutes to calculate and provision the capacity a destination region would need; 25 minutes for new instances to fully boot, register with Eureka, apply Archaius configuration, and register with load balancers; 10 or more minutes to proxy traffic gradually through Zuul so new instances had time to warm up; and about 5 minutes for DNS TTLs to move the bulk of devices.

The team hit its goal and then some: failovers that used to take about 50 minutes now complete in 8. Rolling out the mechanism, and the software to orchestrate it, took a small team of two engineers about six months. Because it runs on capacity Netflix already provisions rather than new standby infrastructure, the approach is cost neutral. Any streaming-path service with no cross-regional dependencies gets it automatically, with no integration work required from that service’s own team.

Netflix first considered “pinning high” — running every autoscaling group permanently oversized so failover capacity was always sitting there. They rejected it: either every team’s autoscaling signal would need to become centralized and divorced from its normal operation, or every one of hundreds of individual scaling policies would need its own manual transformation. Worse, pinning risked quietly absorbing real performance regressions into the spare buffer instead of surfacing them.

The answer instead was “dark capacity”: shadow Auto Scaling Groups, tracked and cloned from production configuration using Netflix’s Edda and Spinnaker tooling, that boot fully but stay invisible. Because Netflix bakes AMIs rather than configuring instances after launch, a dark instance runs through its entire real startup procedure — connections, configuration, everything — while a library keeps it from registering “UP” with Eureka, so Netflix’s Ribbon routing never sends it traffic, and Atlas reporting is disabled so it produces no telltale metrics. An orchestrator built on AWS’s ASG detach-and-attach mechanism can then pluck an already-running instance out of its dark group and push it straight into a live service.

Nimble’s simplicity is also its limit: it works automatically only for services with no cross-regional dependencies, and it depends on Netflix correctly predicting how many dark instances each service needs, based on time of day and expected failover traffic. Get that prediction wrong, and an under-provisioned service could still be overwhelmed during a real evacuation — the post doesn’t report a case of that happening, but the dependency is real.

Netflix was already looking past region evacuation at the time of writing: using Nimble’s pre-warmed capacity as a possible option for emergency capacity boosts, or as a faster alternative to normal autoscaling — both framed as directions being explored, not shipped.

The fastest way to make a slow failover fast isn’t speeding up provisioning — it’s doing the provisioning continuously, in the dark, long before you ever need it.