One flaky network link inside a single availability zone took Slack down twice in a day, so Slack spent 18 months building a button that drains an AZ.

Slack runs a global, multi-regional edge network, but most of its core computational infrastructure lives in multiple availability zones inside a single region: us-east-1. Availability zones are isolated datacenters within a region. The cloud services Slack depends on — virtualization, storage, networking — are blast-radius limited so that they should not fail simultaneously across multiple AZs.

That is the whole bargain of multi-AZ hosting: the availability of the service across a region is supposed to be greater than the availability of any single AZ underneath it. Slack had built on that assumption.

At 11:45am PDT on 2021-06-30, Slack’s cloud provider had a network disruption in one of the availability zones in us-east-1. A network link connecting one AZ to several others carrying Slack servers began throwing intermittent faults. Connections between Slack servers went slow and degraded. Customers felt it.

At 12:33pm PDT the provider automatically pulled the link out of service, and full service returned. After a series of automated checks, the link went back in. At 5:22pm PDT the same link produced the same intermittent faults. At 5:31pm PDT the provider removed it permanently, and service was restored again.

Slack’s incident review found the obvious reading unsatisfying. A piece of hardware failed and errors were served until it was removed — fine. But the multi-AZ architecture was supposed to absorb exactly this. Why was one failed AZ visible to users at all?

Detecting failure in a distributed system is hard, and this failure was ambiguous rather than clean. A single Slack API request — loading messages in a channel, say — can fan out into hundreds of RPCs to service backends, every one of which must complete to return a correct response. Slack’s service frontends constantly try to detect and exclude failed backends, but they have to record some failures before they can exclude a server.

Because of that, the automated defenses never got a coherent picture to act on. Making it worse, key datastores — including Vitess, Slack’s main datastore — offer strongly consistent semantics, which requires a single backend for any given write. If a shard primary is unreachable from an application frontend, writes to that shard fail until the primary returns or a secondary is promoted.

Rather than try to automatically remediate gray failures, Slack chose to make the computers’ job easier by using human judgment. During the outage it was clear to responding engineers that one AZ was the problem — nearly every graph they had, aggregated by target AZ, told the same story. If they’d had a button saying “this AZ is bad; avoid it,” they would have smashed it. So they set out to build one.

The mechanism is siloing: a service is siloed if it only receives traffic from within its own AZ and only sends traffic to servers in its own AZ. Each service then behaves like N virtual services, one per AZ, and failures stay inside the AZ that produced them. The payoff is that draining an entire AZ requires no per-service signal — stop sending user requests into a siloed AZ and its internal services quiesce, because no new work arrives.

Slack had already spent years migrating its edge load balancers from HAProxy to Envoy, configured by Rotor, its in-house xDS control plane. AZ draining then fell out of two off-the-shelf Envoy features: weighted clusters and dynamic weight assignment via RTDS. Reweight an AZ’s target cluster to zero and Envoy finishes in-flight requests while sending every new one elsewhere.

Against those goals, the Envoy/xDS implementation lands well: propagation through the control plane takes on the order of seconds and load balancers apply new weights immediately; drains are graceful, so the load balancing layer abandons no queries; weights give gradual drains at 1% granularity; and the edge load balancers sit in entirely different regions with a regionally replicated control plane, resilient to the loss of any single AZ.

The 1% knob matters more than it sounds. Because a drain produces no errors, it becomes a generic mitigation — an operator can drain an AZ during an incident to see whether recovery follows, and undrain if it doesn’t, all before anyone understands the root cause. Slack’s 99.99% availability SLA leaves under an hour of total unavailability per year, so the tools have to work faster than the diagnosis does. Over about 1.5 years, Slack migrated its most critical user-facing services from a monolithic to a cell-based architecture on this foundation.

When your automation can’t tell what’s broken, the cheapest fix may be to reshape the system until a human can point at the broken part — and then make pointing at it a single, safe, reversible action.