Zuul is Netflix's front door for every user's traffic. The original ran one thread per connection; at 83 million members with multiple devices each, that thread cost was the whole problem.

Zuul is Netflix’s cloud gateway: the front door for traffic from every Netflix user in the world, handling routing, deep insight into service health, protection from attacks, and shifting traffic to other AWS regions when one is in trouble. The original Zuul was built on the Servlet framework — a blocking, multithreaded system that processes requests with one thread per connection. An I/O operation gets handed to a worker thread from a pool, and the request thread blocks until that worker finishes and reports back.

That model worked well on modern multi-core AWS instances handling “100’s of concurrent connections” each.

The blocking model’s weakness showed up when things went wrong. If backend latency increased, or devices started retrying failed requests, the count of active connections and threads climbed. Past a point, nodes would “get into trouble and can go into a death spiral where backed up threads spike server loads and overwhelm the cluster.” Netflix’s answer at the time was defensive: throttling mechanisms and libraries like Hystrix, built to keep the blocking system stable when connections piled up.

The underlying constraint was architectural, not incidental. In a blocking system, the cost of a connection is a thread — with heavy memory and system overhead attached to every single one. Netflix was also trying to solve a scale problem the blocking model wasn’t built for: giving each of more than 83 million members, often with multiple connected devices, a persistent connection back to Netflix’s infrastructure, so features could push updates instead of every device polling for them.

Zuul 2 rebuilt the gateway on Netty, an asynchronous, non-blocking framework. Instead of one thread per connection, async systems run roughly one thread per CPU core, handling all requests and responses through events and callbacks — so the cost of a connection drops to, in Netflix’s words, “a file descriptor, and the addition of a listener” rather than a thread.

Building it was harder than expected, because Netflix’s own ecosystem assumed blocking everywhere. Core networking libraries relied on thread-local variables to carry request context, and thread-locals don’t work once multiple requests share the same thread. Netflix’s approach was to convert Zuul’s filter logic to run asynchronously first, using RxJava, so the exact same filter code could execute on both the old blocking system and the new Netty one during the transition — and used the open-source tool Reactive-Audit to find blocking code hiding inside libraries.

After running in production for several months, the results didn’t match either extreme hypothesis. The Zuul cluster fronting the API service — the most CPU-heavy cluster, doing metrics calculations, logging, decryption, and compression on every request — saw no efficiency gain at all from switching to async; it was essentially equivalent to the blocking version in capacity and CPU. The Logging cluster, which does far less per-request work, saw a real gain: about a 25% increase in throughput alongside a 25% reduction in CPU utilization. The pattern Netflix drew from it: the less CPU-bound the work, the more efficiency async buys.

The scaling win, though, was the one they actually wanted: cheap persistent connections to devices at Netflix’s member scale, and better resilience to retry storms and backend latency than the blocking model gave. The honest cost is operational — in Netflix’s own words, “a system that is much more complex to debug, code, and test,” running inside an ecosystem still built on blocking assumptions.

Async didn’t make Zuul faster at CPU-heavy work — it made holding millions of connections open cheap; know which of those two problems you actually have before you rewrite a gateway to fix it.