Cloudflare evaluated forking NGINX, adopting another proxy, or starting clean — and built Pingora in Rust, cutting CPU by 70% and memory by 67% at over a trillion requests a day.

Cloudflare’s network sits between HTTP clients and the servers of the Internet, and the proxy service doing that job powers its CDN, Workers fetch, Tunnel, Stream, R2 and many other products. For years, that proxy was NGINX with OpenResty and Lua on top. Cloudflare’s own verdict: it was great for many years, but at their scale they had outgrown it.

The trouble started in NGINX’s worker-process architecture. Each request can only be served by a single worker, which leaves load unbalanced across CPU cores. Because of that request-to-process pinning, a request doing CPU-heavy or blocking I/O work slows down the other requests stuck on the same worker.

Cloudflare’s requirements weren’t abstract. They needed to fix the fundamental issue — the worker/process model — rather than keep working around it, because doing so would resolve the load imbalance, the blocking-request interference and the connection reuse problem all at once.

They needed to add functionality NGINX doesn’t allow: when retrying or failing over a request, sending it to a different origin server with a different set of request headers. They needed memory safety, since NGINX is purely C and it’s easy to get into memory safety issues even for experienced engineers. They wanted static typing, which they kept missing when Lua business logic got complicated. And they wanted a community that develops in the open — NGINX’s is not very active, and development tends to happen behind closed doors.

For several years, Cloudflare evaluated the same three choices each quarter. Keep investing in NGINX, possibly forking it to tailor it entirely to their needs — they had the expertise, but the architecture limitations meant significant effort to rebuild it in a way that fully supported them.

Migrate to another third-party proxy codebase. There are good projects, Envoy among them, but this path means the same cycle may repeat in a few years — you inherit someone else’s architecture and eventually outgrow that instead. Or start from a clean slate, building an in-house platform and framework, which requires the most upfront engineering investment of the three. There was no obvious formula. For several years they continued augmenting NGINX, the path of least resistance, until the return on investment of building their own seemed worth it.

They chose Rust, because it can do what C does in a memory safe way without compromising performance. They chose to write their own HTTP library rather than use an off-the-shelf one like hyper — Cloudflare handles the whole Internet’s traffic, including plenty of bizarre and non-RFC-compliant behavior. HTTP status codes are defined as three digits generally expected between 100 and 599, but many servers use codes between 599 and 999, and that was only one of many noncompliant cases they had to support.

In production, Pingora serves over a trillion requests a day. Overall traffic shows a 5ms reduction on median TTFB and 80ms at the 95th percentile — and Cloudflare is explicit that this is not because the code runs faster, since even the old service handled requests in the sub-millisecond range. The savings come from sharing connections across threads. Pingora makes only a third as many new connections per second as the old service; for one major customer, the connection reuse ratio rose from 87.1% to 99.92%, cutting new connections to their origins by 160x. Cloudflare’s arithmetic: 434 years of handshake time saved every day.

It uses about 70% less CPU and 67% less memory at the same traffic load, partly from Rust versus Lua, partly from architecture — in NGINX/OpenResty, Lua reading an HTTP header must copy it out of a C struct into a new Lua string and later garbage-collect it, while Pingora just accesses the string directly. HTTP/2 upstream support landed without major hurdles, which let them offer gRPC soon after; the same work in NGINX would have required significantly more effort and might not have materialized.

The cost is the one they named going in: the most upfront investment of the three options, plus an HTTP library of their own to maintain forever. And they don’t claim to have escaped the cycle. Since Pingora’s inception they’ve served a few hundred trillion requests without a crash caused by their service code — crashes are rare enough that when one happens they usually find an unrelated cause, once a kernel bug, once faulty hardware. But as Cloudflare puts it, Pingora is their latest attempt at rewriting their system, and it won’t be their last.

When scaling out makes a metric worse instead of better, you’re not short on capacity — you’re looking at an architectural assumption, and no amount of tuning will move it.