THE MIGRATION LINE
INCIDENT // CLOUDFLARE-2017 MIGRATED

Cloudflare's edge ran a patched OpenSSL fork alongside a separate Go stack that existed only to handle TLS 1.3. It folded both into BoringSSL.

CLOUDFLARE · 2017 · TLS / SECURITY / CRYPTO / INFRA
System stress over time Breach at T+4
2 → 1 SSL stacks at the edge
Several months Work to migrate
Rebuilt from scratch OCSP support
BASELINE

One edge, two SSL stacks

Cloudflare terminated its edge SSL connections with software based on OpenSSL, running against an internal fork carrying special patches for features it needed. TLS 1.3, the newest version of the protocol, was handled somewhere else entirely: the OpenSSL-based software handed those TCP connections off to a separate system built on Cloudflare’s fork of Go’s crypto/tls library, written to do nothing but TLS 1.3.

As an experiment, the split was useful — TLS 1.3 could be rolled out to clients in relative safety without touching the main stack. As a permanent arrangement, it started to bite.

NEEDS

What the replacement actually had to do

The requirements were unglamorous. One stack should terminate every connection, TLS 1.3 included, with no duplicated logic, no added complexity, and no increase in latency.

It also had to keep the features Cloudflare had been maintaining custom OpenSSL patches for: a private key callback (which is how Keyless SSL works), an asynchronous session lookup callback for distributed session ID caches, equal-preference cipher grouping so clients could pick between ChaCha20-Poly1305 and AES-GCM, and a hook for inspecting and logging the ClientHello.

THE CALL

BoringSSL, Google's fork of the thing they already ran

Cloudflare moved the edge SSL termination stack to BoringSSL — Google’s crypto and SSL implementation, itself a fork of OpenSSL. BoringSSL had supported TLS 1.3 for a long time, having been one of the first open source implementations to do the work, so the separate Go system was no longer needed and all edge traffic converged on one stack.

The switch also carried modern cryptography backward. X25519 — the elliptic curve key exchange proposed by Daniel J. Bernstein and now the default in browsers like Chrome — had been available only on TLS 1.3 connections. With BoringSSL, Cloudflare could negotiate it on TLS 1.2 and earlier too. So could RSA-PSS, the padding scheme TLS 1.3 adopted to replace the fragile PKCS1-v1.5 scheme that Bleichenbacher broke in 1998 with an oracle attack that can recover a secret-key operation in as few as 15,000 messages.

THE COST

What broke on the way, and what they accepted

BoringSSL’s maintainers had been aggressively deleting features nobody used. Mostly this went unnoticed — but OCSP got the axe, and Cloudflare relied on it heavily to offer OCSP stapling to every client automatically. The team spent a few weeks building a replacement, and by their own account ended up with a far more reliable OCSP pipeline than they had before.

The legacy draft ChaCha20-Poly1305 ciphers were gone too — the pre-standard versions that some browsers shipped early and that turned out incompatible with the ciphers later ratified in RFC 7905. Cloudflare checked its metrics, found a significant percentage of clients still depended on them (older mobile devices with no AES hardware offloading and no software updates), and added the ciphers back into its own BoringSSL fork rather than drop those users. Then a test deployment showed NGINX instances starting up slower than before. Perf and flamegraphs pointed at BoringSSL’s base64 decoder, which had been changed to be constant-time and was several times slower than OpenSSL’s as a result.

A dependency swap is an audit you did not ask for: the features the new library refuses to carry tell you which of your patches were load-bearing, and the places it is slower tell you where you were being wasteful all along.

Source — read the original

https://blog.cloudflare.com/make-ssl-boring-again/

A plain-language, AI-drafted and human-edited retelling of the article published on blog.cloudflare.com, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.

← All systems