THE MIGRATION LINE
INCIDENT // DISCORD-2026RESOLVED

Discord moved voice onto Cloudflare's 300+ city edge, then spent weeks hunting European ping spikes that turned out to start inside its own event loop.

DISCORD·2026·VOICE / EDGE / RUST / NETWORKING·✓ Sourced — discord.com
System stress over timeBreach at T+5
~30 → 300+ citiesVoice PoP reach
80%+Traffic on Cloudflare
down 34%Frankfurt ping
above 0.5sPeak client ping spikes
BASELINE

Thirty cities, and the ones that missed out

For most of Discord’s history, the closest voice server it could put you on lived in one of about 30 cities worldwide — wherever the major cloud providers had data centers. Fine in the Bay Area or Frankfurt. Less fine in Reykjavik or Auckland, where hyperscaler coverage was thin, and every extra kilometer of network distance is latency on every packet.

So Discord started moving voice and video onto Cloudflare’s edge network, which runs points of presence in over 300 cities. The first rollouts taught them the premise was not as simple as “closer PoP, better call.” In Iceland, local calls improved (ping down 9%, packet loss down 11%) but mixed-region calls got worse: ping for non-Iceland users on Iceland-hosted calls jumped 2.7x, because Discord assigns one SFU host to an entire call and everyone connects to it. In Rotterdam, users on the French ISP Orange saw latency above one second at peak, traced to a saturated Telia-to-Orange transit handoff. That one was reverted after about ten days. After those, rollouts were paced by peering analysis, region by region.

SYMPTOM

Bucharest and Stockholm start slipping

Late 2025: Discord migrated Bucharest and Stockholm onto the new Rust-based SFU on Cloudflare. Both regions’ metrics began to slide. Ping climbed during peak hours. Voice quality degraded over the following weeks. By February, client-side ping latencies were spiking above half a second, and reports of ping spikes and robotic audio were arriving through support channels and Reddit.

Discord shifted some European traffic back off Cloudflare and started digging. The early theories were the ones you would expect: a routing anomaly, a peering issue, a DDoS hitting their ports. None of them held up.

PROBE

A packet's own alibi

The breakthrough came from a passive probe written by Cloudflare’s performance team. Discord’s voice health-checks carry a fixed payload signature that is identifiable on the wire without inspecting connection state, so Cloudflare built an eBPF program hooked into their host’s network stack that timestamped every Discord health-check passing through. Comparing arrival and departure timestamps at the NIC gave on-metal processing time for every health-check on every host — without touching the traffic.

The probe segmented the path and cleared it. Physical NIC to virtual NIC: clean. Virtual NIC back to physical: clean. The time was being spent inside the VM. Which meant inside Discord’s own application. Given direct access to a container in an affected region, Cloudflare’s team observed futex stalls of up to 860 milliseconds inside the process within minutes. Per-thread sampling showed one thread pegged at 100% of a single core during every spike window, with socket receive buffers growing at the same time.

REVEAL

The event loop that would not stop listening

Discord’s SFU runs eight worker threads, each with its own UDP socket, a single-threaded Tokio runtime, and a select! event loop. Two branches matter: a recv future that pulls packets off the socket, and a flush timer that pushes outgoing packets back out.

RESOLUTION

Nine milliseconds, and a thread that moved house

The fix was a budget. If nine milliseconds had passed since the last send() flush, recv was disabled for that poll of the select!, forcing the flush timer to run. It shipped in early March. The p99 latency dropped right away — and the spikes still showed up at peak hours.

The residual jitter came from Cloudflare’s side. Their VMs exposed a single virtio-net queue each, so all receive softirq processing for a VM landed on one vCPU; when Discord’s worker threads got scheduled onto that same vCPU, softirq processing preempted them mid-execution. Flame graphs taken after the recv-and-send fix showed net_rx_action and local_bh_enable frames lining up exactly with the leftover spikes.

A year in, more than 80% of Discord’s voice and video traffic runs on Cloudflare, and more than 70% of migrated regions show net year-over-year quality improvements — packet loss down 20-60% across several European and Latin American regions, Frankfurt’s ping down 34%. The regions that motivated the whole project, Reykjavik and Auckland, are still ahead of them.

Real-time media on shared hardware surfaces scheduling and queue-contention behavior that HTTP and static workloads never feel — and when the symptom is latency, “it’s the network” is the theory most likely to be wrong.

Source — read the original

A plain-language, AI-drafted and human-edited retelling of the article published on discord.com, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.

← All systems