Spotify rebuilt the pipeline that carries every event it collects — while it was carrying nearly 8 million events per second — and then spent the long tail waiting for 600+ legacy event types to catch up.

Every time someone plays a track, skips one, or opens a playlist, Spotify collects an event. At peak the system was carrying nearly 8 million of them per second — up from 1.5 million — and ingesting close to 70 terabytes a day after compression.

The pipeline underneath had accumulated its own archaeology. Events moved as tab-separated values, with schemas parsed and converted to Avro by a Python library dating to 2007. Processing ran on Hadoop jobs. Events were written to files on disk before being forwarded on, which made the services stateful — and a stateful service can’t be casually rescheduled, which is why the system couldn’t autoscale and couldn’t move to Kubernetes. It stayed tightly coupled to legacy perimeter servers, because every client event had to route through them.

Stop losing mobile events. Shrink the instrumentation-to-insight cycle. Let the fleet autoscale instead of babysitting long-lived, oversized Hadoop clusters. And support deduplication over a window long enough to be honest about how phones behave — the old system looked back within a small window of hours, which is nowhere near how long a device can stay offline.

One requirement was really a permission: the freedom to break backwards compatibility. As long as the old data model had to be preserved, the redesign couldn’t happen. Spotify decided compatibility was the constraint worth spending.

There were 600-plus legacy event types, owned across the company. Demanding that every owner reinstrument before anything shipped would have serialized the whole migration behind its slowest team.

So the team offered producers two paths. Redesign your instrumentation properly against the new model — or keep emitting legacy events and let a transformation pipeline convert them into the new shape. The transformation was explicitly imperfect: some fields would be missing or inaccurate in conversion. That was acceptable precisely because legacy traffic was expected to decay as users upgraded their apps.

The new system inverted the old resilience model. Instead of buffering events to disk and forwarding them, Spotify built a highly available receiver service and pushed resilience out to the clients, which now resend with predefined retry policies. A resend might arrive immediately, within minutes, or several days later — which is exactly why the deduplication window had to stretch from hours to multiple weeks, backed by event message identifiers and lookup indices.

They also refused to rebuild it in the dark. Alpha users with non-critical event types onboarded early, a prototype went to production before the design was finished, and the control and data plane interfaces were stabilized before the mass migration began. Onboarding real internal users early, Spotify says, was an important forcing function that kept quality and operational maturity high.

Decoupling the infrastructure rebuild from the 600-plus event migration is what kept the wall-clock time survivable — the platform could be finished without waiting for every producer. But the tail is long by construction. Mobile app updates take several months to reach a high percentage of users, and embedded devices like TVs and speakers are unlikely to upgrade at all, which means the legacy path has to be supported more or less permanently.

And some of the original problem turns out to be unsolvable rather than unsolved. As the team puts it, it’s actually impossible to tell whether an event was lost in transport or whether a user simply used Spotify and then dropped their phone in the ocean. Retries, longer dedup windows, and message identifiers narrow the gap. They don’t close it, and the design stopped pretending they would.

Backwards compatibility is not free maintenance — it’s a mortgage on the redesign you can’t do until you stop paying it.