Yelp's PaaSTA platform moved from Apache Mesos to Kubernetes so it could run stateful workloads. Flink was the first big tenant — and a custom Kubernetes operator cut cluster spin-up from around 30 minutes to under 2.
A Flink cluster used to take 30 minutes and a human. On Kubernetes it takes one YAML file and under two minutes.
Yelp processes terabytes of streaming data a day across hundreds and hundreds of Flink jobs. The previous home for those jobs was a platform called Cascade, which ran Flink on AWS ElasticMapReduce with YARN. It worked, but the operational shape was heavy: spinning up a new Flink cluster took around 30 minutes, and deployments and scaling needed manual human operator intervention.
The bigger constraint was version freedom. Because Flink ran on AWS EMR, Flink version upgrades were tied to EMR’s support cycles — Yelp couldn’t move faster than the managed service allowed. On top of that sat the operational complexity of running Puppet inside Docker containers to configure everything.
AFTER
One YAML file, under two minutes, no operator paged
PaaSTA — Yelp’s Platform As A Service — moved from Apache Mesos to Kubernetes, a shift that let it support more complex stateful workloads, and Flink was the first major application migrated onto it. On the new platform, provisioning a Flink cluster is a single YAML configuration file, deployments happen automatically via Jenkins on git commits, and the same unified PaaSTA commands work identically across every Flink cluster.
The headline number: cluster spin-up dropped from around 30 minutes to under 2 minutes. And because Flink now runs in Yelp’s own customized Docker containers rather than on EMR, version upgrades are no longer gated by a managed service’s support cycle.
THE BRIDGE
An operator that watches the clusters so people don't have to
The core of the migration is a custom Kubernetes operator that automatically watches over the deployment and the lifecycle of Flink clusters. Configuration lives in Kubernetes Custom Resources; TaskManagers are scheduled via Deployments; the JobManager is discovered through a Service; and the web dashboard is exposed through an Ingress. The work a human used to do by hand is now encoded in software that runs continuously.
Yelp deliberately did not rewrite everything at once. The legacy Supervisor component from Cascade — responsible for triggering savepoints — was retained, now running as a Kubernetes Job. Keeping it reduced the development scope of the migration and gave users a smoother path off Cascade.
WHAT IT COST
Kept scaffolding, and a roadmap still open
The trade Yelp accepted was carrying a piece of the old world forward: the retained Cascade Supervisor is legacy code living inside the new platform, kept to make the migration smaller and the user transition gentler rather than because it’s the long-term design. And several capabilities are explicitly future work, not shipped state — Apache Beam support for Python stream processing, auto-scaling, per-job cost reporting, and defining complex pipelines from a single configuration file.
Migrating a platform is easier when you let the new system absorb the old one’s leftovers as-is — a retained legacy component can be the price of a gentler cutover, not a failure of the redesign.
The same PaaSTA Mesos-to-Kubernetes shift, seen through its two hardest stateful tenants — Flink here, Cassandra next door.
A plain-language, AI-drafted and human-edited retelling of the article published on engineeringblog.yelp.com, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.