Instead of asking every team to rewrite their workflows, Pinterest built a scheduler that could speak the old system's language until it didn't need to anymore.
Pinterest had already decided to leave its legacy in-house workflow system, Pinball, for a new platform called Spinner, built on a custom branch of Apache Airflow 1.10. The team’s requirements for the migration were strict: minimal user code changes, uninterrupted execution of production workflows while being migrated, and a hard date to finish deprecating Pinball.
Instead of asking teams to rewrite workflows, Pinterest built a migration layer directly into the Airflow scheduler that translated a legacy Pinball workflow definition into a native Airflow DAG on the fly, during DAG file parsing. Every job in the legacy workflow became a wrapper operator that launched a Kubernetes pod running the old system’s own execution image — mimicking Pinball’s exact runtime environment so translated tasks behaved identically. The result: a self-service migration tool where a user’s active involvement dropped to selecting their workflow, scheduling it on Spinner, and verifying the output — a process measured in minutes instead of the hours a manual rewrite would have taken.
The translation layer used a custom PinterestDagBag that parsed DAGs from a migration metadata file rather than a Python file, retrieving the legacy workflow’s specification from a “Token Fetcher” container that hosted Pinball’s own parser. Because Airflow’s native DAG model didn’t support dynamically generated layouts — a requirement carried over from Pinball, where a workflow’s shape could depend on external service state — the team built an entirely new feature, DynamicDAG, computing and snapshotting a workflow’s layout at the moment each run was scheduled instead of at file-parse time.
The team’s own account is candid that migrations don’t move evenly: per the Pareto principle, roughly 20% of workflows accounted for 80% of the migration’s visible progress, showing up as two sharp spikes in an otherwise gradual completion chart, each spike corresponding to a single team finishing their migration. A dedicated CI/CD pipeline validated every new build of the legacy execution image through a canary suite of workflows before it reached production, specifically to catch a bad import in Pinterest’s shared monorepo before it could break every migrated task depending on it.
A migration doesn’t have to ask every user to do the work — it can be the platform’s job to speak the old system’s language for as long as it takes.
Source — read the original
A plain-language, AI-drafted and human-edited retelling of the article published on medium.com, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.