Discord outgrew Derived, its five-year-old in-house orchestrator, and picked Dagster plus dbt against four hard criteria.

For about five years, Discord’s data analytics ran on Derived, an orchestration system built in-house. It was engineered when the user base was smaller and the data volume more manageable, and by Discord’s own account it played its part well.

What changed was not the data so much as the people. As the data organization grew, the requirements shifted from “software engineers can run this” to “data engineers and data scientists need to serve themselves.” Derived was built for the first world, not the second.

Before shopping, the team wrote down what actually mattered. Declarative automation, because self-service depended on it and users were already accustomed to that kind of flexibility. A modern UI that acted as a single pane of glass for data engineers and data scientists, from observability all the way to operations.

Reliability and scalability, which for Discord meant a concrete filter: any serious contender had to run orchestration workloads on Kubernetes. And integration with existing tooling, measured by how quickly existing Airflow jobs, CI/CD scaffolds, and data quality solutions could move over without too much disruption.

On the modeling side the search was short. dbt, Coalesce, and SQLMesh were all in the frame, but the breadth of functionality around dbt made it, in Discord’s words, a straightforward pick.

Orchestration took longer. Since Airflow was open-sourced in 2015, the space had filled up with Argo, Prefect, Dagster, Kestra, Mage, and others. The team spent extra time here precisely because the pain points with Derived were pain points their customers felt too.

Dagster was less battle-proven than Airflow, and Discord says so plainly. It won anyway because it hit all four criteria: out-of-the-box deployment and execution on Kubernetes, built-in declarative automation, a UI that let producers and consumers see the state of their data assets, and an Airflow integration plus Python APIs that made moving existing jobs less of a burden.

Two things tipped it further. Running Dagster locally turned out to be straightforward, which was never a requirement but let pilot testers feel out the API in a mock environment. And Dagster’s willingness to work with Discord and build new functionality for their scale gave the team confidence to break ground.

The bet came with real bills. dbt did not support parallelism well: for incremental models it stages data in a temporary table before merging, so running multiple instances of dbt run against the same model raced to delete that table. Discord adjusted dbt’s temporary-data logic to run partitions of the same asset in parallel. Backfilling partition-by-partition also went badly with BigQuery, producing extremely lengthy backfills, so they pushed an open-source commit with the Dagster team to configure how many partitions an asset can backfill at once. Atomicity — keeping an asset’s code version consistent across partitions, even mid-backfill — was not supported out of the box either, and had to be built from Dagster’s GraphQL interface plus sensors and jobs.

What they bought with that: a system, internally called Transformation 2.0, that now powers over 2000 dbt tables covered by more than 12,000 dbt tests, with roughly 4000 materializations triggered on a typical day. “Why isn’t my data asset updating?” became a self-serve, at-a-glance question. Quality checks can warn or block downstream runs, and table owners get told about failures through a Discord app. Petabytes of data churned through the move off Derived.

When your orchestrator’s real users stop being the engineers who built it, “can they answer their own questions without you” becomes the requirement that outranks maturity.