Every engineering pillar at Instacart had its own data tools, its own event framework, and its own Airflow. Ground Truth collapsed them into one stack in six months.

Instacart runs on data — decisions, product insight, machine learning. Over the years it accumulated a tremendous amount of it, and over those same years the tools for handling it multiplied. Before 2023, each engineering pillar had picked its own systems: several different event frameworks were in use simultaneously, including Snowplow, Segment, and an internal one called Mongoose. Orchestration was worse. There were a huge number of scheduling and orchestration systems, including at least a dozen separate Airflow instances.

None of this was anyone’s mistake. It is what happens when a business, a product, and an engineering organization all grow at once and every team solves its data problem the day it appears.

As the complexity of Instacart’s systems exploded, deriving value from data got harder rather than easier — in most domains, increasingly so. Teams stayed productive by finding creative ways around the obstacles, which is exactly the behavior that hides the problem from management.

So here is the engineering question. You have a dozen Airflow instances, three event frameworks, a BI tool people avoid, and thousands of production tables whose contents nobody can vouch for. You cannot freeze the business while you fix it. What do you standardize, and in what order, so that the fix compounds instead of adding a fourteenth system to the pile?

Instacart’s answer was to treat it as one program with one name, not a dozen team-level cleanups. In H1 2023 they launched Ground Truth: a few dozen engineers pulled in alongside the Data Infrastructure team to overhaul the entire lifecycle of data at once — instrumentation, extraction, transformation, and consumption.

The rebuilt stack is a chain, each link replacing several. Production tables now flow continuously into Snowflake via change data capture, built on Debezium and Kafka, across thousands of tables. The three competing event frameworks collapsed into one, Snowgoose, which handles billions of events — hundreds of terabytes a day across thousands of event types. Every scheduling and orchestration system, the dozen-plus Airflow instances included, consolidated onto a single dbt plus Airflow deployment carrying thousands of data models written by hundreds of authors across dozens of teams.

Around that spine they added governance: Immuta for fine-grained access policies, an internal Declarative Data Checks platform now running hundreds of critical validation checks, an Annotations framework used to document and enforce governance on tens of thousands of columns across dozens of databases, and Amundsen as the catalog, where datasets get certified bronze, silver, or gold so consumers know what they are picking up. Mode Analytics replaced the old BI tool and passed 8,000 dashboards in six months, cutting failed dashboard runs by 7x and Snowflake costs by 3x compared to its predecessor.

Here is the part that makes this more than a tooling story. For a decade, companies at Instacart’s scale answered data complexity with a large central data engineering team — often 100 or more engineers — matrixed into product organizations. That made sense when data tools were arcane and demanded specialists. But it scales badly, and it introduces a toss-it-over-the-fence boundary that separates domain expertise from data expertise.

Because the modern stack requires substantially less specialized expertise to operate, Instacart argues the centralized model is no longer necessary — and Ground Truth itself became the training program. A few dozen engineers from across the company spent a half embedded in data work and went back to their home teams as local data ambassadors. At the start of H2 2023, Instacart announced the transition to a decentralized model, with the expectation that all product engineering teams will be self-serve for data engineering work from then on.

Platform consolidation is rarely just about the platform — the real prize is the organizational shape it makes possible, and the tools are only the precondition for the reorg you actually wanted.