Every one of DoorDash's data pipelines worked, yet data arrived late and cost too much. The culprit was the shape of the collection, not any single pipe.

DoorDash had a handful of data pipelines carrying events out of its legacy monolithic web application and into Snowflake, the main data warehouse. Each one had been built at a different time, in a different way. Each could process exactly one kind of event. Each made multiple hops through different messaging and queueing systems before the data finally landed.

Individually, they worked. The Dasher assignment team got its assignment data. The ML platform got its delivery events. Mobile errors reached the monitoring backend.

What the team actually observed was three things at once: high data latency, significant cost, and operational overhead. No single pipeline was the villain. Nothing had fallen over. The events arrived — just late, expensively, and with nobody able to see where the time went.

Meanwhile the volume kept climbing toward the billions of real-time events a day the business would soon generate, from services, from microservices, from mobile and web devices, headed for a growing list of destinations including third-party data services.

Trace a single event and the pattern shows itself. It doesn’t take one path to the warehouse; it takes several hops through multiple messaging and queueing systems, each a different kind of data transport. And the pipeline it travels on exists only for its kind of event — the next event type has its own pipeline, built differently, trying to achieve a very similar purpose.

So the cost isn’t in any one pipe. It’s in the fact that there are many of them, doing nearly the same job, and that none of the hops between them were designed with observability in mind.

The answer was a single real-time event processing system, Iguazu, built to replace the legacy pipelines — and a deliberate shift in strategy away from heavily relying on AWS and third-party data services toward open source frameworks that could be customized and better integrated with DoorDash’s own infrastructure. Kafka became the pub/sub layer because it unifies heterogeneous data sources at high throughput; Flink became the processing layer for its low latency, native event-time processing, fault tolerance, and out-of-the-box connectors.

The tuning is where the cost came out. A replication factor of two with one minimal in-sync replica instead of the typical three saves disk and broker CPU while still providing adequate redundancy. Producers get their acknowledgment as soon as the partition leader has persisted the data, not the followers. A sticky partitioner with a linger time between 50ms and 100ms significantly improves batching. Altogether, that tuning reduced Kafka broker CPU utilization by 30 to 40%.

In two years the system scaled to process hundreds of billions of events per day at a 99.99% delivery rate. Onboarding a new event, which had been a support-heavy task involving pull requests across several Terraform repositories, was automated with a GitHub App and a Cadence workflow that opens and merges those PRs programmatically — cutting onboarding from days to minutes, with a Retool UI on top so users can search schemas and start the workflow themselves.

When several systems each work correctly and the whole is still slow and expensive, stop debugging the parts — the defect is that there are several of them.