Airbnb's centralized Resque cluster was at-most-once and bounded by a single Redis instance, so the team built Dynein — a delayed job queue on SQS with a scheduler on DynamoDB.

For years, Airbnb ran background work on a centralized cluster of Resque workers, with Resque Scheduler on top and a custom scheduler bolted on for longer delays. It was easy to use. It was also built for a monolith, and Airbnb was moving toward a service-oriented architecture.

The jobs on that queue were not incidental. Matching hosts in the Open Homes program to non-profits and evacuees is compute-intensive enough to belong on a queue. The reminder a guest gets before check-in is a job scheduled potentially months or years into the future — and it has to arrive.

Airbnb went and talked to the teams using the queue first. The list that came back was specific. Reliability, meaning no data loss on failure or restart and at-least-once delivery of every job. Scalability, meaning horizontal — capacity planning, not heroics. Isolation, meaning one application’s queue backing up must not affect another service’s job processing.

Then the sharper ones. Timing accuracy — a p95 scheduling deviation under 10 seconds, because many use cases need jobs to run within seconds of their scheduled time. Efficient queuing — per-message acknowledgment, dead letter queues, and a separate worker pool per consumer rather than one shared pool. Usability — the client library should hide the scheduler and the transport entirely, and should push best practices like exponential backoff rather than exposing internals. And unscheduling — any job cancellable at any time by a unique id.

An in-house MySQL-based system already existed for long delays and stronger delivery guarantees. It was built to be highly consistent, not highly scalable.

For the queue, Airbnb chose AWS SQS, and the reasoning is unusually honest about trade-offs. SQS has no strong ordering guarantee and isn’t a storage system like Kafka. With those constraints removed, what’s left is exactly what a job queue wants — trivial to provision a new queue (at Airbnb, a pull request to a Terraform repo), which is what makes per-service isolation practical; very high throughput and low enough latency; at-least-once delivery, which is what Dynein wanted anyway; and dead letter queues, per-message acks, access control and encryption at rest for free.

For the scheduler, the first version of Dynein used Quartz, the Java scheduling library Airbnb had already run in production for years. It worked. But Quartz distributes poorly across schedulers and databases, has a large API surface that’s hard to standardize across teams, and — in the words of its own manual — “is not a job queue.” Coupling scheduling to queuing left Airbnb with two bad org choices: one team maintaining a Quartz cluster running every service’s application logic, exactly the monolithic Resque problem again, or every service running its own Quartz cluster.

The number that forced the decision was 1,000 QPS. That was the ceiling per r4.8xlarge RDS instance — 500 enqueues and 500 dequeues, not counting immediate jobs — and Airbnb was running multiple instances just to route around it, which is expensive both operationally and financially.

The deep dive found query amplification. In Quartz, one enqueue becomes four SELECTs and three INSERTs. Every one of those queries serves a real purpose — for features like repeated schedules that Airbnb simply never used. All they needed was to dispatch a specific payload to a specific SQS queue at a specific time.

An enqueue is now one PUT — one write capacity unit. A dequeue is a GET, a SET and a DELETE — two writes and one read. Against AWS’s published pricing, Airbnb reports hitting 1,000 QPS at less than a third of the cost of that RDS instance, with linear scalability and no manual sharding across tables or database instances. Because DynamoDB’s provisioned throughput can be adjusted, they no longer have to commit to an instance size big enough for peak load all year round.

Quartz also assumed a statically defined set of schedulers, listed in its properties at startup — which becomes a real problem as Airbnb’s services move onto Kubernetes, where pods rotate far more often than EC2 hosts do. Dynein’s scheduler pods instead watch their own ReplicaSet and deterministically compute which partitions they own from the current pod list, so adding or removing a pod just redistributes partitions. That’s what makes an autoscaler possible where a StatefulSet would have been the awkward alternative.

The costs are stated plainly. The new scheduler supports a deliberately limited set of features — Airbnb concluded that adapting Quartz to their case would take significant time because the trade-offs were fundamentally different, and chose to give up generality instead. And the whole thing sits on a managed AWS service, with the vendor coupling that implies.

A general-purpose tool that does seven things well can lose to a special-purpose one that does a single thing at a seventh of the cost — but only if you keep the interface stable enough to swap the engine without anyone noticing.