THE MIGRATION LINE
INCIDENT // AIRBNB-2017 SHIPPED

Airbnb's Rails monolith had no strongly typed service contracts and shipped every payload as bulky JSON. This is how they standardized on Thrift IDLs and generated Java services so a new one could take production traffic in weeks.

AIRBNB · 2017 · INFRA / SOA / THRIFT / DROPWIZARD
System stress over time Nominal
~3 weeks New service to prod
2-3 weeks Time saved per Java service
Dropwizard (Java) Framework
Apache Thrift Interface language
BASELINE

A monolith with no contract and heavy payloads

Airbnb started, like most fast-growing companies, with a monolithic Rails application. As the business grew, the plan was to break it apart into a service-oriented architecture — but breaking up a monolith exposes a problem the monolith itself hid: services have to talk to each other, and there was no shared, machine-checkable definition of how.

The old style of communication had three gaps Airbnb names directly. There was no clearly defined and strongly typed service interface and data schema. There was no robust way to generate RPC clients across the different languages the company used. And the payloads themselves were JSON over HTTP — a format the team calls large and inefficient for service-to-service traffic.

REQUIREMENTS

What a real service contract had to provide

Before choosing tools, Airbnb settled what a service framework actually had to guarantee. First, a strongly typed interface and data schema — one definition, checkable by machines, that both sides of a call agree on. Second, generated RPC clients in more than one language, because the fleet was not all Java. Third, an efficient wire format to replace bulky JSON.

On top of the interface, they wanted the operational concerns that every production service needs — mutual TLS, retry logic, circuit breakers, and metrics — to come standard, not to be re-implemented by each team that stood up a service.

THE OPTIONS

Keep JSON-over-HTTP, or adopt an IDL

The choice was between continuing with the existing JSON-over-HTTP style and adopting a formal interface definition language (IDL). Sticking with JSON meant no schema to generate from, no type checking across the boundary, and the same inefficient payloads — the status quo that had already proven painful.

Airbnb chose Apache Thrift as the service IDL, and kept Dropwizard — the Java web service framework built on Jersey and Jackson — as the runtime. Concretely, that shifted transport from JSON-over-HTTP to Thrift-over-HTTP. Vanilla Thrift did not cover everything they needed, so they extended it with custom type support, including Date and DateTime, rather than force those into primitive fields.

THE CALL

Generate the service, not just the client

The decision that made the contract pay off was to generate more than data classes. From the Thrift definitions, Airbnb generated the RPC clients in Java and Ruby, and generated the service boilerplate with standard practices already wired in — mutual TLS, retry logic, circuit breakers, and metrics. A team defining a new service — the article uses an example service named “Banana” — got the production-grade plumbing for free.

Because the interface is strongly typed and the clients are generated from it, a rename or a type change surfaces at build time instead of in production, and every service starts life speaking the same efficient Thrift-over-HTTP dialect.

CONSEQUENCES

Three weeks to production, and a Ruby gap to close

Airbnb’s own number for the payoff: several new Java services went from inception to taking production traffic in only three weeks, which the team frames as saving two to three weeks of engineering time per Java service and dramatically increasing development velocity. The gain comes directly from not hand-building the interface, the clients, and the operational plumbing each time.

The honest edge they name is coverage. The generated RPC clients and the smooth path exist for Java first; extending the same support to Ruby services was still future work. The contract-and-generate approach only fully pays off once every language in the fleet is on it.

Define the contract first and generate everything downstream of it — the interface, the clients, and the standard operational plumbing — and a new service becomes setup instead of a build.

Source — read the original

https://medium.com/airbnb-engineering/building-services-at-airbnb-part-1-c4c1d8fa811b

A plain-language, AI-drafted and human-edited retelling of the article published on medium.com, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.

← All systems