Samza gave LinkedIn a stable, fast engine for stream processing since 2013, but building on it meant writing windows and joins by hand and wiring jobs together through raw Kafka topics. Samza 1.0 layered a high-level API, Beam portability, and a SQL interface on the same engine.

LinkedIn standardized on Kafka as the transport for all tracking data back in 2012, and Apache Samza was built the following year to solve the problems every application hit consuming that data — checkpointing, managing local state, handling failures, scaling out. It became a top-level Apache project in 2014. By the time this post was written in November 2018, Samza had over 3,000 applications running in production at LinkedIn, spanning anomaly detection, fraud, performance monitoring, notifications, and real-time analytics, and had reached a peak of 1.2 million messages per second on a single machine.

That engine, though, offered only a simple callback-based API operating at the level of an individual message. Anything more complex — a window, a join — had to be implemented by hand on top of it. Chaining logic together meant wiring multiple Samza jobs through raw Kafka topics, with one job’s output feeding the next job’s input.

Samza 1.0, released in November 2018, kept the same low-level execution engine underneath — and inherited its performance — but added a high-level API with built-in operators like map, filter, join, and window, so complex pipelines could be expressed by combining operators instead of hand-coding them. On top of that, applications written against Apache Beam’s portable API could now run on Samza directly, which also brought Beam’s event-time-based processing (sliding, tumbling, and session windows, with configurable triggers) to handle data that arrives late or out of order.

Samza SQL, built on Apache Calcite, let engineers define streaming pipelines declaratively without writing Java. A new Table API simplified joining a stream against external data — LinkedIn cites Couchbase, Espresso, and Rest.li integrations — with built-in throttling and caching so a high-rate stream wouldn’t overwhelm a downstream service. And a standalone mode let Samza run outside YARN, coordinating parallel instances via Zookeeper, for teams who wanted Kubernetes, Mesos, or the cloud instead.

The bridge was architectural discipline: every new capability sat on top of the existing execution engine rather than replacing it. The high-level API “uses the same execution engine as the low-level API, inheriting all its performance benefits.” Beam integration reused that same engine underneath a different, portable API surface. None of the years spent making Samza reliable and fast had to be redone to make it usable.

LinkedIn’s own framing of the release is unusually candid for an engineering blog: “Samza is 1.0, but we are far from being done.” The team lists hot-standby containers, auto-scaling, machine-learning use cases, and end-to-end exactly-once processing as areas still being actively invested in at the time of the 1.0 release, not things it had already solved.

Making a distributed system reliable and making it pleasant to build on are different jobs, done in sequence — Samza spent years earning trust with a low-level API before it was safe to layer a friendlier one on top without disturbing what already worked.