LinkedIn's batch and stream jobs needed somewhere fast to land online, and the answer was Voldemort Read-Only stores plus a scatter of proprietary systems. Venice replaced them with one platform, now open source.

LinkedIn’s batch and stream processing jobs produce enormous volumes of derived data — recommendations, feature values, and other results computed offline that then need to be served online, fast. Before Venice, that serving layer was patched together system by system: LinkedIn leaned on Voldemort’s Read-Only stores plus, in the company’s own words, “several proprietary” systems, with different teams building their own bridge from an offline job to an online store.

Work on a replacement started small. The very first commit in what became the Venice codebase landed roughly eight years before LinkedIn’s 2022 open-sourcing announcement, but development only picked up in earnest once a fully staffed team took it on at the beginning of 2016. Venice entered production at the end of that year.

LinkedIn’s own description of Venice compresses the requirements into one sentence: a “high-throughput, low-latency, highly-available, horizontally-scalable, eventually-consistent storage system with first-class support for ingesting the output of batch and stream processing jobs.” Read plainly, that means the same store has to accept a nightly Hadoop job’s full output and a live Samza stream through the same front door, then serve production reads fast enough that a feature like a recommendation ranking doesn’t notice the difference.

Venice settled on multiple write paths rather than forcing every producer through one pipe: Full Push jobs replace a dataset wholesale, Incremental Push jobs add to what’s already there, and streaming writes arrive continuously through Apache Samza, with support for stream reprocessing in a Kappa Architecture style. On the read side, applications can do Single Gets, Batch Gets, or Read Compute — declarative operations that include vector math over embeddings, the kind of ranking and matching work LinkedIn’s recommendation systems lean on. A client library called Da Vinci adds a fourth option: caching state locally on the application host itself, eventually consistent with the source of truth.

Underneath, Venice moved its storage engine from BDB-JE to RocksDB, runs self-healing via Apache Helix, and supports multi-region replication and multi-tenancy with configurable quotas.

The migration off Voldemort was concrete and dated: by 2018, roughly 500 Voldemort Read-Only use cases running in production had been fully migrated to Venice. Four years later, LinkedIn’s own numbers describe a platform running at a different scale entirely — 1,800+ datasets, 300+ applications, average ingestion of 14 GB/second (39 million rows/second) climbing to a peak of 50 GB/second (113 million rows/second), 1.2 petabytes and 3+ trillion rows moving through it daily, and read traffic peaking at 6.25 million single-get queries per second. It runs across 17 production clusters per region on roughly 4,000 physical hosts, pushes more than 1,400 automated jobs a day, and powers features including “People You May Know,” “Jobs You May Be Interested In,” LinkedIn’s A/B testing platform, and its feature store, Feathr. On September 26, 2022, LinkedIn open sourced it.

A derived-data store earns the right to replace a dozen ad hoc ones by treating every way data enters — batch, incremental, or streaming — as equally first-class, not by being fastest at just one.