LinkedIn left Kafka Mirror Maker for Brooklin in 2018 and got stability — but its mirroring pipelines still blew their latency SLAs, and the reason was hiding in how partitions were counted.

LinkedIn runs multiple data centers, with multiple Kafka clusters in each, each holding an independent set of data. Mirroring — replicating topics between those clusters — is what makes cross-data-center analytics, fault tolerance and outage recovery possible. It runs at a scale of more than seven trillion messages per day.

In 2018, LinkedIn migrated that mirroring off Kafka Mirror Maker onto Brooklin, its own distributed streaming system, and open-sourced Brooklin the following year. By LinkedIn’s own account, switching from KMM to Brooklin Mirror Maker brought stability and reduced the operational overhead KMM had imposed. The migration worked. That’s the calm before this story.

Brooklin splits a datastream — a pipe from a source cluster to a destination — into tasks. Each task is a long-running process replicating a subset of the topic’s partitions, and they all run concurrently. On paper, spread the partitions evenly across the tasks and the work is evenly spread too.

In practice, some mirroring pipelines showed higher latency to replicate records, and in near-line, business-critical pipelines that meant blown delivery SLAs. Meanwhile the operators couldn’t say how many tasks a datastream actually needed. They improvised the task count by hand whenever the workload changed, over-reserved tasks in case they’d guessed low, and wasted hardware doing it.

Look at how the partitions get distributed and the first strange fact appears. Brooklin — and the other Kafka replication systems like it — distribute partitions across consumers by partition count, not by how much data each partition actually carries. Not by choice, exactly. Kafka does not natively provide partition throughput metrics, so count is the only thing there is to balance on.

That means there was no direct way to identify a hot partition, and no direct way to redistribute it. It also means a partition with a throughput rate higher than a single task can support — a topic with too few partitions, or with poor partitioning logic skewing its traffic — could sit there hurting every other topic sharing its task, invisibly.

The metrics did exist, just not where anyone could use them. Cruise Control, which runs on every Kafka cluster at LinkedIn, emits per-partition incoming and outgoing byte rates and message rates to a sample store. But it pauses collecting them during a rebalance — to avoid triggering another one — and a rebalance can run for hours.

The root cause wasn’t a bug in Brooklin. It was the assumption underneath the assignment. Balancing by partition count is only fair if partitions carry roughly equal load — and across a heterogeneous workload, they don’t.

The fix begins by making the missing metric exist. LinkedIn added a feature to Cruise Control so partition metrics keep flowing to a separate sample-store topic during a rebalance, without accidentally triggering another rebalance. An Apache Samza job then ingests those custom-serialized topics from every Kafka cluster, filters and converts the records to Avro, and aggregates them into a single Kafka topic — because Apache Pinot, the real-time OLAP store they chose to serve the data, ingests in real time from only one topic per table. The result is the Partition Metrics Provider.

With throughput finally visible, Brooklin’s Kafka connectors were moved off the high-level Kafka consumer APIs onto manual partition assignment, which is what gave Brooklin the power to place partitions itself. A new assignment strategy — contributed to Brooklin open-source — queries the PMP on a rebalance trigger, predicts how many tasks the recent traffic needs, and distributes partitions by load so that hot partitions don’t group together and tasks no longer need equal partition counts.

A monitoring service watches message delivery latency and triggers a rebalance in Brooklin when it crosses a threshold; the rebalance typically completes within seconds, and assignments stay sticky in ordinary situations like onboarding a new topic. In LinkedIn’s A/B evaluation, a control cluster running plain Brooklin Mirror Maker and a candidate cluster running the load-balanced version mirrored the same topics under the same load. The control cluster’s metrics were significantly skewed across hosts. The candidate cluster’s deviation was minimal, and a host running twice the tasks processed twice the records — which is the real prize, because it means the number of tasks a host can carry is finally predictable.

Migrating to a better system fixed LinkedIn’s stability problem but not its fairness problem, because both systems balanced on the only number Kafka gave them — and the number was wrong.