THE MIGRATION LINE
INCIDENT // ZERODHA-2021STABILIZED

Dozens of Zerodha's internal systems stayed in sync by calling each other's APIs directly — a maze that grew for years. They cut the direct connections and let every system publish to, or read from, one shared Kafka bus instead.

ZERODHA·2021·KAFKA / EVENT-DRIVEN / DECOUPLING / INFRA·✓ Sourced — zerodha.tech
System stress over timeNominal
up to 7 days → instantUnsubscribe latency
dozens, org-wideSystems involved
~zero, for yearsMaintenance overhead
KafkaBus
BASELINE

A maze of API calls holding several sources of truth in sync

As the number of standalone but interconnected systems inside Zerodha grew, so did the effort to keep them synchronized. Several different kinds of customer databases, and their subsets, spread across departments, all had to stay in sync with certain sources of truth. Whenever data changed in one system, that change had to be pushed out to many other, disjoint systems — via direct API calls between them. The maze had grown slowly and organically over many years, starting from a single system and sprawling into dozens.

REQUIREMENTS

Most of the maze was really just data that needed to arrive, not a call that needed a response

Looking at the maze honestly, most of these API connections existed for one reason: a downstream system needed to stay consistent with a source of truth. That’s a narrower problem than “systems need to call each other” — it’s “systems need to know when something changed.” Zerodha’s framing: “good old PubSub to the rescue.”

THE CALL

Identify sources of truth, standardize topics, disconnect everything else

The fix was to identify which systems were the actual sources of truth, spec out and document standardized data structures for what they’d publish, and set up a central message bus and queue — Kafka — that those source-of-truth systems publish to on standardized topics as their data changed. Anything that needed that data simply consumed from the relevant topic and did what it needed with it. Zerodha describes removing “unnecessary API layers from all systems, physically disconnecting them from each other” — the direct calls didn’t just get replaced, they were deleted.

In practice this was implemented as small sidecar utilities sitting next to the more complex systems, connecting things like the CRM, the trading platform’s user database, and the mailing list manager — without those systems ever connecting to each other directly.

CONSEQUENCES

An instant unsubscribe, and years of not having to think about it

The concrete payoff Zerodha names: when a user deactivates their brokerage account, they’re now unsubscribed from the mailing list instantly, instead of waiting “up to 7 days” — which they call an industry first. More broadly, systems “stopped being aware of each other,” which is what let Zerodha evolve, change, and even remove entire systems across departments with minimal disruption. The setup, Kafka installation included, has run with “practically zero maintenance overhead over years.”

Zerodha doesn’t quantify message volume, list which topics exist, or describe what happens when a consumer falls behind or a schema needs to change — the account is a design retrospective, not an operations postmortem, and it reads that way.

Decoupling isn’t an architecture diagram decision, it’s an org decision: once a system no longer needs to know who’s listening, you can change what’s listening without asking permission.

Source — read the original

A plain-language, AI-drafted and human-edited retelling of the article published on zerodha.tech, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.

← All systems