A JSON blob is cheap right up until the first read that only needs one field out of it.
By 2018, Slack offered 165 customizable workspace preferences — things like default language, default channels, and who’s allowed to create or archive a channel — across more than 150 Enterprise Grid organizations and 70,000+ paid workspaces. Every one of those preferences lived in a single JSON blob, stored as one column in the workspaces MySQL table alongside the rest of a workspace’s core data. At Slack’s scale, that blob had grown past 55 kB.
The replacement is a classic entity-attribute-value table: workspace_id, pref_name, pref_value — one row per preference per workspace instead of one blob holding all of them. Fetching a single preference now means fetching a single row, not the whole workspaces table. Adding a new preference means inserting rows, not altering a schema. And because each preference has its own cache entry, changing one no longer invalidates a workspace’s entire cached preference set.
On top of the schema change, the project generated individual getter functions for every preference, so a developer pulls exactly the value they need with an enforced, specific return type. In the process, the author specified concrete types for 12 preferences and found 14 that were already deprecated and could be dropped.
The cutover ran in stages. First, updates double-wrote to both the old blob and the new EAV table, built on Slack’s Vitess cluster, while a backfill script worked through every existing workspace to populate the new table from scratch. Next came a “dark mode” — every read queried both tables and compared the results for inconsistencies, while still serving the answer from the old table. That comparison stage did real work beyond validating the migration: it surfaced a pre-existing bug, unrelated to the re-architecture, involving how a workspace object was being passed by value during preference persistence. Only once dark-mode reads looked clean did light mode roll out gradually, behind feature flags — first to test workspaces, then internally at Slack, then to everyone.
The work was carried out by Alisha Ukani, a Harvard junior interning on Slack’s Enterprise Architecture team — worth noting, because moving a live schema underneath 70,000+ paying customers isn’t normally intern-sized work, and it’s a sign of how mechanical and reversible the double-write, dark-mode, light-mode pattern actually made it. The honest cost is the code-generation layer itself: every preference now needs its own typed getter, more upfront structure than “read a key out of a blob,” in exchange for never again invalidating a cache entry you didn’t touch.
A JSON blob is cheap until the first read that only needs one field out of it — model each preference as its own row before the cache-invalidation tax outgrows the convenience.
Source — read the original
A plain-language, AI-drafted and human-edited retelling of the article published on slack.engineering, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.