Grab's Hive Metastore had become a single point of failure whose query planning time scaled linearly with partition count, while some ML datasets fragmented into thousands of sub-1MB files per partition. Moving to Iceberg cut one navigation query from 70 seconds to 6 — but adopting a second table format meant Grab had to build its own way to hide that fact from everyone downstream.

Hive Parquet, managed via the Hive Metastore with a directory-based layout, had been the backbone of Grab’s data lake for years. At scale, four specific problems emerged: the Metastore itself became a centralized failure point, with query planning time scaling linearly with partition count under high concurrency; some ML datasets fragmented into thousands of sub-1MB files per partition, driving up S3 listing costs and slowing scans; data engineers absorbed constant manual toil registering partitions, since there was no native support for UPSERT or DELETE; and because the Metastore, not the storage layer, was treated as the source of truth, direct writes to S3 routinely left the catalog stale and out of sync with what was actually on disk.

Moving high-value tables to Iceberg produced concrete, measured wins. Z-ordering — co-locating rows with similar values across chosen dimensions so Trino can skip irrelevant files using min/max statistics — cut a high-traffic navigation query’s runtime from 70 seconds to 6. A heavily-queried operations table saw its daily S3 API costs drop by up to 95%, with no query changes at all, from larger file sizes and the elimination of expensive object listing. A funnel-analysis dataset needed roughly half the cluster resources it did before, and a separate ML feature pipeline gained improved feature freshness as a side effect.

Migrating an actively-queried, multi-team, multi-engine lake was never going to be a single flag flip, so Grab moved its highest-value tables first rather than converting everything at once. That created a real, unplanned second problem: with Iceberg, Delta, Hudi, and Hive tables now coexisting, only one format can be Spark’s default catalog, so querying anything else required a format-specific prefix — meaning every engineer had to know which format each table actually used, and any later format migration would silently break hardcoded downstream queries. Grab’s answer was UnifiedSparkCatalog, now open-sourced: it loads metadata from the Hive Metastore, detects each table’s real format automatically, and transparently routes the operation to the correct underlying catalog — deliberately naming itself spark_catalog specifically because legacy Hive code paths check for that exact name, and reusing existing catalog instances so lineage tools like OpenLineage kept working unmodified.

Production rollout surfaced real friction: “zombie locks” in the Hive Metastore blocked commits under high load, traced to a metastore-side read timeout set too low and fixed by adjusting retry intervals and raising it; Spark 3.4’s new timezone-naive timestamp type clashed with Iceberg’s timezone-aware default, breaking legacy Hive views until Grab built a custom migration workflow and patched their Trino deployment for consistent casting; and generating Iceberg metadata for a table means reading its historical data, which can trigger a real one-time cost spike as files shift between S3 storage tiers. Grab managed that cost by prioritizing which tables to migrate based on actual scan frequency and API cost, rather than migrating the whole lake in one uniform pass.

Iceberg is now the default table format for both of Grab’s core data platforms — Slide (batch transformations) and Hugo (online-to-lake ingestion) — with adoption still expanding across other compute platforms. Grab is already looking past this migration: experimenting with Storage Partitioned Joins to remove Spark shuffle stages, and watching the Apache XTable project for future interoperability between table formats, rather than assuming Iceberg is the last format they’ll ever need to support.

Solving a storage migration’s technical problems isn’t the finish line if it leaves a developer-experience gap behind — build the abstraction that hides which system is under the hood, or that complexity just resurfaces as tribal knowledge.