Uber moved 160+ petabytes of its on-prem Hadoop data lake to Google Cloud Storage while keeping its existing Kerberos-based security model intact, building an intermediary token-exchange service to bridge HDFS authentication and GCS IAM without any of the 150,000+ existing access policies needing to change.

Uber’s on-premise data lake stored more than an exabyte of data across several HDFS clusters, secured through Kerberos-based authentication and HDFS ACLs, with a Policy Management System translating dataset-level access rules configured by data platform users into native HDFS access-control entries. As part of a broader cloud migration, Uber set out to replace HDFS itself with Google Cloud Storage while keeping the rest of the analytics stack — YARN, Hive, Spark, Presto — running on GCP Compute Engine.

Uber migrated more than 160 petabytes of data to GCS, with 19%+ of its analytical workloads now running on GCP. Under the finished architecture, no human identity is synchronized into GCP IAM at all — every access to data in GCS, whether from a person or an automated workload, is mediated through Uber’s own intermediary authentication layer instead.

Uber considered two paths: relying on native GCP IAM constructs, or building an intermediary system to handle authentication and authorization itself. Beyond the 250-binding IAM limit, Uber judged that synchronizing its entire footprint of on-prem users and groups into GCP IAM would itself widen the attack surface — more identity-based attack paths and more policy misconfiguration risk, for a system meant to be more secure, not less. Uber built Storage Access Service (SAS) instead, forked from Google’s own open-source gcp-token-broker project and heavily customized: SAS intercepts every FileSystem call a standard HDFS client makes against GCS, authenticates it, and issues time-bound GCP access tokens scoped to specific GCS prefixes — taking over the same authentication and authorization role HDFS’s NameNode used to play, using Hadoop’s existing Delegation Token framework as the exchange mechanism.

Uber’s own HDFS clusters handled roughly 500,000 requests per second at p99 latency near 20ms, with about 2ms of that budget allotted to security checks — but GCP’s Security Token Service, the API SAS relies on for token exchange, was rate-limited to 6,000 requests per minute (roughly 100 RPS) with observed latency around 50ms during prototype testing, orders of magnitude short of what Uber needed on both dimensions. Uber closed that gap with three layers of caching: a Redis-backed remote cache proactively generating and storing every likely token combination ahead of demand, an in-memory local cache on SAS servers for frequently requested tables, and a client-side cache on each workload avoiding repeat calls to SAS entirely for objects under the same already-cached prefix. The combined strategy brought average token-access latency down to roughly 0.026 milliseconds, verified across more than 45 benchmark iterations replaying real HDFS audit logs against the new system.

Migrating a security model isn’t just swapping which service authenticates you — it’s confronting the exact place your old system’s assumptions (like near-unlimited access policies, or a security check budgeted at 2ms) don’t hold in the new one, and engineering around that gap deliberately.