Grab's authentication tokens lived on one Redis node, and it was about two months from choking. Here's how they moved to a 9-node ElastiCache cluster with zero downtime and no invalidated tokens.

Grab — 55 million app downloads and 1.2 million drivers at the time — kept its authentication tokens in a single Redis node, with a MySQL store as backup. That one node absorbed a peak load of about 20,000 requests per second, and the read-to-write ratio was roughly 200 to 1: mostly reads validating tokens, comparatively few writes issuing them.

For a while, one big Redis node is a perfectly reasonable place to keep tokens. It’s fast, it’s simple, and reads are cheap. The trouble is what “one node” means when the node runs out of room.

The team estimated they had a maximum of two months to react before the single node would start to choke under growth. And because it was a single node, it was also a single point of failure for authentication across the platform.

The shape of the choice was between three targets. A single larger Redis instance would just recreate the single-point-of-failure problem at a bigger size. A hand-rolled Redis cluster would work but drag in client-side sharding complexity. Self-hosting Redis on EC2 was possible but deferred for later.

Grab chose a managed AWS ElastiCache cluster: 3 shards, 2 replicas per master, 9 nodes in total. That answers the availability question. It leaves the harder one open — how do you copy a live, constantly-changing set of login tokens from the old store to the new one without a window where a token exists in one place and not the other?

The answer was a six-step migration where every step was governed by a configuration flag, so any step could be rolled back instantly. First, a one-time bulk transfer of existing data using Redis SCAN, DUMP and RESTORE. Second, asynchronous shadow writes to the new cluster — writing to both without adding latency on the request path. Third, promote those to synchronous writes so the new cluster is authoritative-in-waiting. Fourth, asynchronous validation reads that fetch from the new cluster and compare against the old one. Fifth, switch reads to the new cluster and stop reading from the old one. Sixth, stop writing to the old Redis and clean it up.

The migration hit its constraints: zero downtime, no invalidated tokens, no users logged out. It worked not because the cutover was perfectly timed, but because the new store spent an entire phase being read from and compared before anyone depended on its answers.

When the data is live and losing any of it is unacceptable, don’t cut over — dual-write, read the new store in the background to verify it matches, and put every step behind a flag you can reverse.