Grab's Counter Service served a billion requests a day off a wide-column store. Moving to Aerospike with a map-based schema cut disk from ~3 TB to ~1 TB per node and p99 read latency by roughly half.

Grab’s Counter Service handled tens of thousands of QPS — about a billion requests per day — backed by a wide-column database. It tracked counters at three granularities: 15-minute, hourly and daily buckets. The old storage footprint was around 3 TB on disk per node.

The expensive part was the write path. The schema stored one row per bucket, and updating a counter meant a read-modify-write: read the current value, add to it, write it back. Every incoming event turned into roughly four network round-trips against the datastore, which is a lot of chatter to pay per event at a billion events a day.

On Aerospike, Grab redesigned the schema from row-per-bucket to a map-based layout, which cut the number of records by more than an order of magnitude. Increments now use Aerospike’s atomic MapIncrementOp — the database adds to the counter in place, so the read-modify-write round-trips disappear.

The numbers moved together. p99 read latency came out consistently better than the original setup, roughly a 50% improvement. Cost dropped to about 45-50% less per node. The disk footprint fell to around 1 TB per node from about 3 TB, and the primary index took about 50 GB of the roughly 100 GB usable memory per node — leaving headroom while keeping the index in RAM.

Swapping the store under a live billion-requests-a-day service was staged behind a dual-read mechanism with three modes. The reader started on the single original backend, then moved to WithShadow mode — sending a slice of reads to Aerospike in the background and comparing — ramping the shadow percentage 5%, 20%, 50%, then 100%. Only then did it move to WithSplit mode, serving real traffic from Aerospike at 5%, 30%, 70%, and finally 100%.

Because of that ordering, correctness was proven on shadow traffic before Aerospike ever answered a real request, and the live cutover was a dial, not a switch. A bad surprise showed up as a mismatch in shadow comparisons rather than as a wrong count returned to a user.

The path wasn’t clean. The Aerospike Rust client was synchronous-only at first, so the team switched to an async client. They hit a DNS bug where the client didn’t re-resolve seed hostnames during a topology refresh, which they fixed upstream. And an experiment to move the primary index onto NVMe instead of RAM caused p99 spikes, so they reverted it and kept the index in memory.

They also spent some durability to get the savings. The replication factor dropped from 3 to 2, which Grab calls acceptable because the writer keeps producing increments from the source event stream and can rebuild — while openly acknowledging that a simultaneous multi-AZ failure is a risk they judged tolerable for this workload.

Half your latency and two-thirds of your disk can come from the data model, not the database — an atomic increment beats a read-modify-write no matter how good the engine underneath it is.