Datadog's event platform outgrew its clustered search-system storage. Husky is the third-generation rebuild — a columnar store that decouples compute from storage on commodity object storage — and this is the design that guarantees exactly-once ingestion into it.

Husky is Datadog’s third-generation event store: a distributed, time-series-oriented, columnar store optimized for streaming ingestion and hybrid analytical and search queries, whose architecture decouples storage and compute so the two can scale independently. Data lands in commodity blob storage; compute reads it back. That decoupling is the whole point — but it made one thing genuinely hard.

Because of the nature of Datadog’s product, Husky’s storage engine is almost completely optimized around serving large scan and aggregation queries. It can run needle-in-the-haystack searches, but it is not designed to perform point lookups at high volume and with low latency. That single constraint is what turns “don’t ingest the same event twice” into a design problem instead of a lookup.

The team stated the bar plainly. Ingestion had to guarantee that every event enters Husky’s storage engine exactly once, with no duplicates, ever. It had to work with Datadog’s existing multi-tenant ingestion pipelines. It had to keep ingestion latency reasonable. And it could not blow up costs — which, for a system that pays per file written to blob storage, is not an abstract concern.

Costs and correctness turned out to be the same problem wearing two hats. Husky isolates each tenant’s data into a dedicated table and never mixes data from different tables in the same file, so there is an almost-linear relationship between the number of tenants a writer touches and the number of output files it uploads. More tenants per writer means more files, more spend, and more work for the downstream Compactor.

Without any structure, events would be routed randomly across the ingestion workers. That keeps things simple and load-balanced, but it scatters copies of the same event across many workers — so deduplication would have to be global, and every worker would have to know about every event ID it might ever see. Both the correctness check and the per-tenant file cost get worse the more workers each tenant is smeared across.

The alternative is to introduce locality: deterministically map each event to a shard — a group of Kafka partitions — by its ID and timestamp, via an upstream service called Shard Router. An event with a given timestamp and ID is always routed to the exact same shard, no matter when or how often it arrives. The question was whether determinism could survive real customer behavior, where a single tenant can suddenly increase volume by one to two orders of magnitude and force its shard assignment to change.

Datadog chose locality. Shard Router deterministically maps each event to a shard, and each shard carries traffic for only a relatively small subset of tenants. To handle the reality that a tenant’s shard set has to change over time — because of scaling actions that add shards, or rebalancing actions that move them — they bound assignments in time with Shard Placements, so a given ID’s routing is stable within a window even as the overall map evolves.

Because routing is deterministic, deduplication only ever has to happen within a shard. Each downstream writer therefore holds a much smaller set of event IDs than it otherwise would — small enough to fit in memory — so the writer can check “have I already seen this?” locally and fast, without ever asking the scan-optimized storage engine to do a point lookup.

Locality pays off twice. Deduplication becomes cheap and in-memory instead of a global lookup, and limiting how many tenants each writer concurrently processes cuts the number of output files — and therefore blob-storage cost and Compactor work — because of that near-linear tenant-to-files relationship. The scan-optimized store gets to stay scan-optimized.

The cost is honest and structural. Determinism now depends on every Shard Router node making the same routing decision for the same event, or the system risks the very duplicates it set out to prevent — so consensus between distributed router nodes becomes load-bearing. Assignments still have to shift as tenants burst by one to two orders of magnitude, and each shard still has to receive a roughly equal slice of traffic to keep the ingestion nodes balanced. The hard part didn’t vanish; it moved from the store into the map.

When your storage engine can’t answer “have I seen this before?” cheaply, don’t make it — route events so the question can only ever be asked in one small, in-memory place.