Prometheus doesn't lose data when a counter resets — it loses the one increment that happened right before the reset, and Airbnb had to engineer that back.
Airbnb’s services were instrumented with StatsD, collected by Veneur sidecars and forwarded to their vendor — with an internal fork of Veneur handling aggregation, stripping away instance-level labels like pod and hostname before submitting metrics, specifically to keep costs under control at Airbnb’s scale.
Airbnb’s stated approach for any monitoring migration is to frontload metric collection — get everything flowing into the new system first, exposing scale and correctness bottlenecks early, before touching user-facing assets like dashboards and alerts that need real data to validate against. Doing that meant answering, for the first time, where to fork the metrics, whether to adopt OpenTelemetry, and whether their existing metrics would even work well against a Prometheus backend.
For instrumentation, Airbnb adopted OpenTelemetry Protocol (OTLP) as the recommended path for internal services and used a dual-write strategy — the shared platform metrics library used by roughly 40% of services was updated to emit both StatsD and OTLP simultaneously, making broad rollout low-friction. For the harder problem of aggregation, replacing what the Veneur fork used to do, Airbnb evaluated and rejected four alternatives: continuing with Veneur meant a costly rewrite for Prometheus’s protocol; Prometheus recording rules required storing raw data first, undermining the cost-saving goal entirely; m3aggregator was battle-tested elsewhere but architecturally more complex than the team wanted to take on; the OpenTelemetry Collector itself lacked real aggregation support despite proposals to add it; and Vector had no built-in horizontal scaling and was written in Rust, a language with little existing adoption at Airbnb.
Moving to OTLP alone dropped CPU time spent on metrics processing from 10% to under 1% of total production CPU samples, and the new centralized aggregation pipeline cut overall metrics cost by roughly an order of magnitude. But the migration surfaced a subtle correctness bug: Prometheus computes rate-of-change from cumulative counters starting at zero, so a counter that increments once and then resets — say, from a pod restart — before it increments again loses that single increment entirely, since rate() never sees two data points to diff. Airbnb’s low-rate, high-cardinality counters (tracking things like requests per currency per user per region) hit this edge case constantly, undercounting metrics that product teams depended on. After rejecting several workarounds that would have pushed complexity onto every team writing dashboards, Airbnb built “zero injection” directly into the aggregation tier — flushing a synthetic zero the first time any counter is seen, transparently matching Prometheus’s expected semantics without a single dashboard or alert needing to change.
Moving to a new system’s data model doesn’t just change your tools — it can quietly break assumptions your old system enforced for you, and those gaps are worth hunting for even after the “migration” itself looks complete.
Source — read the original
A plain-language, AI-drafted and human-edited retelling of the article published on airbnb.tech, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.