Cloudflare's billing jobs run on a petabyte-scale ClickHouse cluster. After a routine partitioning change, they started slowing down for no visible reason — I/O, memory, and rows scanned all looked clean. The real cause was a mutex.

Cloudflare runs its billing pipeline as daily aggregation jobs on a petabyte-scale ClickHouse cluster — time-critical jobs, since they determine invoicing across hundreds of millions of dollars in revenue. Months before anything went wrong, the team had migrated a table called Ready-Analytics from a (day) partitioning scheme to (namespace, day), so each namespace could carry its own data-retention policy instead of a single 31-day window for everyone.

The team’s assumption going in was that this wouldn’t touch query performance: every query already filtered by a specific namespace, so the number of parts a single query had to read shouldn’t change either way.

Following that migration, the billing team reported that their daily aggregation jobs were slowing way down. The stakes were plain: if the jobs didn’t complete, bills don’t go out.

The first pass at diagnosis found nothing. I/O performance was normal. Memory usage was fine. Individual query metrics showed the queries weren’t reading any more data or parts than before. As the team put it, all the usual suspects looked clean — I/O, memory, rows scanned, parts read.

Several days into dead ends, someone plotted query duration against the total part count across the whole cluster — not just the parts an individual query touched. The correlation was undeniable: as total part count grew, query performance degraded right along with it, even though no single query was reading more parts than before. Something else in the system was reacting to how many parts existed cluster-wide, not to how many any one query needed.

The team turned to ClickHouse’s trace_log system to generate CPU flame graphs on leaf SELECT queries. That surfaced a function called filterPartsByPartition, consuming 45% of leaf query CPU time filtering a vector of parts by partition ID. A first patch reordering the filter’s heuristics bought only a 5% improvement — the right neighborhood, but not the actual bottleneck.

So they changed instruments. CPU traces only sample threads that are actively running; the team switched to “Real” traces, which also capture threads that are waiting. The new flame graph showed the true cost: more than half of leaf query duration was being spent waiting for a mutex protecting the list of active parts.

The query planner’s routine, on every single query, was: acquire an exclusive lock on the MergeTreeData mutex, copy the entire list of parts in the table, release the lock, then filter the copy down to what the query actually needed. With tens of thousands of parts in the table and hundreds of concurrent queries all needing that same lock, threads were, in the team’s words, all just standing in a single-file line waiting for exclusive access — one at a time, no matter how many CPUs sat idle.

The first fix replaced the exclusive lock with std::shared_lock, letting concurrent readers proceed together instead of queueing single-file. The flame graph’s lock contention essentially vanished immediately. But copying was still expensive on its own — roughly a quarter of query duration went to copying the full parts vector, another quarter to filtering it, which involved copying again — so the team added a “shared copy” strategy where read-only operations reuse a cached parts list and only copy the smaller filtered subset they actually need, regenerating the cache only when the underlying data changes. Both fixes were merged upstream as ClickHouse PR #85535 and shipped in version 25.11.

When the problem first surfaced, the cluster held about 30,000 parts per replica. A year later it held 160,000 — nearly five times as many — and query durations stayed stable, held there by the three fixes rather than by the part count staying small. The team is candid that the underlying question isn’t fully closed: the same per-namespace partitioning scheme has also caused separate problems for ZooKeeper’s metadata tracking, so whether this partitioning strategy is sustainable at even greater scale remains open.

When a metric degrades in step with a total system size instead of with any individual request’s workload, stop profiling what’s running and start profiling what’s waiting.