Why Percentiles Don't Work the Way You Think
Most monitoring systems calculate p99 metrics and store them in time series databases, then average those values when resampling—which is mathematically meaningless, because you can't average percentiles without the original population of events.
Read Original Summary used for search
TLDR
• Time series databases resample data at different time resolutions by averaging values, but averaging percentiles is mathematically invalid—you need the original event population to compute percentiles
• Banded metrics (histograms over time) solve this by storing counts of observations in value ranges, which ARE safe to average and can be used to compute approximate percentiles
• Despite being "wrong," stored percentile metrics might still be useful since they show outlying behavior and worsen when things get worse—but you risk confusing teammates who don't understand the limitation
• Heatmaps (stacked histograms over time) reveal the full distribution shape, providing far more insight than any single number like an average or percentile
In Detail
The common practice of calculating percentiles (like p99) and storing them as metrics in time series databases is fundamentally broken. When you request data at different time resolutions or when databases archive data for long-term storage, they average the stored values. But averaging percentiles is mathematically meaningless—percentiles must be computed from the original population of events, and time series databases don't retain that population. As one engineer put it: "Percentiles are computed from a population of data, and have to be recalculated every time the population (time interval) changes."
The proper solution is to store the underlying distribution using banded metrics, which divide the value space into ranges and count how many observations fall into each range over time. This creates essentially a series of histograms. Unlike percentile metrics, banded metrics are robust to averaging and resampling because the calculations are commutative over all time ranges. From these bands, you can compute approximate percentiles, and more importantly, visualize the full distribution as heatmaps—which reveal far more about system behavior than any single number.
That said, the author acknowledges a nuanced reality: despite being mathematically wrong, stored percentile metrics might still be useful for many purposes. They tend to show outlying behavior and get bigger when things get worse, even if the exact values are incorrect. The key is understanding the limitation and not misleading teammates who might assume the metrics are accurate. As the author warns with a colorful analogy: "If you give people a bottle labeled 'alcohol' and it contains methanol, some of them will drink it and go blind."