So when does this happen? When you instrument your code and add a new metric, sometimes you attach more context than you need. This means that the team is generating a lot more monitoring data than before - or perhaps, accidentally, too much. What happened here is that perhaps someone introduced a label that can have so many values that the number of series (a.k.a. But then you’ll notice that the active series drastically spikes. See how the line slopes down a bit on the left of the graph? Someone may have changed some relabeling rules to drop an irrelevant label. The image above shows what a sudden spike in cardinality looks like. This change can have a big impact on your observability system’s performance - and your bottom line. Something our customers bring up a lot is sudden spikes in cardinality, which is when a metric with medium or lower cardinality suddenly transforms into a metric with high cardinality. So within this blog post, we will refer to the number of series interchangeably with cardinality.

Keep in mind that cardinality corresponds to the number of metrics series. It’s when you have high cardinality that you and your team can start to face challenges with your observability systems, such as high resource usage. You can have lower cardinality (1:5 label-value ratio), standard cardinality (1:80 label-value ratio), or high cardinality (1:10,000 label-value ratio). In the example above, the label status_code has a cardinality of 5, environment has a cardinality of 2, and the overall cardinality of the metric server_responses is 10. To put it simply: Cardinality is the overall count of values for one label. Cardinality spike: Basic diagram of cardinality in Prometheus. In the world of Prometheus and observability, label cardinality is extremely important because it impacts the performance and resource usage of your monitoring system.

The basic definition of cardinality is the number of elements in a given set. Here we will give you a basic overview of what cardinality is and why it’s an important factor in your observability setup, especially when there is a dramatic uptick. At Grafana Labs, we spend a lot of time talking to our customers, and something we’ve heard from people in a wide range of organizations is that they want to be able to better manage sudden spikes in cardinality.

Cardinality of a multislot clips