Prometheus vs Grafana on Monitoring Kubernetes

Will this trio stand the test of time (or tech innovation)?

Perhaps you were the person that introduced observability into your organization. You were in charge of designing the organization’s first Kubernetes applications, and you needed a way to determine if all was well in your development, test, and production clusters. You searched Reddit, Stack Overflow, Twitter and all of the other usual sources for product information, and came across Prometheus. Installation and configuration were quick and easy, and soon you had exactly what you were looking for: easy dashboards and alerting that could warn you when your systems needed some care. Perfect.

Soon, Kubernetes took off in your organization. Suddenly, you had 50 production clusters, each generating metrics about Kubernetes and the services running inside. Prometheus usage exploded as other sysadmins saw its value (and being free didn’t hurt either). All those new Prometheus servers were tough to manage, but the value it provided outweighed the effort, so you and the team forged onward.  

Then business applications started sending metrics to that system as well…and you started worrying.

Prometheus needed more and more disk space to take in and store all that data. Servers regularly crashed with OOM errors, and restarts forced you to lose important data. Critical dashboards started taking longer and longer to return results. You begged for more memory and scrambled to pare back non-critical metrics in an effort to stave off the metrics tsunami.

Eventually, it became time to think about what comes after Prometheus.

What is Prometheus?

To be fair, we’ll say this: the Prometheus system is an amazing solution for metrics and monitoring.

Originally built by SoundCloud starting in 2012, Prometheus has been adopted by many organizations and enjoys the support of a robust, active developer community. Its success was emphasized even more when it became the second hosted project at the Cloud Native Computing Foundation in 2016. With the availability of a huge library of metrics exporters for systems ranging from Redis and MongoDB to Minecraft and bitcoin, Prometheus and Grafana became a critical component for many infrastructure teams.

As a key to understanding both its strengths and shortcomings, it’s important to first understand what capabilities Prometheus provides. Fundamentally, there are four key features:

• A time-series database, with a key-value data model that facilitates easy identification, querying and grouping of metrics;
• A robust and highly flexible query language called PromQL;
• A pull-mode “scraper” for retrieving and storing metrics;
• An alerter component for customized notifications;

The architecture of the system is simple and straightforward:

‍

The Prometheus server is responsible for “scraping” metrics from infrastructure and applications in your organization’s computing environment, storing it as time series data with millisecond-level timestamps and flexible tags called labels that facilitate queries. Alertmanager handles raising alerts in response to rules you specify, and Grafana dashboards provide visualization capabilities to make the environment’s current state easier to understand and monitor. For cases where metrics cannot be scraped, the Pushgateway allows you to “push” those metrics into the Prometheus system.  

Metrics can be exposed to Prometheus via exporters that reside in (or near) the system to be monitored, via HTTP discovery endpoints, or by custom application code. The Prometheus community has created a large number of ready-to-use exporters, and client libraries for many languages are available to write your own if needed.  

So what’s not to love? Before we dig into that, let’s take a closer look at the other key part of many implementations: Grafana.

What is Grafana?

Grafana is a hugely popular open source visualization tool for time-series data.  Grafana Labs, the makers of Grafana, have over 2,000 customers across a broad range of enterprises, and as of July, 2022 the Grafana product itself has more than 900,000 active installations around the world.  

There are a large number of visualization types available, including all of your favorites: bar charts, histograms, heat maps, candlesticks, pie charts, geomaps, and many more. The system’s own plugin-based extensibility gives you access to many additional panels that can be incorporated into your own Grafana dashboards.

Connecting Grafana to Prometheus is straightforward: use Grafana’s “Configuration” menu to create a new Prometheus data source, point to your Prometheus instance’s network endpoint, and click “Save & Test”.  Once the data source is available you can create graphs and other visualizations by specifying a query expression that references metrics in your newly-created Prometheus data source. If you’d prefer, you can even use one of Grafana’s shared dashboards to get yourself started. 

Kubernetes Monitoring with Prometheus and Grafana

Using Prometheus and Grafana to monitor Kubernetes is fairly common, for a number of reasons:

• It’s completely free! Other Kubernetes monitoring solutions may have free tiers, but most impose fees based on data volume, number of hosts, user seats or some other mechanism.  
• There is not only an active community to provide information and support, but various resources that provide exporters, sample dashboards and many other resources. This simplifies getting a basic Prometheus and Grafana platform up quickly.
• Customization is easy (or even better, automatic via service discovery)

Prometheus itself can be installed as a containerized application, and Helm charts are often a preferred mechanism for deploying into a Kubernetes cluster. The kube-prometheus-stack chart installs Prometheus, Grafana, and various additional components to allow you to manage your Prometheus system. Separate Helm charts are also available to simplify separate Prometheus and Grafana installations.

(Prometheus is also developing Prometheus Operator, a native deployment and management mechanism.  As of this writing, Prometheus Operator is in beta status. A comparison of Prometheus Operator with Helm charts and kube-prometheus is included in the README file of the Github repository.)

Once Prometheus and Grafana are installed, Kubernetes metrics are obtained via cAdvisor, kubelet and kube-apiserver.  This article from Grafana gives a good overview of the process, the required Prometheus configs, and some sample dashboards.

Prometheus and Grafana: The Challenges

As attractive as Prometheus may be, it has a number of serious shortcomings that become more apparent as the size of the observability environment grows.

Single-server architecture

Prometheus is, fundamentally, a single-server system. As the number of Kubernetes clusters grows and the number of collected metrics increases, the load on the Prometheus server grows.  At some point your requirements may exceed the capabilities of a single server; however, Prometheus’ architecture is not designed for horizontal scalability, so expanded monitoring means a larger Prometheus and Grafana server footprint.  

Memory utilization, and the “cardinality” problem

Collecting large volumes of data takes system memory.  Prometheus does a good job of managing ingestion of metrics to the database, but there are two factors that can seriously affect how much memory Prometheus takes to do this: the sheer volume of metrics, and the design of the labels used to identify each metric. There are discussions on Stack Overflow and elsewhere that describe this problem in detail. Memory constraints can be exacerbated if the server is required to perform large numbers of PromQL queries.

Long-term data storage

Benchmarks and operational experience have shown that Prometheus can consume prodigious amounts of disk space when storing time-series metrics. Several blog posts and social media threads are dedicated to this concern (more on that later) and projects such as VictoriaMetrics, Mimir, Promscale and Thanos are working toward solving this problem.  

Lack of “hybrid observability”

As systems largely designed for time-series data, both Prometheus and Grafana work very well for numeric metrics. Integrating non-numeric data - for example logs and spans data is often needed for troubleshooting, and the inability to easily correlate that data with periods of irregular application or infrastructure behavior makes diagnosis more difficult. That being said, Grafana and Prometheus are definitely in the right direction and heading into this terrain with features like exemplars and the introduction of Grafana’s logs database Loki. 

Who will be the successor of Prometheus?

Given the benefits and challenges mentioned above, should you be prepared for a Prometheus successor? There are a few products on the horizon that you just might want to consider.

Grafana Mimir

Grafana Mimir is an OSS product whose mission is to become “… the most scalable, most performant open source time series database for metrics”.  Using Mimir, you can continue to allow your existing Prometheus instances to scrape metrics from your infrastructure and applications; however, rather than writing to Prometheus’ database you forward the metrics to Mimir. The key advantages to Mimir include support for PromQL queries and Prometheus alerting, horizontal scaling and auto-clustering, and significantly improved long-term storage capabilities.

Grafana Labs recently shared some interesting information about Mimir scalability on their blog:

• How we scaled our new Prometheus TSDB Grafana Mimir to 1 billion active series
• Scaling Grafana Mimir to 500 million active series on customer infrastructure with Grafana Enterprise Metrics

VictoriaMetrics

Like Grafana Mimir, VictoriaMetrics is an OSS solution that comes in both self-managed and hosted versions, and it solves many of the same challenges.  The VictoriaMetrics feature list is full of goodness: clustering / sharding / replication for scalability, a Prometheus-compatible pull model to ease migration and additional support for other protocols such as InfluxDB or OpenTSDB, and a robust, PromQL-inspired query language (VictoriaMetrics also supports the Graphite query language in its VictoriaMetrics Enterprise version.)

In a 2020 benchmark using the Prometheus node_exporter target performed by one of the VictoriaMetrics founders, Aliaksandr Valialkin, VictoriaMetrics running on Google Cloud was able to store the metrics data using significantly less disk space than Prometheus, with far fewer and less intense disk spikes.

Even if we ignore the fact that this project is practically built and maintained by one guy, VictoriaMetrics is nothing short of an amazing engineering solution. It keeps hitting every benchmark in the time series database domain and positioning itself to be a widely used alternative for years to come. Just recently VictoriaMetrics was compared to Grafana Mimir in this benchmark which shows just how great this technology is. 

Promscale

Promscale brings an interesting approach to the world of metrics. In addition to Prometheus remote write capability and full PromQL support, Promscale implements long-term metrics storage via TimescaleDB and Promscale extensions on top of PostgreSQL. This enables users to leverage standard SQL to analyze the behavior of their systems, including the correlation of metrics and trace data. Grafana integration via SQL, PromQL or Jaeger lets Prometheus users continue to leverage familiar visualization features.

In a 2020 blog post Timescale (the makers of Promscale and TimescaleDB) indicated that Promscale offered ... horizontal scalability to over 10 million metrics per second and petabytes of storage”, along with a solution to high-cardinality metrics and many other features.

Who’s going to take the crown?

It’s interesting to read through the social media discussions and blog posts on this topic to see what the maintainers and large customers think about this question.  Some see the potential for a hybrid Prometheus-Mimir future:

Other posts reinforced the idea of a hybrid future, with long-term storage needs being addressed by Thanos and others:

VictoriaMetrics has appeared prominently in several discussions, and this user shared his experiences with that product’s disk storage characteristics and query speed:

Interestingly, Grafana Labs is itself moving to become a full-stack observability company with new products such as Loki (“Like Prometheus, but for logs” according to the Loki Github repository), Traces (a distributed tracing service available in fully-managed and self-managed OSS versions) and Grafana Enterprise Metrics (a more scalable “Prometheus-as-a-Service” capability). Grafana Cloud combines all of these capabilities into a unified, fully managed observability platform that can integrate with existing Prometheus installations.

Keep an ear to the ground

Prometheus and Grafana have served the Kubernetes observability need (and observability in general, for that matter) quite well.  As IT infrastructure grows, the need for new solutions to address concerns around long-term storage, query speed, cardinality and hybrid observability will be increasingly apparent.  All of the contenders in this space are changing and improving rapidly, so whether you’re just now considering Prometheus/Grafana or are running it with hundreds of Kubernetes clusters, it pays to keep monitoring this conversation.