Prometheus-native monitoring: Availability and reliability

on June 17th 2021

The world of monitoring has fundamentally changed. Today’s monitoring tools were not designed for the complex, dynamic, and interconnected nature of cloud-native architecture. Companies need a monitoring solution that is as scalable, reliable, and flexible as the cloud-native apps they need to monitor.

Prometheus’ single binary implementation for ingestion, storage, and querying makes it ideal as a light-weight metrics and monitoring solution with quick time to value — perfect for cloud-native environments. But simplicity and ease has its trade-offs: As organizations inevitably scale up their infrastructure footprint and number of microservices, you need to stand up multiple Prometheus instances. This means you’ll start to deal with high availability and data locality issues – all of which require significant management overhead. 

Why Prometheus-native and what does it mean?

Because of the time and challenges involved with running your own Prometheus deployment, many organizations are exploring moving to a hosted, or a managed, metrics and monitoring SaaS offering. To ensure a smooth transition to the solution and avoid future lock-in, it’s critical that the SaaS monitoring solution be fully Prometheus-native. There are specific capabilities you should look for in a Prometheus-native monitoring solution, including: 

  • Prometheus ingestion protocol support.
  • 100% PromQL compatibility.
  • Prometheus Alert Manager definition support.
  • Prometheus Recording Rule support.
  • Grafana dashboard support.

With our working definition of Prometheus-native basics in hand, we can dive into a discussion about requirements for the monitoring solution itself. 

Prometheus-native monitoring: 4 must-haves

This four-part blog series explores the key capabilities that organizations should consider when selecting a Prometheus-native monitoring solution, covering: 

  1. Availability and Resiliency (current blog)
  2. Cost and Control
  3. Security and Administration
  4. Performance and Scale

Availability and resiliency requirements

Let’s start by talking about a primary reason you’re investing in monitoring in the first place: ensuring performance of your cloud-native apps. In order to ensure your infrastructure, applications, and services are performing optimally, you must have highly available and extremely resilient monitoring. For this reason, it’s important that your monitoring is more highly available than even your critical production apps. Here are some important requirements you should consider.

High availability and reliability

Under the strain of increased data volume and cardinality, monitoring systems become unreliable: They lose data or experience downtime. Without monitoring, teams are flying blind and won’t be able to respond to issues in real time. To make sure this doesn’t happen to you, pay close attention to the availability and reliability of your SaaS vendor. 

Service level agreements (SLAs)

When discussing SLAs, most people jump straight to “How many nines of uptime does the vendor offer?” This is a critical question as you’ll want to have a monitoring solution that is more highly available than your production environment, meaning ideally you’ll be looking for 99.9% uptime. 

Beyond the actual number itself, it’s important to look at how the vendor defines and monitors the SLA and at what point they notify customers. A best-in-class solution will proactively monitor their own systems for downtime and count any period greater than a few minutes of the system not being accessible as downtime and immediately notify customers.

SLA checking 

Proactive SLA checking is a great start, but a simple ping check against an endpoint does not tell you much about whether the system is performing as expected — it only tells you that it is returning 200s successfully. A proper SLA check for a hosted monitoring solution should check the basic read and write paths to ensure that data is persisting as expected and no data is lost.

Dedicated endpoint 

One of the leading causes of downtime in SaaS monitoring solutions is due to noisy neighbors. This is when another tenant in a multi-tenanted SaaS environment impacts performance of other tenants. To avoid this, you would ideally want a dedicated environment with dedicated endpoints that are not shared with other customers.

Cloud provider choice and circular dependency protection

Most engineers are familiar with the concept of a circular dependency — when A depends on B, but B also depends on A — but most people don’t think about this in the context of their SaaS monitoring service and production. 

If your production applications are hosted in the same cloud provider and region as your SaaS monitoring solution, you’ve got a circular dependency. The primary risk is that if there is a service disruption in that region, you could have a simultaneous outage of both your production environment and your monitoring solution. This is one of the worst possible times to incur an outage to your monitoring solution as it is the system that is meant to inform you of your production outage. Best practice is to host your SaaS solution in a different region, AND with a different IaaS cloud provider, if possible, to avoid this circular dependency.

Next time, in part two of this series, we will explore considerations around cost and control. Stay tuned.

Other blogs in this series:
Prometheus-native monitoring: Cost & control

Prometheus-native monitoring: Security & administration

Other resources you may be interested in

The first monitoring solution purpose-built for cloud native deployments.