Events

Chronosphere at PromCon Online 2021 – Replay

Location: Virtual | PromCon Online 2021

Auto-suggesting and generating recording rules for Prometheus

High cardinality metrics often cause alerts and dashboards to time out when they try to fetch too much data. Prometheus provides recording rules to speed up queries by pre-generating the queries, however, they have to be configured manually and require reconfiguring alerts and dashboards to point to the recorded series. The performance degradation often happens as new metrics are introduced with more instances or deploys and a working query may break all of a sudden.

In this talk, we will show you how slow queries can be preemptively detected and automatically sped up without any manual reconfiguration. This is done by automatically analyzing the widely available inbuilt Prometheus query log and generating suggested recording rules for frequently queried metrics that take considerable time to execute.

We’ll walk through a concrete demo of the tool which can also use parameters min-query-time and min-query-count to help suggest the most impactful recording rules.

Auto suggesting and generating recording rules for prometheus at PromCon Online 2021.

Speakers

A man participating in a Dev Talks discussion at a restaurant table, utilizing Smartly.io technology to monitor data through Chronosphere.
Shreyas Srivatsan

Sr. Software Engineer
Chronosphere

A man participating in a Dev Talks discussion at a restaurant table, utilizing Smartly.io technology to monitor data through Chronosphere.
Shreyas Srivatsan

Sr. Software Engineer
Chronosphere

Shreyas Srivatsan

Sr. Software Engineer
Chronosphere

A black and white photo of a woman smiling.
Gibbs Cullen

Developer Advocate
Chronosphere

A black and white photo of a woman smiling.
Gibbs Cullen

Developer Advocate
Chronosphere

Gibbs Cullen

Developer Advocate
Chronosphere

Gibbs Cullen is a developer advocate at Chronosphere and makes it possible for the community to understand the concepts behind Prometheus and using M3 as a long term storage, in addition to helping the community with best practices in alerting, monitoring and configuring their deployment of Prometheus and M3 in Kubernetes. Prior to Chronosphere she was a product manager on the AWS Data Lab team.