Introducing Shaping Impact for aggregation rules

As companies adopt and scale cloud native architectures, they see their observability data accelerate along with their costs. A recent Enterprise Strategy Group study found that 69% of companies said they are concerned with the growth rate of their observability data.

Paying more isn’t necessarily a bad thing; it comes down to the value the data delivers. Chronosphere’s Control Plane allows companies to optimize the value to cost ratio of their data using aggregation rules. With Chronosphere, companies like Snap, Doordash, and Abnormal use aggregation rules to optimize their data, which reduces their initial data volume significantly and allows them to manage their costs.

Given the dynamic nature of systems and changing business needs, the effectiveness of aggregation rules may change over time. Users deploy new services, add and remove metrics, and inevitably change the shape of the data. At Chronosphere, we want to ensure that customers can continuously optimize their observability data over time, not just on day one.

In order for observability teams to do this, it’s important for them to have visibility into the effectiveness of the aggregation rules they have in place over time.

With the launch of the new Shaping Impact functionality for aggregation rules, observability teams can now easily understand the value and effectiveness of their existing aggregation rules and continuously optimize their system, increasing their value to cost data ratio.

An observability team needs to be able to answer multiple questions in order to make confident decisions about what aggregation rules to keep, modify, and remove from their system. This includes:

• How effective the rule is. Is it actually matching to any incoming data points?
• How efficient it is. How well is it transforming incoming data into aggregated data points? 
• How it impacts their license. Is it adding to or conserving their persisted data license? 
• How useful it is. Are the aggregated data points actually being used anywhere?
• Whether any of these attributes have changed over time.

The Shaping Impact functionality allows users to quickly answer all of the previous questions, with a ranked list of least to most useful rules based on the user’s unique needs.

Say for example, an observability manager is focused on reducing the overall data they’ve stored in their system to reduce their costs. They want to ensure that all of the existing aggregation rules in their system drive value (i.e. are actually being used) and optimize for their persisted (stored) data license. They can sort by “Least Valuable (by Persisted Impact)” to quickly understand which rules are being used the least while adding to their persisted license.

Regardless of license spend, observability teams want to ensure the aggregation rules they’ve instrumented are working — as in the rules are matching to incoming data and effectively aggregating that data. They can sort by “Least Effective” to quickly understand which of their rules don’t match to any incoming data at all and therefore don’t provide any value.

Observability teams also want to understand patterns and trend lines in their data shape and whether a rule’s value has changed over time. For example, they notice a rule is not currently matching to any incoming data. They then ask:

• Was it effective at rule creation?
• Has its effectiveness fluctuated over time?
• Was there a specific point in time where that effectiveness changed?

To answer these questions, they can click on the rule to reveal historical data for shaping impact metrics. They can zoom in on any point in time of the rule’s existence for further analysis.

If after analyzing the value and effectiveness of existing rules, they find a rule that may need modification, they can quickly copy its configuration and apply any updates in the Chronosphere CLI, Terraform, or API.

What rules are not providing any value for my end users?  

The Shaping Impact functionality allows you to sort rules and identify unused or infrequently used ones, and see how they affect the persisted data of your system — and decide if you want to remove them.

What rules are not working at all?  

You can sort rules by efficiency and identify ones that aren’t matching to any data at all. You can use the historical data charts to understand if the data fluctuates over time or the rule is persistently ineffective — and decide if you need to modify the rule or remove it.

How well am I optimizing my system overall? 

As you identify the least valuable and effective rules and make the necessary changes, you can ensure that the remaining rules operate at maximum value for cost. You can use the existing maximally effective rules as templates to create new aggregations.

The Shaping Impact functionality gives observability teams full insight into the value and effectiveness of their aggregation rules to help identify opportunities to optimize. This transparency means they can continuously trust that Chronosphere is built to serve their observability needs over time.

In the near future, we’ll be introducing functionality that provides deeper visibility, including:

Shaping Impact Preview: Understand the value and effectiveness of new rules before they’re created to prevent breaking changes.

Full Shaping Impact Coverage: Shaping Impact functionality will be extended to all types of shaping rules.

Prava is a Product Manager at Chronosphere working on the Control Plane. She focuses on building tooling to help observability teams align the value to cost ratio of their data.