Tecton upgrades from DIY Prometheus to open source-compatible Chronosphere 

Whether your organization is small and just getting started in your market, or a large enterprise embarking on your cloud native journey, Prometheus is a great place to start for metrics-based monitoring. As an open source toolkit, Prometheus works out-of-the-box and can be initially run in-house and by a smaller team. Later, as your environment grows, and the inability to scale becomes a problem, it’s seamless to switch to a hosted Prometheus-compatible observability platform that is purpose-built for cloud native.

This was the case with Tecton, which is on a mission to standardize feature management in machine learning applications. When Tecton first launched, the team was using an in-house Prometheus setup to monitor its environment. As Tecton’s customer base grew, so did the scale of its environment, and its open-source monitoring system started breaking down. The company partnered with Prometheus-compatible Chronosphere once teams were spending more time firefighting monitoring issues than innovating.

Since their setup was instrumented from open-source Prometheus, the move over to Chronosphere was simple.

Even as a relatively small startup, Tecton’s monitoring system was still a source of major pain. The team was spending too much time on monitoring.

• Each instance of Prometheus was completely siloed
  • It was impossible to get a global view of the entire Tecton environment
  • During incidents, the on-call engineer had to switch between isolated Prometheus deployments, which made it time consuming for them to find the relevant metrics that could help them troubleshoot the issue
  • Any change to Prometheus had to be made to each deployment, and silencing a noisy alert was a manual change
• No long-term storage
  • Tecton was only able to store metrics data for about a week, and metrics would then drop out of memory
  • If a customer brought up an incident that happened more than a week prior, there was little Tecton could do to investigate
  • On top of that, the persistent volumes Tecton was using for storage weren’t reliable
• The system kept breaking
  • Their Prometheus setup was buggy, and the team had to drop key tasks to focus on fixing metrics problems

After building a custom way to get multi-regional support for Prometheus, the Tecton team realized they had outgrown their in-house system and it was time to make a change. “We said, this is not the right way to do it,” Trivedi said, a software engineer at Tecton. “We’re basically building what we know is technical debt.”

Tecton couldn’t properly meet business expectations by operating off of a broken monitoring system. The team was spending an inordinate amount of engineering time building custom features just to make Prometheus work, while also wasting time putting out monitoring fires.

The team had three key criteria as they set out on their search for a new monitoring solution. They required:

• A global view across regions and customers
• Control over data retention and the ability to keep historical data
• High availability

Tecton needed an out-of-the-box monitoring solution that wouldn’t break, and a feature set that wouldn’t need custom engineering work.

After comparing proof of concepts with other monitoring solutions, Tecton chose Chronosphere for two key reasons:

• Confidence in commitment to support
• Cost savings compared to the current approach

The biggest benefit of moving to Chronosphere is having an out-of-the-box monitoring solution that doesn’t break all the time and has a full feature set that doesn’t need custom engineering work.

Chronosphere’s cloud native observability platform provided Tecton with:

• Less burdensome on-call rotations, reducing noisy alerts and alert fatigue
• Useful data retention

With Chronosphere, Tecton no longer needs to think about monitoring as the teams spin customer deployments up and down. Instead of spending hours urgently debugging the monitoring system, teams spend their time focusing on Tecton’s core product and delivering on customer commitments.