Meet the Chronosphere Team: Jared Jenkins, Member of Technical Staff

I’m a Member of Technical Staff. We have a relatively flat software engineering organization.  I work on the limits team, and we’re part of the larger storage organization that works on M3 and Chronosphere’s control plane. Our customer is the central observability team, and that’s who we optimize for.

The limits team helps observability teams manage their budgets, and, informally, we own a lot of the ingestion flow. When customers emit a metric from their data center, we’re responsible for making sure it gets all the way into the database.

I was looking to go back to a smaller company after being at Google for almost four years. Two big reasons helped me decide to come to Chronosphere: one was scale and the high bar of correctness and reliability in this domain. Some of our largest customers are writing several million metrics per second. You really have to be thoughtful about how you build things – and you don’t often get that with a smaller company.

The other reason is every engineer has gone through some frustration with poorly performing observability tools. It drags your whole process down and introduces a lot of toil and outages. It’s great that we’re bringing people on because our customers are getting better uptime when they make the transition to Chronosphere. One of our interesting challenges is creating the systems that allow us to maintain that reliability as we rapidly build new features.

I worked at Google on a team called Ads Integrity – which was a couple hundred people. The whole team in general is trying to fight abuse in the ads domain, which is actually very complicated. People try to use the ads network for all kinds of fraud, financial abuse, trademark violations and even malware. It’s a very topical thing these days. So, Google spun up an entire organization around just trying to clamp down on that problem.

The team that I worked on was an infra team that built a very big pipeline that other developers could use to build ML (machine learning) models and enforcement tools. But it has a lot of the same problems that observability teams have with our pipeline, right? There are many consumers (downstream teams) that want to put data into a shared pipeline and store it. And the costs can get wildly out of whack. So, you really care about building a platform that people can understand – like “this team is using this much of the budget” or “they added this new thing, how do we control that?”  It was a really cool team, and I enjoyed it a lot.

I would say the whole company is doing a lot of really interesting work. With the number of engineers and things that we’re doing, it’s pretty cool stuff. I think in observability today, it’s way too easy to emit data. And that’s great. I think that most engineers are encouraged to go through the thought process of “Hey, let’s add more metrics, let’s add more traces.” But the problem is, there’s this weird imbalance of how valuable the data is vs how much it costs. And it puts a lot of burden on the COTs and SREs who are trying to maintain this license for their entire organization.

They either have to spend more money, or spend a ton of time hand holding teams to get them to emit less data or be more efficient. Google tried to do this with Monarch and StreamZ. If you look at how Prometheus works, it looks a lot like those technologies. But, they’ve only solved it at scale for Google. And in some ways, they solved it in a different way than we are at Chronosphere. We are now trying to give this control plane to our customers who may look at the problem very differently – depending on their team dynamics and organizational needs.

Google’s engineering culture and their processes and tools are sort of fixed at this point. They’ve gotten to the point where, after 20 years of development, this is how you’re going to do something—there is no other way to do it. But if  you look at Chronosphere’s customers, they don’t work that way. You can’t just tell them, “Why don’t you do it this way?” because they don’t have the same requirements from their observability stack.

So I think it’s fun being part of Chronosphere because you have to get familiar with different businesses and really make sure that you’re building something that is useful for different use cases.

I’ve had my hand in quite a few things since I’ve joined the company, and I think that speaks to the size of the company – just the amount of impact that anybody can have in a company of this size. When I first started, I worked on our first-run of label caching – which basically makes searching for metrics and building queries faster. This is a big pain point for large customers who have hundreds of thousands for metrics. And the other thing I worked on was rearchitecting our limits service. We have a service that controls whether or not you’re over your quota. We onboarded some new customers and went through this whole process of “how do we actually make that thing scale?”

Soon, I’ll actually be focusing exclusively on the Collector. The Collector is effectively our agent that customers run on our behalf. We give them the software, they run it and it allows them to collect traces and metrics. In the near term, we’re going to be focused on making it easier for our customers to be able to debug stuff, which also helps our sales and account management folks. We don’t have to hand hold as much, which is really important as we scale as a company. Our customers are also engineers who want good debugging tools.

I’m also really excited about being able to bring the control plane closer to the customer as well. There’s a lot of cost savings that we can provide our customers – for instance, the ability to control the data flow even inside of their own data centers.

I’m actually an Eagle scout. In Boy Scouts, it’s basically the highest award. I spent a ton of time camping when I was a kid, and it’s something that I’m really looking forward to once my kids get a little older. It’s a great experience to get outside.

I’m trying to get more into the habit of writing local benchmarks with Golang. Golang has this fantastic set of tools for benchmarking and a wonderful standard library. Writing local benchmarks is a great way to test things out locally before you get them into production. The earlier that you catch an issue, the cheaper it is to fix.

I would be a Shitzu. I used to have one, and I know it’s a little specific, but they’re just great companion dogs. They get a lot of love (I would obviously want to be a well-loved Shitzu.) I think I would sleep most of the day, and in this theoretical situation, be getting plenty of treats.

Dan Post: he was my manager at Motiv – the company that created a heart rate tracking ring. Dan was a patient person,also very kind, and just incredibly technical moving around from server to mobile client to firmware. He’s someone that didn’t have a formal CS degree. He helped me get over my lingering imposter syndrome and gave me the confidence to tackle anything. I still talk to him every once in a while just to catch up.

A quote from my old manager, Zach Brock, at Square. Earlier in my career, I’d get frustrated at the state of things – because I care a lot about code health and operational excellence. He would say “Whenever you get frustrated, assume that the person making that decision had the best intent given the information at the time.” I think that brings humility to the problem; it makes you realize that things are not always easy or straightforward in the moment. There’s a human aspect to work.

The second aspect that it made me realize, is that a lot of technology is fungible and most engineering problems have several approaches to get there. The hardest engineering challenges don’t have a clear solution. It’s always worth reevaluating what you’re doing and why you’re doing it. That has helped me be really successful, realizing that a lot of decisions are made in the moment when they need to be. They’re not made perfectly all the time and that’s totally fine.

His other favorite phrase was: “just fix it!”. At the end of the day, you’re always going to be frustrated with stuff – just take ownership and fix it. At Chronosphere we give engineers the flexibility and autonomy to fix issues that they find, which is an important facet of a healthy engineering culture.