Off-Call with Paige Cruz: Is Mobile Observability A Misnomer? Part 2

Summary

• Triage and Understanding with Mobile Data
• Unplugging with Vinyl Records
• Walk, Run, Crawl for Mobile SLOs
• Building Bridges between Mobile and the rest of Eng

Hanson’s Recommended Resources

• DroidconSF 2023: Combating sampling bias in production
• OTel Community Day 24: OpenTelemetry for Mobile Apps
• O11ycast: Ep. #72, Mobile Observability with Hanson Ho
• Test Case Scenario: Mobile App Testing Automation

Paige Cruz: Welcome back to part two of Off-Call with Hanson Ho of Embrace. Today we explore what it’s really like being on-call for mobile incidents, how to prepare for the unexpected, and why triaging issues in this space is as much about asking the right questions as it is about the code itself. Hanson also shares some solid advice for those looking to break into mobile. Plus, hear how open source efforts within OpenTelemetry are raising the bar for visibility for mobile engineers. Let’s dive in.

[00:00:34] Investigating Mobile Issues

Paige Cruz: We’re gonna turn a little bit back to learning more about you and then we’re gonna dig right back into mobile. We do want to meet the person behind the pagers.

Are you still on-call today?

Hanson Ho: Yes, as a startup. With mobile, on-call is less- yes I am. But I don’t expect to be paged. I am not on pager duty. because incidents don’t happen through mobile apps or SDKs like that.

Paige Cruz: It’s more be on hand as big changes are going out. Be here when the big stuff is rolling out.

Hanson Ho: Yeah, it’s like level 2. If there’s like a level 1 sometimes I get pulled in, it’s not that often. It’s usually the “Oh, something isn’t working quite right I think. Can we have a look?” So we have a rotation like that and I’m part of that.

Paige Cruz: Nice, like a support investigation. 

For folks who are interested in getting into mobile, what on-call advice do you have for them? If getting paged is not a frequent occurrence and you don’t expect it, oh my gosh, how do you prepare for the time that you are called in?

Because on the back end, we’re kind of like, “Oh my god I’m so fatigued, we get paged all the time” It’s an over alerting problem. So how do you prepare when the situation is you’re not really expecting that call

Hanson Ho: You try to figure out where the problem is by answering some basic questions. Okay. 

• Who is this affecting? 
• When did it start? 
• Did anything happen right before it started? 

Usually it is a problem that is transient. Some certain networks go down.

We’ve seen issues where for a certain city network is saturated, or certain countries decided to block certain services from use and suddenly, “Whoa. Look at that graph go down!” It’s “Oh yeah Turkey’s decided to you know— or whatever it is, right?

Paige Cruz: Stuff that your observability platform could never dream of pre-filling out or letting you know.

That is just so real world, you would have to have that intimate knowledge of what’s going on in your space.

Hanson Ho: And frankly a lot of times it’s out of our hands to address. It’s not something inherent in the mobile app to to fix and if it is generally there’s a feature flag. If you turn on a feature flag that caused some issues, you turn it back off. Usually we’re just bubbling up an issue that is happening in the backend.

Oh, so a certain endpoint or certain workflows are failing in a high degree for certain regions. In the backend you don’t see that because it’s only a small percentage of your total workflows and so it doesn’t dip below the alerting threshold.

And you basically say, “Hey, have you looked at the backend, filter it by these criteria.”

“Oh yeah, this did dip to 50%. Oh yeah some certain clusters are behaving weirdly”.

Generally it’s about asking the right questions to make sure that it’s not your problem.

And if it is your problem then you go in and try to figure out where in the app it is and who ultimately should do it. It’s rare that the on-call person from mobile receives a page that is actually on their component and that they can fix. It’s usually about triaging.

[00:03:21] What Observability Is All About

Paige Cruz: OK, this is great because what I’m hearing is if you are interested in joining a mobile team, switching to mobile, or starting your career there. When you’re joining your team, ask somebody to do a tabletop exercise of the last incident that happened, and have you go through the questions.Try and find who the right owner would be. That kind of thing sounds like it’ll prepare you for When, if and when, you do get paged. I love that.

Hanson Ho: I would say there’s a lot of institutional knowledge baked into mobile teams. And it’s not the best way to function as an on-call to basically have to get that either from a doc or from a person.

Which is why you want to have observability to basically have the data tell you where the issue is instead of you finding the code. If you actually have the proper instrumentation, know what’s failing, what part of what is failing, then you don’t even have to look at the code. And that’s what observability is all about.

That’s why we want observability on mobile is because we don’t want to have to answer questions like the backend- you can’t know every single microservice. You have to let the data tell you what is failing so you can actually talk to the people who own that. And that’s what we want on mobile ultimately.

[00:04:31]

Paige Cruz: Okay. Oh my gosh. I’m thinking about how horrible I am as a user and how hostile my mobile phone environment is. I run my apps all the time. iPhone just told me I hit my 200 tab limit on Safari mobile, so I am like the nightmare user. That’s probably the edge case, the like anomaly on every–

Oh, is this pretty standard?

Hanson Ho: The typical case- that’s why it makes things chaotic. People have habits that they’ve developed in the past. they kill apps, instead of letting-

Paige Cruz: With wild abandon. If it’s slow, if I can’t press a button, if I can’t scroll, you’re done. You are out. I’m like the OOM killer. I’m ruthless.

Hanson Ho: People used to also clear caches often because back when devices didn’t have a lot of disk space, they would clear caches because, downloading a lot of images, it takes a lot of space and they’re full. They got to install a new app. They do sideloading on Android.

You can’t prepare for all the weird things that happen leading up to the problem. Sometimes the problem is fatal but only happens if you go through these certain sequences of things which is why production data is vital if you have a big enough app and that you care about everybody.

You can’t simply rely on your synthetic tests.

[00:05:41] Generational Differences wrt Tech

Paige Cruz: No. Do you all see a difference in usage across generations? Do younger people use their phones differently than Millennials versus Gen X?

Hanson Ho: Unfortunately…

Paige Cruz: Oh how would you get that data?

Hanson Ho: Yeah. So I’m certain some companies have this, some companies with very big market caps and 100 mobile engineers and 25 working on mobile performance alone.

Those companies might be able to tell you especially if it optimizes their ad delivery. Twitter did not have that.

Paige Cruz: Okay. Yeah. Because I’d been seeing things lately about how Gen Z and Gen Alpha, because everything’s been cloud based, they don’t know about the save button. They don’t know about locally storing stuff on disk.

And you’re like, “Whoa, we’re all using computers. Y’all are the future of tech. We gotta go back to some basics here. You would think they’re more tech fluent but it’s not necessarily the hardware, knowing about devices. It’s more about the app layer, the cool stuff you can do with apps now.

Hanson Ho: Every generation has a different expectation like I know people coding today starting, Pointers? What are pointers?” So it’s the same.

[00:06:48] Our Shared Apprecation for Vinyl Records

Paige Cruz: Getting paged and being on-call even if it’s a rare occurrence, and even if you’re not the person necessarily on the hook for fixing it, it’s still stressful.

You still gotta put in your time and you gotta make sacrifices for that. What are you getting up to when you’re off call? What does on-call take you away from?

Hanson Ho: Back when I was really on-call I didn’t have any kids so the things that it was taking me away from were like pursuing all my hobbies.

Paige Cruz: Hobbies, yeah.

Hanson Ho: Right now my main hobby that I have time for is listening to music again in a much more deliberate way. We all when we code we just put on the headphones and we have whatever Apple Music, Spotify running and you just play an album over and over again or some people do playlists.

If I want to listen to an album, I’ll listen to an album I just do a repeat all the time.

Paige Cruz: Love that.

Hanson Ho: But it’s passive. It’s in the background. You’re not fully paying attention to it.

So what I’ve been doing recently is- I got myself an entry level record player and I started buying all my old albums that I liked on vinyl and also new albums too if there’s special or whatever editions.

Paige Cruz: So much is getting pressed! I had no idea that vinyl was still well and thriving today.

Hanson Ho: It has exceeded CD sales I think last year in dollar amounts. I think this year there’s an article recently that talked about vinyl volume being down sales year to year. It’s just the cost of vinyl has also gone up because of the special editions.

Records are anywhere from like-

CDs used to have a very common price new releases were like $12.99 whatever and if you buy old catalog releases $17.99 it was on sale it was $9.99

Paige Cruz: It didn’t hurt you were like, that’s fair. That’s like a dollar a song. That’s fine.

Hanson Ho: Yeah! But vinyl these days, a brand new vinyl, it could be a special edition, it could be a regular edition. You get it for $30 Canadian or it could be $50. It just depends on a number of things which are completely out of my control. Vinyl is becoming more of an enthusiast, or rather, it’s getting a new group of enthusiasts.

So for me, it’s about the process of putting the record on and then putting on the headphones hearing the crackle and then paying attention. And I’ve upgraded my audio equipment slightly.

Paige Cruz: It happens. It happens. Every year that you’re into vinyl, the upgrade fever hits.

Hanson Ho: For me because I’m coming from a background of tech I like to experiment. I like to know the A/B tests I control. So for the same album, I’ve changed the source, I changed the headphones, I changed the amp, I tried using Bluetooth codecs.. Most of the time I can’t really tell the difference. The biggest difference I’ve been able to tell is using open back wired headphones being a huge difference.

Paige Cruz: I was suspicious. I’m like, how much, had these nice headphones I got from an old employer. They’ve been working great the Sonys and I put on the open back and I was like I’m being transported to another universe and this song is the only thing that exists. It makes a difference.

Hanson Ho: And not being able to repeat tracks, having to like, I gotta go play the record. I don’t do it during the day, I do it at night when I have my chips and candy or whatever. I put it on and listen to it.

Then I would hear- the last album I bought was Veruca Salt’s American Thighs I bought the Newbury Comics reprint, hot pink vinyl. I have to get it from the States, it was more than I want to pay for but whatever.

Third album I bought ever on cassette now I have it on vinyl and CD and there are parts of it that are so much clearer. Maybe just because I’m just paying attention to it sitting and not like in the background. Oh I didn’t notice this! And it’s a record I’ve literally had for, oh, almost 30 years, if not exactly 30 years.

Its a fun hobby. It has made me not think about work or other stuff. I’m just listening to the music.

Paige Cruz: It’s more present and you have to listen to everything as the artist intended in the order.

There’s no skipping the songs you don’t love. That’s what I have found. It’s brought me a greater appreciation for albums that I’ve had for so long, but I’ll just skip. I had an iPod in high school. You just skip that stuff you don’t like, and then you’re only listening to the same three songs over and over till you’re sick of them. Vinyl’s brought a lot for me to say, “oh, this artist was telling a story- they were creating an entire milieu to exist in and even if I don’t love the song, I should respect the song’s context.

Hanson Ho: Track ordering is so important. I used to make year end mix CDs where I’d just pick out 50 favorite songs and burn CDs. And I would select different artists, different albums I like, and I wouldn’t pick songs that were too long. I would do the transition so that you don’t just go bang, bang, bang, heavy, heavy, heavy, and then slow. I want to, take it up, take it down, take it up, and it’s almost like a sushi omakase. You want to sequence. It’s not just about what you give. It’s about the sequence in which you give it.

I do believe there’s an art to doing track orders and whether you change side A from side B. Maybe A has a different vibe than B or you have double LPs.

Paige Cruz: Oh my gosh.I read this book that was written in the 70s. It was like all about technological change, which is funny to think of in today’s context. One of its thesis was: as we get more abstract and a lot of our stuff is on screens and technological, we press a button and then things happen that we don’t even see. That we crave the physical, we crave the analog. We need to rebalance and for me, vinyl totally fills that need. You gotta sit and you gotta listen and you gotta enjoy.

But it is an expensive hobby. The headphones and then you’re like, “Ooh, the little stylus, what if I just got a better one?” And then you’re like, “What’s the next one up? What’s the next one up?” And the speakers? Don’t even get me started on speakers. They cost as much as a Bentley.

It’s nuts.

Hanson Ho: I’m only doing entry level for everything because I

Paige Cruz: You gotta start with the basics.

[00:12:16] Walk, run crawl for Mobile SLOs

Paige Cruz: Oh man so I want to wrap up digging a little bit more into mobile. Earlier you mentioned setting SLOs. I’m curious, aside from the e-commerce use case where you’re really concerned about checkouts and that flow from browsing to checkout, what are the things you need to think about to set effective SLOs in such a chaotic environment? How do you know that it’s not just one person with an overtaxed device on a bad network that’s causing the issues? How do you know what’s real and separating the signal from the noise?

Hanson Ho: So you basically want to start as a high level as possible before getting into the weeds and the details. First you have to know that something has happened or not happened then you add the context. Its important to track things like the metadata around a particular workflow,the user, some identifier for the app install, session. Things that basically bring you additional context. That’s when you can basically tell “Oh yeah, it’s this repeat failure from this one device or one device model or one OS version.”

So you’re able to basically slice and dice at your whim without having to re-instrument. Mobile is something that gives and takes. The easy part is that, relatively speaking, it is a simple process. It’s mostly one process with a handful of threads with a small number of cores. The hard part about it is that you don’t control any of that. The OS decides to give you the scheduling. Even assigning which core your thing runs on is dependent on the OS. Background threads are deprioritized compared to the main thread. Things like that.

There are many variables, but they’re small. There are many variations to the combination of variables.

Paige Cruz: The cardinality is high.

Hanson Ho: Cardinality is extremely high but what you can control is actually not that much. On the good end is there aren’t that many places you could be screwing up. On the bad end, there could be a number of reasons why you could be screwing up.

Paige Cruz: The why is-

Hanson Ho: The why is hard and the hardest part is to basically capture all the necessary context and not do it in such a way that it overloads each signal you send. So how do you efficiently capture the telemetry imbue it with the context and also encode it in such a way that it’s efficient and you’re able to send it to the backend in a way that’s not going to blow up your data charges or require too much serialization which then takes away from your actual app doing your actual things?

Because anytime you have overhead and on mobile performance is, because the resources you’re given are so small, the overhead for instrumentation is quite high. So doing it improperly: capturing too much data, sending too much data, all that stuff could ironically drag down the performance of the thing you’re trying to measure.

Paige Cruz: Can cause someone just to say, “nope, you’re running up my data bill. What is this?”

Hanson Ho: Instead of trying to capture every minutiae first capture the high level of success/failure and then add context to it. Then you can start selectively understanding what is important, what is not important and add additional telemetry based on that.

The most important thing is context. I know I’m repeating this, but context, context, context. That’s the most important thing. Maximum slicability ,infinite, hopefully not infinite, super high cardinality, not being afraid of cardinality.

Paige Cruz: Don’t! Spend it where it counts.

That’s what I say is cardinality in and of itself is not a bad thing. It got a bad rep because the tools of yesteryear couldn’t keep up with it and it caused a lot of expensive bills but it is just a measure of how much uniqueness is in a set of data. So If that uniqueness is helpful to you, you should totally be paying for it.

If it is not helpful ruthlessly cut that out because you’re paying for stuff you don’t need. But it’s not bad, it just is.

Hanson Ho: And sometimes you can have high cardinality but you don’t need to necessarily aggregate by it, right? When you care about a certain device, like a device ID, you want to track what’s happening on a particular app install. You’re not aggregating by that, you just want everything associated with it. So you do have to persist it in a way such that aggregation is possible, retrieval, fast retrieval is possible. Or can you just say, “you know what? I can index by it and then I can query for it. That’s good enough.”

[00:16:45] Platform Specific SLOs: Yay or Nay?

Paige Cruz: We talked a lot about Android and obviously Android versus iOS was the main players in the space. I keep hearing murmurs of Linux on mobile every so often. It’s a small group. Where does it make sense to separate telemetry or dashboards or views or even SLOs by which platform?

And when does it make sense, “oh no, these measurements, it does not matter where they’re running, it’s just helpful to be able to facet by one or the other.”

Hanson Ho: So technically, I think, I want to say all iOS and Android both run Linux kernels. So technically Linux is on mobile.

There’s several stages of your journey. In the first stage we’re just wanting to understand what is happening there’s not a lot of differences technologically, there’s things you instrument will be different, but you want to track very similar things regardless if you’re doing, Android, iOS, or, Kotlin multi-platform, or React Native, or Flutter. Or whatever write once deploy multiple platform thing that you have.

Paige Cruz: Oh yeah.

Hanson Ho: Which is lots of those around, those are important too.

Paige Cruz: I understand the desire, but I like something that’s purpose built. I like to use things designed for the platform they’re on. That’s a personal opinion, I’m not a business owner, but this is my podcast, so I’m going to say my opinion.

Hanson Ho: My favorite cross platform thing is the browser. If you’re gonna do cross platform, do browser, but that’s me.

Paige Cruz: That’s fair.

Hanson Ho: Make a really good mobile web experience is my thing.

Paige Cruz: That’s the main takeaway.

Hanson Ho: Exactly. To be able to know, understand, and then explain what’s happening in each platform, you do have to go deeper. But before you run, you have to walk. So the walk is really about capturing the context of what is happening and then just add some basic instrumentation. Whether it’s just simple traces from very important workflows, whether it’s user visible or not user visible get that first then add on to the instrumentation to give you a little bit more information.

If you want to capture a platform specific things like platform API calls, and obviously you have to build within the platform, but those come after you’ve solved your first big bags of problem.

Paige Cruz: Can this person write a post? Can they look at their profile? Can they log in? Log in, even?

Hanson Ho: Yeah, Also make sure that you do it on the client side and track that. Just because your backend doesn’t see it doesn’t mean it hasn’t happened. Making sure you handle that and make sure that the data when it’s captured is delivered.

If you don’t have the network at the moment of export, you’ll have it sometime, so make sure it’s properly cached. When you go and furiously close apps, make sure that if telemetry is being sent at that point and you’re closing an app, that is not fully lost. That is somehow, you retry, you get a confirm and whatever it is.

Doing the basics right first will get you 80% of the way and then you talk about how do I add additional information to this?

Paige Cruz: What more do I need now that we’ve got a strong foundation.

Hanson Ho: Correct.

Paige Cruz: Oh my goodness, I cannot believe we are almost at the end.

I have learned so much, I have so much empathy for mobile engineers. Will it change how I use my app and close things with wild abandon? No, I am who I am. I’ve built these habits up over time. But it is good to know I’m one of a big crowd. I am not an atypical mobile user. Y’all are used to even more extreme conditions than what I put my iPhone through.

[00:20:05] Using OTel to Remix Instrumentation

Paige Cruz: We talked earlier about the fool’s errand of a single pane of glass- that observability platform that can stretch to fit every single engineer’s use need,, and domain. So where do you see Embrace fitting in with a company’s overall five different observability platforms or tools?

What is the ideal place that Embrace has and how do you hope that it’s being used?

Hanson Ho: So the Brits have a phrase called horses for courses and I believe that.For your mobile developers who want to use mobile specific optimized UI, having the Embrace dashboard is the best thing to do.

It optimizes for your use case. However, and very underline, double underline, however, is that the data that we actually record has to mingle well with the data that is captured in the backend, which is why OpenTelemetry exists. The consumption might be specialized for certain users. I do want to have that data add to the single pane of glass that the backend SREs use to add context to it.

Whether you use Embrace, whether you use another mobile SDK to capture data in OpenTelemetry, it should provide additional context to inform what is being used. What is generating the requests that is causing your super slow traces? Maybe there’s some context in the client instrumentation that’s not represented because you’d have to break out the request body to find out that, “oh yeah, it was using a very specific ad campaign” or something like that.

That data is readily available on the client side. We can actually have a session that says ad campaign is blah and then you’re going to be able to suck that in without having to worry about all the processing costs. It’s just going to be there. So adding color to your existing telemetry is an important role of mobile telemetry.

And Embrace with OpenTelemetry is going to enable that. Everybody wants to use it differently but if the data is the same we can do remixes on it.

Paige Cruz: Totally. I think we will be successful, the royal we as an industry, will be successful when we have somebody- like if I’m a backend engineer able to go from, “ah, something kind of funky got flagged for me” I’m investigating I go to Embrace or the observability platform of mobile engineers and I followed a breadcrumb of data to get there, I’m looking at their dashboards, or their query interface and I can make sense.

I can at least start to do my own little research because with our big distributed systems, I think you’re responsible for the things you own and you’re on-call for but I call it the service neighborhood. I got to know my upstreams, my downstreams, who I’m affecting, and I should be able to go into the tooling and at least do some cursory looks and make sense of things.

We’ve got a ways, we got a little bit of a ways to go. I’m watching the CNCF group that is working on the unified observability query language. I will be very curious to see how all the vendors handle that when we get a spec landed. But that’s my dream at least, is that while we have specialized data instrumentation and views within different tools that you can just hop around with ease and still find your way around and just learn more about what the heck is happening in these wild systems that we have today.

[00:23:25] How to Be A Better Partner With Mobile Devs

Paige Cruz: So our final questions, what can folks on the back end, front ends, operations, security do to be better partners to mobile engineers? Aside from listening to this episode and just learning. What all y’all are dealing with over there? What, what would make folks, what would make a mobile engineer’s day?

Hanson Ho: Teach them. Mobile engineers, we talked about this before, live in silos. Everything is different. Pipelines are different. Everything. The deployment is different. We’re not used to the same type of observability and instrumentation that we have in the backend. Go gently on them when you talk about traces and spans and they don’t know what you’re talking about and you’re like “you do know what I’m talking about these are just new vocabulary that is being introduced” So observability, we talked about, there are words that we take for granted like cardinality, span. Those are not words that a lot of mobile developers are familiar with at least not one without a background in backend.

A lot of them come from code camps and they come straight out of school. The first job they had was work on React Native app. Their experience is limited so be gentle. Teach what you’ve learned and then they will absorb. They just haven’t been introduced to this.

Paige Cruz: Yeah. The gentle introduction. I love that. And to acknowledge like that is the price of siloization if you’ve got to be more intentional about building those bridges.

[00:24:47] Every Mobile Request Is A User

Paige Cruz: Now, what could mobile engineers do to be better partners to the rest of engineering?

Hanson Ho: I think giving folks, the backend folks, a customer’s perspective. So when p99 on the back end means some anonymous server or cluster is behaving at some level that is not acceptable. For users, or at least client side instrumentation, p99 is 1% of all users and it’s generally the ones that are not experiencing the apps in an optimal manner. And they tend to keep being the ones who because of their slow phone or bad internet connection, they keep experiencing this. So saying that, “Oh, it’s just p99. p50 is fine, right?” Be gentle with the back end folks. Every request is a single user and a lot of times it’s the same user sending a lot of p99s and p99.9s.

We care about that. The edge cases because these are not edge cases. These are potential users who churn. They are much more- they’re people. There are people behind clicking furiously, changing apps and things like that. Understanding that and just knowing that p99 means something very different for mobile than does on the backend will help ground the panic and urgency.

“It’s only 1% of users experiencing this.”

Yeah but they might uninstall the app and imagine losing 1% of users overnight because of a stupid mobile bug. Not great.

Paige Cruz: No. Oh my gosh. So I am hearing share your world between mobile and other types of engineers, share your world, break down those silos, make a friend, empathize.

On-call is not fun or easy for everybody but we are, I believe, all working to make this better day by day in OTel. And thank you all for the work that you have done to contribute to OTel and bring us into the beautiful future that you and I know can exist with observability.

[00:26:48] Wrap Up

Paige Cruz: Thank you so very much. Is there anything you would like to share resource wise with folks? I’ll have everything in the show notes, but any last pearls of wisdom or a good conference talk you’d love, you’d like to leave folks with?

Hanson Ho: If you go to conferences about observability ask for mobile content.

Mobile content is not the most appealing on the surface to a bunch of SREs, like we talked about observability, but this is an emerging field and we would very much like to speak at more conferences and talk about this stuff.

Paige Cruz: Submit talks! Encourage mobile engineers at your company to submit talks about the cool stuff that they’re up to, the lessons they’ve learned, challenges they’ve faced.

Hanson Ho: Oh, sorry. Join the CNCF Slack and join the various mobile focused groups SIGs, there’s an Android one, there’s a client one, there’s iOS, Swift.

Get involved is basically it. We need as many people as possible. I work on the Embrace SDK, but I also help out in the OpenTelemetry AndroidSDK. The only way this works is if you’re a bunch of different people, have a bunch of different applications, all writing to the same spec.

So we want help in all aspects.

Paige Cruz: Yeah, contrary to some of the criticism, we actively seek end users and folks who do not work for observability companies. That is the beauty of open source. So we look forward to seeing you in the CNCF Slack and the OTel community and thank you very much.

Hanson Ho: Thanks, Paige.

Paige Cruz: My big takeaway is that it’s important to invest in asking the right questions to set up a solid foundation for observability where your data can be the guide. And that it’s very important we all keep working towards a future where mobile observability is as robust as it is for those of us on the backend.

And now, a word from our sponsor Chronosphere. 

Let’s talk about something exciting! Chronosphere has launched Differential Diagnosis, also known as DDx, and it’s here to speed up your investigations with a single click. It might feel like magic, but let’s be real, it’s just some very clever math behind the scenes.

Here’s how it works. With one click, DDx analyzes hundreds, thousands, all of the spans and span dimensions we have that are tied to a particular service, endpoint or workflow. DDx spots patterns and ranks the top probable contributors to an issue and lets you compare the current state back to when things were running smoothly.

Is the issue tied to a particular region? A release? Are the majority of errors stemming from the same upstream service or third party? DDx lets you quickly run through these hypotheses and more, all from the same screen. No tabs or clicking away required. If you’re curious to see this in action, head over to Chronosphere.io or hit the link in the show notes to go directly to the demo. Thank you Chronosphere.