Conference recap: Learning From Incidents (LFI) 2023

On two snowy days in February, a horde of self-professed incident nerds descended upon Denver for the inaugural Learning From Incidents (LFI) conference.

It was a magical gathering of industry folks, resilience engineers, technical leaders, and academics sharing tales of helming incidents, strategies for seeding a culture of continuous learning and discussing the latest resilience engineering research findings.

This community began in 2019 and is “challenging conventional views and reshaping how the software industry thinks about incidents, software reliability, and the critical role people play in keeping their systems running,” according to the community’s website.

I myself was first introduced to the LFI way of life back in my first SWE role when my company hosted the SNAFU Catchers. The researchers were studying anomaly responses, particularly — how ChatOps controlled the costs of coordination during incidents and facilitated recalibration aka “I didn’t know it worked that way”!

Their presentations shifted my paradigm of “the system” from a purely technical perspective of code, configuration and infrastructure to include people and the human side of the organization, what we call sociotechnical today.

Their research led me on a journey following LFI luminaries Dr. Lorin Hochstein, Nora Jones, J. Paul Reed and many more. Tweets like Amy Nguyen’s below made me laugh, think, and want to learn more about this whole LFI philosophy.

A few years — and many incidents later — I was an SRE at a company that experienced an incident dubbed “Kafkapocalypse.” If there was ever an incident to learn from, it was that one! Instead of following the same post-incident review process, I donned my analyst hat and interviewed the responders, reviewed the available data, and synthesized it into a report that highlighted the uncovered themes and knowledge gaps. My first incident analysis! The context gleaned from the incident analysis was a goldmine of insights and I was hooked on this new approach to incidents.

This year’s conference included 5 main tracks that attendees could join to learn about incident theory, research, and real-life scenarios.

CasesConf: The Incident Story Track

You won’t find a recording of my talk (or any Incident Stories) due to the Chatham House Rule : Anyone who attends the meeting is free to use information from the discussion but cannot reveal who made any particular comment. This meant speakers could include specifics about architecture or company dynamics and not worry about litigation.

My presentation, “There’s No Place Like Production,” walked folks through my “big one” — when I was a contributing factor to a major incident for an issue that could only happen in production. I retold how this incident turned into a gift of learning to my former company.

The Intentional Hallway

This track was for opportunistic idea generation, lively discussions, and ad hoc collaborations. One of my favorites was the discussion facilitated by Jeff Martens around Managing Incidents When Your App is Built on Other Apps. The big question was: “How do third party cloud dependencies impact your ability to run reliable services?”

The ancient SRE saying goes: “You can only be as reliable as the most unreliable component in your system.”

This question spawned a juicy conversation that benefitted from having so many differing perspectives in the room from managers. The one point of agreement among the group was that vendor status pages are not the most reliable sources of information. We touched on the sometimes adversarial nature between vendor and customer and what reasonable expectations are.

While you can’t go back in time to be in the room for our discussion — ask folks in your organization this question to spark your own especially if you’re embattled in a build vs. buy debate.

Learning From Incidents Techniques

This track featured practical takeaways for individuals and organizations to learn from incidents and increase resiliency.

Organizational Learning

A current challenge for the LFI community is how to measure or quantify the impacts of investing in learning from incidents in a way that is sensible to business leaders. For this track, Niall Murphy’s video skit IC Meets VP in a Tale of Incident Management set the stage for a conversation between Mr. VP and Mr. IC, which will be all too familiar for those that manage incidents and operate complex systems.

We invested a lot in Incident Management last quarter, and incidents went up. Is that expected?” asked Mr. VP

“It’s not that we’re finding more incidents as such, it’s really more that we’re trying to handle them better and the investment really helped the team to do that!” said Mr. IC

“Really? Here it says that not only did we have more incidents but they also lasted longer on average,” posited Mr. VP 

The video closes with Mr. VP outlining the constraints he is under and what signals are needed from the Incident Management function to continue getting investment. He throws a challenge all of us will contend with investing in resilience work: ”Give us something that can be managed roughly the same way everything else is or lead a revolution in how things are managed”

Let the revolution begin!

Academic and Applied Research from Other Industries

Part of what made this conference unique was the blend of academia and speakers from industries beyond pure technology.

It was exciting to discover that there are actual fields of study such as safety sciences, human factors, organizational theory, and cognitive engineering with brilliant folks that study how we as humans can better manage and maintain complex digital systems that seep into almost every facet of modern life.

Ohio State University Professor Emeritus Dr. David Woods even gave attendees homework! It was to reflect on personal aha moments throughout the conference. He asked us to identify new patterns that became visible and asked, “What facilitates learning? What hampers it?”

Speakers from other industries served as a reminder of how our society relies upon complex and distributed systems and highlighted how critical learning from incidents is as a society. Highlights include tales investigating accidents with the U.S. Forest Service from Dr. Ivan Pupulidy and Nikki Vande Garde’s experiences as the Director of Patient Safety at Oracle Health wrangling electronic health records.