Well, good morning. How is everybody? How many of you, with a quick show of hands, have ever used a streaming app to listen to music, a podcast, or entertainment? Get your hands up. Okay, keep your hands up if in the last five years you've experienced a glitch with that app. Oh, not bad. Pretty good performance.
When that happens, it's annoying, right? And it causes you to question, why have I chosen this app? Why have I chosen this brand? What am I doing here? Often we move on to something else. We close one app, we open another. And for those of us who are running companies or building companies, we know that incidents that impact your customer experience are on the rise. In fact, in the last year, they're up 43%. And we're talking about major incidents, incidents that stop you from doing what you're hoping to get done with the technology and services that you're using. As a leader, I can tell you that a major incident on average costs nearly a million dollars for a company, because we rely so heavily on technology at the edge or consumer applications to drive revenue, which drives our profitability, our operating margin, and enables us to fund innovation and growth.
So this has become a real business challenge, not just a technology challenge. And the reason I'm here, I'm Jennifer Tejada. As Martin mentioned, I'm the CEO of PagerDuty. We're a public company that's 19 years old, really built around helping customers, more than two-thirds of the Fortune 100, half of the Fortune 500, half of Forbes AI 50, mature their operations and improve their operational resilience through automating your digital operations. That's a lot of techy buzzwords, but at the end of the day, we help your organizations run their tech more effectively in service of the customer experience that you are spending all your time and energy creating, building, and providing to those folks who are either trading their data, their information, or their hard-earned dollars for your products and services.
And I'm thrilled to have with me today someone who I think of as a true expert as it relates to both innovation and operations and how you scale incredibly creative software engineering in service of end consumers to deliver a delightful experience. With me today is Tyson Singer. Tyson is the VP of Technology and Platforms at Spotify, a little streaming app that many of you may be familiar with. Tyson has the unique challenge of worrying about what, 650 million listeners on any given day. I mean, when things go wrong, a lot of people notice. Fair to say. Tyson owns all of infrastructure, technology strategy, developer experience. And this is something Tyson takes really seriously. We see across our customer base, particularly large-scale customers, that when you take developer experience seriously, you deliver better products and services, but it's a lot harder than it sounds.
So Tyson, while we're sitting here, I'm just curious because you've been in the business for nearly nine years.
Over that nine years, technology has evolved a lot. I feel like we need a word that somewhere lands between revolution and evolution. We've seen an incredible pace of change and an incredible amount of change. How have you thought about operations within your business and how are you leveraging AI to improve the quality of the products and services that you deliver for your end customers?
Yeah. So, I saw some hands go down on the like, have you experienced an app making you feel good. Well, not when their hands went down, because most likely folks are actually using Spotify. But we do have quite a resilient system. And so we can share a little bit about what we've done in the past to make things resilient and what we're doing today, because things have changed a lot versus when I started at Spotify nine years ago. I think everything first and foremost starts with the code. Bad code means more incidents, and you said incidents are on the rise. That's not happening at Spotify.
And maybe there's some reasons behind that. So right now what's going on is more and more folks are generating a larger portion of their code with agents, and that introduces maybe not poor quality code but certainly higher variability of the code.
And more code. So one of the things that has been very important, sort of pre-AI revolution and I think also during the AI revolution, has been to have a lot of standards in the process. When I joined Spotify it was this context that a lot of people know, the Spotify model, which was basically chaos. Just to be blunt, there wasn't any standardization. There wasn't a whole lot of structure around how we did things. And so my job was to really try to introduce more structure and more standards into the software development process so that we had code that would be high quality and reliable. A lot of what that meant was making the code more consistent, using the same frameworks, the same platforms and technology across the fleet of software. So today we have roughly in the order of 100 million lines of custom code, and dependent on probably about a billion lines of code beyond that. And you can't treat code like a pet anymore. You have to treat code like cattle, and that means you have to be able to transform code at scale.
Wait, did you catch that? You can't treat code like a pet. You need to treat it like cattle.
Yeah. So they said, back in the switch from how we treated servers, right? You used to treat servers like your pets and then you're like, no, we have to do this all the same. Well, we have to deal with our code as well. And so that means having tools that allow people to understand and set those standards and then transform code. So we use a tool that we call Backstage, which is our internal developer portal that helps us set all the standards across the codebase and manage the codebase in a very consistent fashion. And then we use an approach called fleet management to change the code at scale. And now in the context of having agentic capabilities and really high quality LLMs, we can change that hundred millions of lines of code at scale to address issues that have impact on resiliency, security issues, and even cost issues. We can just go across the entire fleet and do transformations of code, which is important at scale.
And it's much faster than in the past.
It is much more predictable.
It is more predictable, to some extent.
Yeah, with the LLMs, it's a little nerve-wracking when you're saying like, let's change everything very fast. So the blast radius can be very large when you're doing these large-scale changes, but it can be very effective. So that's a little bit of the trade-off that we deal with between these.
And I'm sure when you showed up and announced that you were going to remove some of that autonomy and deploy standards, everybody was thrilled and just jumped right on board.
