Olivier Pomel (Datadog): Reimagining observability for the AI-powered enterprise

Thanks. Thank you.

Yes. So, thanks for having me here, and I apologize for forcing you to say observability so many times in one hour. It's the worst word to pronounce in a fast pace like that. So what we define observability is, it's basically the idea of understanding how your applications are working, how they're working internally, how they're working from a technical perspective, who they're working for, the engineers that built them, but also who they're working for, the business and the customers that end up using them. So the idea is to understand everything from the very low level, the CPU and the networks and things like that, all the way up to how much revenue applications are generating and what they're doing for the business, and bringing all that together. So that's what we think of as observability. It's been around for a while, so ever since the first computer was sold or the first computer software was sold, there was software that was sold with it to manage it. At the time it was called monitoring, not observability, but it was the same idea. I think it's becoming more and more relevant because there's been an explosion of complexity for applications in general. So it used to be that the engineers would spend years shipping small pieces of software that would run on one computer. Today with all of the gains of productivity in a few minutes, you can build something that's going to be shipped to customers immediately and run on thousands of machines and include like hundreds of services. And AI is just accelerating that. There's just way more can be built way faster and also the applications we are building with AI are not as deterministic as the applications we used to have before. It used to be that everything an application did, you'd spend weeks and months understanding them, building them, specking them. Today with AI you can either build an application on top of an AI model, which is stochastic in nature, it's not deterministic. Or even if you write code, it's very possible that a lot of the code you won't write yourself, like the AI will write most of it and you actually don't understand really what's happening there. So if you zoom out a little bit, and if you look at the broader picture, what's happening is that we're spending a lot less time building and a lot more time running applications. And so a lot of the understanding of whether these applications are doing the right thing, they're working right, they're doing what's expected for the business or if they're secure and performing as expected, all of that you have to understand in production environments using observability as opposed to doing that at the design phase when you are first building your software.

So we're not there yet. And I'll give you a little bit of a, again, background, like if you zoom up. The work of fixing an application when it goes down at night, 'cause usually it's at night. I don't know why, but you have to wake up people and you have to get them to fix an application. The work they're doing is actually pretty complicated because applications are all fairly different, so if you compare say, an application with self-driving, self-driving looks like it's around the corner. It's been around the corner for a while, but it looks like it's really around the corner now. It's really happening. It's arriving. Anybody can drive a car, like anybody over the age of 16 can drive a car. And it's something that most humans do pretty well versus solving a problem in an application usually involves a team of really skilled engineers. Maybe you have a few PhDs in the mix and people work really hard to understand, formulate hypothesis, understand what might work and not work, and you know how to solve the issue. So it's an inherently more complicated problem. As I said earlier, also, there's an explosion of complexity, and the problem is that no, we're not standing still. So everyday applications are becoming more complex. So as companies such as Datadog, for example, build tooling to auto remediate, to understand issues, the applications themselves keep getting more complicated. So far we've been playing catchup game with that. Now that being said there's been a real step change in the capability of AI. I think what used to be science fiction a few years ago, which is for example using all of the documentation that's been written about a product, understanding all of the product interactions from end users. All of that is possible now, and actually it's fairly accessible with the recent AI models. So we see, we're seeing that we can start catching up and starting to live up to that dream of not having to wake up people and have them work for hours to understand why an application is broken and to have them fix it. So we shipped a product. We have an AI including an SRE agent, for system reliability engineer. So what it does basically is it investigates outages on its own and it understands what went wrong and then gives a pointer to fix it. And we started shipping that, it's actually in production and it's being used by customers and we see that in a large number of cases, by the time customers actually wake up and they go into their systems, there is already a hypothesis of what's wrong. And instead of starting a zoom call with 20 people and working for three hours in middle of the night, maybe in 15 minutes they have a good idea what's wrong and they can just wake up the two people who need to fix it. So we're still not in the world where everything is fixed in the morning and people don't have to wake up. But I think we're getting there.

So I think we're still a little bit early to think about like very broad decision making that is being made by AI. The way we think about it is we're focusing on two things. One is freeing more time for the deeper thinking, the more strategic thinking, the big decisions. And we do that by trying to not spend as much time in reaction mode, so reaction mode would be what I talked about earlier with waking up and fixing issues. It also happens throughout the day. If you ask engineers what they spend their day on, they'll tell you there's so much of their time they spend firefighting, with various things that are breaking or not working as expected or last minute changes or things like that, so if we can get rid of that and have the AI do a lot of that we free a lot more time to think, to build, to understand what matters and to plan what's going to come next. Another area we focus on for that is on the security side, we also built an agent for analyzing security signals and telling customers whether these signals are benign or need attention. And it's the same idea. Like, instead of spending 80% of your day on reacting and chasing so many benign issues; if you can just focus on the few that are meaningful, then you can spend the rest of your time thinking hard about what in the broader sense needs to change about your security posture, what you can do to be better in the future. And so we want to free time for that. So that's one area we focus on - saving time. The second area we focus on is getting better data. And I think the AI in general, and we, maybe we can come back to that, but AI in general is only as good as the data that feeds it. And so for that, if we can get better data, faster data, cleaner data we can make better decisions. And so we focus on that as a company, like the way we look at all of our various teams and business areas like, whether that's front office, back office, finance, go to market or engineering. We focus a lot on getting the right data and getting it as fast as possible so we can make the right decisions. As a Product builder, we also invest in helping our customers understand how their data is flowing, capturing the best data for us. We capture it at the source, which is the application that actually runs the business, so if you have an e-commerce application, it runs your business. That's the best place to get your data and to get it flowing and to get it right. But then we also realize that data typically follows a lot of different steps until it is being fed either in business dashboards or in AI models. And so we also realize that we need to build a whole stack for data quality and data observability. And we even acquired a company that built that observability into a product.

Oh look, all the bigger decisions they're not for us, are not gonna be made by AI for quite some time, I think, resource allocation, things like that, where we spend our time, which products, areas we're going to. But, I think we're automating a lot of smaller real time decisions with AI. And I would say there, what you can automate, I think it depends on the domain. And it depends also on the customers and their culture. So I can give you a few examples. With an AI SRE you can automate restarting systems, and not messing with services that might be running or not running, updating code, rolling back code and things like that. With a security agent. You can also automate shutting down systems that might be affected by an issue. And there's a number of other things you can do, like blocking users. We found that at least today. Our customers tend to be more confident with automating action based on security signals than on operational signals. And the reason for that is the upside, the downside is calculated differently, for a security issue. The downside, if you overreact, is that you're going to cause some availability issue. But the downside, if you underreact is that you might have a security incident, which is a much bigger, long lasting issue. And so customers are fine being a little bit over automated and taking action very quickly. On the operational side, I think the calculus is a little bit different. So users prefer to be in the loop today. I think this is going to change over time, but you can see that in different parts of the business there's a different tolerance for automation or desire for automation.

Yes and so it's a good point. So that's one lesson we've learned very early on with Datadog. It's that customers, when we talk to customers and everything we do starts with customer conversations. There's one lie we hear from customers, which is: 'hey, I prefer the false positive because I will decide if I need to pay attention. I want to be aware of it and then, if it matters, I will take action'. The reality of it, and we know because we've tested it, is that if you send three or four positives in a row to a customer, they'll just turn you off. Nobody wants to be on the receiving end of that. And so we need to do the work of understanding what needs to be shown to customers, and we need to have a high bar rate for it. When you think about AI automation the bar is even higher. Like it's higher for AI than it is for humans. I think it is when you think of all the situations where we see our teams or our customers go through outages and try to understand what's going right or wrong, it's very frequent for humans to be wrong. Like humans will have hypothesis and they'll communicate that to their coworkers, and in the end won't be. And it's okay for humans to be only right 60, 70% of the time. It's actually a great hit rate. But if an AI is only right 60 or 70% of the time, it's not going to work. We have higher expectations from AI. We have a lower tolerance for being sent in the wrong direction by a machine than by a human coworker. And so we need the standard to be a lot higher. And so the whole process we have around building AI agents is to have first the right evaluations for those internally so that we can know when they're good enough to be shipped to customers. Usually it means with a success rate in the order of 90%. And then that depends a little bit on the use cases. I think that can be different quality criteria depending on the use cases.

Yeah. So it really depends again, it depends on the cultural business. It depends on the use case. Like when we measure the models. So I give you two examples. Again I talked earlier about the AI SRE and I talked about the AI security agent. The quality for both is measured very differently. For a security agent, you need the agent to be very stable and reliable. In that, if you ask the agent about the same issue twice, you should get the same answer. Which, in generative AI is not a given. A lot of things can change. When you re-roll the stochastic process a few different times and look, everybody understands there might be an error margin. Like same thing with humans as an error margin, but you want the outcome to be the same every time, or you completely lose the trust of the user. On the other hand, if you compare that to the SRE agent, so the agent that looks at outages and solves outages, it's actually okay if you get a slightly different result every time you run it. And the reason for that is you're never going to run it twice on exactly the same data set. Like it doesn't make any sense. The situation is changing over time. However, what's really important for it is for the solution to be or the model to run very fast. Because it is very relevant, if it runs in a few minutes, if it runs in half an hour, it's not relevant at all. And so we place the quality evaluation in different places. And then of course, then around that, like there's different guardrails you can put in place to guarantee that the models are not going to do certain things. Those we find tend to be very specific to the customer, their culture and how fast they want to move and what is particularly important for their business.

Yeah. I think what's tricky about the edge of AI is that everything looks good in a demo. And, I would say in most cases the reality is not as advanced as a demo, or rather the demo is the absolute best case and what you actually get when you try the products might vary quite a bit. I'd say, there's two pieces of advice I would give. The first one is to decide as a business what's completely core to you. And where you're going to be a leader, an early adopter, and where you're just going to be a fast follower and an educated buyer because you can't actually innovate and be at the forefront everywhere at the same time. So for us, the decision was really obvious. Like, we build for engineering teams and product teams and security teams. And so we had to build product for that. So we have to be at the forefront for that very specific part. For the part that we don't build ourselves, we also have to be at the forefront. Like we have to work with the best companies to understand who our customers work with. For the rest, the back office, everything that has to do with finance, go to market, we don't have to be at the forefront. We don't have to invent that technology. We don't have to be the very first to try the product. It's okay if we give ourselves a little bit more time and we see what's working best and not. So that's one piece of advice. The other piece of advice is when you think of how you're going to adopt the technology internally. Like one big question is. Do you centralize the innovation? Do you decentralize? Do you do top down? Do you do bottom up? And my advice is to do both. I think it's good to have a few big initiatives that are driven top down. And you want to make sure they reach everywhere in the org. And not every part of the org is easy to move when it comes to changing the way they work and technology they adopt. But at the same time, it's important to give room for all of the various teams and pockets in the organization that have creativity or that understand a specific issue and they want to solve it. It's important to give them room to build and innovate. And so it's good to have this ability for various teams to adopt in a bottom up way. And I think some of that is gonna work. So that's not, it's not gonna work. In the end, as the market mature, I think everybody's going to end up in a better place one year, two year, three years from now.

Yes. So the part that's top down is the part that you want to make sure absolutely everybody in the org does, so it might have to do with the common software you use for collaboration, for managing the business, managing the internal HR process - things like that. Like you want to make sure that you have some common base and that it reaches everywhere in the org. It might also relate to education, like how you educate people to the new technologies they can use, make sure it comes across their radar, make sure people get the memo that they need to change the way they work. So all of that I think is fairly top down. Bottom up might be more very much more team specific. So a team is doing customer support and they have some ideas of specific problems they can automate or specific tools they want to try or so there needs to be room for all of that. I think it's too hard, too early to understand everything that can be done top down. And also you want to be able to identify the people and the teams that have a burning need, that have the creativity and that can get going there. Now you said the problem is, how do you evaluate that? And I think like we're right in the middle of it. I think the question is for each of those, like internally, the way we look at all those projects is we ask ourselves what does it look like if it works? So what are we trying to do? Is there a way we can measure whether it works or not? Is there a metric we can back to it of some sort? How do we know that we should keep going or stop? And we try to set that from the get-go. Like we say, in six months, where are we? How do we know we can keep going? How do we know we should stop? We're still early though. So right now most of the initiatives are still going, so I can't tell you whether we're doing a good job at it or not. What I know is that there's a lot going on in many parts of the org. That's exciting. It's a little bit scary too.

Yeah, look, a lot of the focus and a lot of the management bandwidth goes into the product building side of the org. Because for us it's a huge market and a huge opportunity, to serve AI. There's two big parts for us. There's observability for AI. So basically helping our customers understand the AI systems. And it's a huge array of investment. And the other half of it is AI for observability, which is what I mentioned earlier, which is how do we automate everything for our customers? And it's a very large investment for us to do all that. So there's a lot of focus that goes into that. We built a whole new series of functions of dialogue for that. We built an AI research team, like we didn't have a pure research team before. As an aside. I started my career in the US in the research team. I worked for IBM in research. And the way research worked at the time was that you would invent something, develop something, publish a paper and it would probably be turned into a product maybe in 10, 15, 20 years if ever. And today with AI, we see research that turns into products within six months, and that is not published. For companies such as ourselves, we need to have an AI research team. So we are hiring researchers. So that's a whole new function that we created. So again, a lot is going on the product side. A lot is going on there. Also on the way we develop, so we build some AI agents, also some developing agents, so coding agents. But we also use a lot of the agents that have been written by others. So we use Cursor, we use Claude Code, we use a lot of those and our developers have quite a bit of flexibility in terms of what it is they can use. And so we've put a lot of attention also into how we transform the way we build product, the way we write code, and how do we measure that. I would say on how do we measure that still, we are also still early. We have a pretty good sense of usage, pretty good sense of adoption. We see that the pace of adoption is exploding, so to give you some sense of what we've been through we started adopting the coding agents or the coding assistance, I should say two years ago with GitHub Copilot. The adoption was slow. And some people really loved it, but some people were adverse to it on the team. Then we started adopting the second generation of coding agent with Cursor and Claude Code and the others, and then we saw explosive adoption. We saw developers couldn't adopt them fast enough. It spread through the team extremely fast, which gives us a sense that developers see value now in terms of measuring the value for the business, we're not there yet. I can't tell you that we're that much better at shipping functionality that has an impact on the business. I think we're still working on that. I know it's a problem we have. I know it's a problem our customers have, and it's something that we're investing quite a bit on. If you look at the rest of the business outside of building products we still working with the same existing KPIs and the first area is we want to show value for AI are the ones where we can show those well understood KPIs. So for example the time it takes us to answer support requests that's a very easy one. We have good metrics. The metrics are stable. We should be able to see the impact there fairly well.

I think, and by the way, it's not a Cursor versus GitHub kind of thing. I think GitHub also has agents now. I think I'm more comparing the first generation and second generation of agents. I think second generation is just that much better and automates a lot more of the painful work. So it removes a lot of the friction for developers, which is something that developers love. I'm an engineer. I know, understand that I can finally start coding again, so after, so I was coding for the first five, six years of Datadog. Then I stopped because it took me more time to set up the development environment than I actually had to code. But with coding agents, all of that grunt work is getting out of the way. Again, I can just get things done very quickly. I don't know, I'm not quite sure it's high quality work, but I can actually get things done. And that's a very big change. When I look at the rest of the organization. I think we see some signs of that happening, but I think we're still in the V1 of agents for the rest of the domains and not V2. Again, if we look at what happened on the developing side, have pretty high level of confidence that we're going to see similar adoption patterns for the rest of the business.