Measuring the Impact of AI Agents - with Todd Olson, CEO Pendo

Well, thanks Ray and it's great to be here. I'm honored to be one of the first guests of this podcast and it's a super exciting topic and one that I'm very passionate about. So I am the CEO and co-founder of Pendo. Pendo is roughly a 12-year-old software company whose focus is on improving experiences in software applications. Our products essentially embed themselves within applications themselves, capture analytics on how people are engaging with and using those analytics, and then surface back insights to teams to make them better. We also complement it with tools to actually improve the experience, teach users how to use it more effectively. You know, our goal is driving adoption of technology so that it ultimately delivers on the ROI and promise, which I think connects very nicely with this podcast.

Absolutely. Yeah, thanks for that. And no, I think you're completely correct. When we talk to our customers and large enterprises, we heard the same thing: everyone has a mandate right now to run experiments, to try AI, to actually launch AI in their organizations. But what we also learned is that people didn't have a really effective way to measure whether those pilots, whether those AI investments were actually yielding success for the users, hence the ROI. So we kind of stepped back and really, if you think about this problem, it's very similar to what we would do for traditional applications. You know, we're capturing and measuring data on how people are using traditional applications and feeding it back to make it better. What we needed to do is to expand what we collect to get more insights into what the agent and the user are doing, so we could start essentially providing insights for those types of applications. So that was kind of the thesis around agent analytics. So what we do is we leverage the existing installation we already have. You can then direct us towards the generative AI agents or agentic interfaces you have, register them with Pendo, and we'll start collecting the actual conversational information. And by having that and processing that, we can start understanding what are the types of questions people are asking, how good the responses are, what percentage of your user base is leveraging it, what types of themes are they coming back and asking the same questions because they're getting good answers? Are they asking different questions? So, i.e., what's the retention of certain prompts, themes, areas in this agent. We're also looking in the broader context of the traditional application, you know, like when do people go to the agent and when do people use the traditional parts of the application and what's that experience like? Do people come in, get frustrated, go to an agent and then kind of go back to the traditional side? What's that sort of hybrid workflow look like? So essentially the vision is giving people more quantitative data on how these things are working, are they working, and then hopefully giving them the tools to make it better. We'll tell them you're really bad at this or people are really frustrated when they ask these types of questions, which should be the action and impetus people need to go fix it for what the users actually want.

Yeah, absolutely. And what's exciting is, and I think part of the challenge and the reason this MIT study sort of sees this, the root cause of it is people haven't had a way to really measure this. I think what we've seen in the market is there are solutions for developers to measure like performance, measure memory utilization, you know, like the classic developer observability metrics, those exist, but up until our announcement, there hasn't really been user-focused metrics. How are the users actually interacting with it? Are they getting value out of it? Do they like it or not? You know, honestly, the state-of-the-art for a lot of the large enterprises we speak with is a basic survey. You know, we see people roll it out and they ask people, 'Hey, was this successful or not?' Yes or no. I mean, that's not a way to make something better.

Yeah, I mean I think we're looking at things like I already sort of referred to it earlier, retention, which is how often people come back and ask similar types of questions. That's kind of core. Like if people like something and they're getting value out of it, they're going to keep coming back to it and keep using it. That's a really good sign of healthiness and it's doing something well. So I think retention is a key area. I think just overall, broad-based adoption, what percentage of your user base is interacting with this? So I think that's an interesting metric and how often, which is stickiness, which is a metric we talked about in the prior podcast. And the other thing that I'm looking at as well is frustration signals. So like in the traditional world we have this concept of rage clicks, people kind of clicking, you can all imagine someone getting very frustrated and clicking a mouse a million times, thinking some button's supposed to do something and it's just failing silently behind the scenes. You know, the AI analog for this is rage prompts. People just getting irritated by the solution and sort of telling it, 'Can you just answer my question please?' or using capital letters or using exclamation points unnecessarily, right? So really measuring frustration, we think if you can just eliminate people getting pissed off, I think you'll have a better experience. I still, you know, that's why when I get into those voice activation loops, right, and I'm like, 'Agent, agent, that's what you're doing for AI agents.'

Look, I think that's certainly something we're looking at in the future. It's not doing that today, but I think this idea that if someone really gets stuck and we have real-time analytics on it, like how do we push that data to someone to get real-time assistance to that end user? Yeah, that's something we're certainly exploring.

Yeah, I think that's an exciting piece because it's not just about giving you metrics. It's how do we actually make the experience better and we make it better by directing the end users to what we know the agent's good at answering, you know, and we can do that by data. So it's like, hey, look, welcome to this agent. What does this thing do for me? We'll say, you know, 90% of users start by asking these types of questions. So we can even do it by role persona, you know, anything about the user, like users like you tend to use this agent for this. So that's the kind of action piece that we're doing, that's something very exciting.

Yeah. And I always found that unless your application software is really kind of natively capturing the outcome, maybe it's the ticket resolution, it's really hard to capture that. So are you seeing Pendo somehow playing to make it easier to actually measure and report on outcomes?

I mean 100%. Yeah, we absolutely are. And I mean, you know, if you look at, there's a variety of pricing startups that are starting to emerge that are actually integrating with things like Pendo because we have really useful data for outcome-based pricing. So yeah, I think it's going to be an interesting trend. And look, I would 100% agree with you. People talk about outcome-based pricing and the best and most notable cases are things like ticket resolution because it's so easy to measure. You know, I think, Finn, which is a pretty popular agent, I heard at least at one point there was like 99 cents a ticket resolved. That's a very clear I understand what I'm paying for and it's very obvious. Not all software products have that. Like take an analytics product. How can I know whether a report I gave you was insightful or not? Will you learn something or not? And are you going to tell me when you do? I mean, that's the other big challenge. So that's interesting. I was talking to a variety of CEOs recently on this topic and they had an agentic solution. They were trying to measure quality conversations. What's a quality conversation? Now you could say we could as an independent arbiter of agent and agentic solutions give you sort of a quality score, like how well do we think this went, and you could charge based on that. But I look, I'm super excited about this. We have a newer product as well, which is a sort of, you talked about it a little bit earlier, but like a prediction product. So it's traditional machine learning but also leveraging it in an agentic way, and we're charging on predictions, the number of predictions we make, which is the first time we've sort of had a very outcome-based model for pricing and it's very clear. And if you want to run the model more frequently, guess what, you're going to pay more because it's more predictions, but you can also run it less frequently, so I think depending on what you're getting from it and how you want to use it, it'll modulate, etc.

Yeah, 100%. So, this is called Pendo Predict. It's a product of an acquisition we made a few months ago. And the thesis is actually pretty simple. It's taking Pendo data, so data about how people are engaging with software, and then looking at customer data from systems like Salesforce.com or HubSpot or others. And it's combining them in a way to create bespoke machine learning models that can do things like predict churn, predict upsell, or predict which lead that came onto your website is a high quality lead that we think will convert. So you're kind of taking this amazingly rich data set, combining it with other business information, and then driving predictions, and then we're feeding those predictions back into other systems. So for example, if we think someone's going to churn, we can Slack the account owner and say, 'Hey, go call this customer,' or we can push that information to a CRM like Salesforce. When you log in or looking at a record, you'll see we think this thing's going to churn, you know, be aware, right? And oh by the way, we can also give explainability, like the model flagged this because of these five reasons: they are not doing this or they are doing that, you know. So I think the power here is how do we take this amazing data set and help it drive more ROI for your organization.

Yeah, 100% horizontal model. Yeah, that's nice for some end user stuff, but when I'm doing a true end-to-end automation or you have a lot of transactional data, so does your Pendo Predict need to do some training on your customer data before it gets turned on and goes live? Yeah. Yeah. Basically, it's all sort of automated because we have all the data or have most of the data. It's Pendo data, but yeah, it's 100% trained on the customer's data. So we're looking at the customer's data. We're looking at what, so for example let's say you want to do a churn model. We're looking at which customers churned versus ones who did not. We're comparing their behavior and we're leveraging models to essentially build a health score which sort of ranks people by their likelihood to churn and then we're giving you the reasons why. So what's powerful here is often we as humans have bias towards what we think are the reasons people do these things. And by leveraging a machine learning model, it sort of strips the bias away and just lets essentially the numbers drive the predictions.

Yep. And it's got to write that to maybe your Zendesk ticketing system, whatever it is. So, you're telling me that with your product, you can actually track that data lineage and when someone says, 'Hey, I want to know how many times we touch source data systems across this process, you're going to be able to show that.'

Yeah. Yeah. I mean, that's a use case that I wouldn't be like one of our, you know, probably earliest use cases we lead with, but I think getting a sense for where data came from, tracking it. Yeah, I think that's pretty interesting.

Look, I mean, yeah, I think this is obviously a vision of the future, but what we're starting to see is let's think about some of this technology is sort of certainly offloading work from a lot of our teams' plates. Like for example, we have an agent that lives on our website that's qualifying booking meetings for companies who want to talk to people at Pendo about learning more information. That is offloading work from a human being that traditionally does that, essentially the SDR function. That's sort of a growth example. But look, I mean, we have now agents that are helping our engineering teams to handle certain tasks. So for example, one of the more recent product announcements we had was around internationalization. You know, you can get Pendo in Japanese and German and your favorite language. How do we do that? Well, we leverage agents to do a lot of the heavy lifting because if you think about what is internationalization, it's a pretty well-known pattern that can be a little bit tedious. You know, searching for strings embedded in your code, replacing it with the ability to look up within some locale what this is in certain languages. That's an area where AI greatly accelerated our roadmap plans. So it's something that would have taken years. We got down to six months and used half the people. So that freed up those developers to go build other things, other innovation. So that ultimately unlocks more throughput and velocity for our teams, which is what gets me really excited.

Yeah, it's a great way of thinking of it. Yeah, it's exactly the way to think of it because like anything, if an agent, if you can kind of think of an agent as a human, humans aren't perfect. They do need to be performance managed. They need to be coached. They need feedback. Agents need the same thing. They need coaching. They need feedback. They're like, 'No, you're really bad at answering this question. Like, leverage this context or try this thing to make it smarter.' So I think we're constantly trying to teach these agents what context you need in order to provide a better answer because we find context is key. The more context you provide to an agent, the more accurate the results. It's really as simple as that. And here's the great thing. Same thing for humans. If I walk up to a human and ask it a question and I don't give it a lot of context, it is just guessing and filling in the blanks of what it thinks I mean by that question. Now, if I'm very clear in my question, add a lot of context and even tell the human like where to go find this thing and how to go do this or here how I've done this before, I'll obviously do a much better job. So the same with agents, and yeah, the same rules apply.