Step one is give your team some time to play around with things. Give them space. You will see the gains later, but you need to do it first. That's ultimately the vision is how do you complete this feedback loop?
AI has made it easier than ever to create software, but the risk is you'll use your new AI superpowers to create endless slop. To avoid the AI slop eclipse, today we're going to discuss how to create great software with AI and how you even measure how good the software experience is for your users. Beyond this, I want to understand the implications of new AI workflows on product engineering and design teams.
Rage prompts are everything from exclamation points to capital letters to profane language. That could be bad for everyone. You don't want to end up on the front page of the Wall Street Journal for leaking people's data.
Our guest is Todd Olson, the CEO and co-founder at software experience management platform Pendo. Todd has spent the last three decades alternately managing product departments at software companies and founding his own software companies. That means he's got a keen sense of what great software products look like.
One-way door is you go out, locks behind you, and you can't get back out. Two-way doors, you walk in and you don't like what you see, you can walk back out the other way. Some decisions are one-way doors. Once you go there, can't put the genie back in the bottle.
Let's find out Todd's software secrets.
Hi Todd, welcome to the show.
Yeah, great to chat with you. So to begin with, I feel like I spend a lot of my day complaining about software and I think the same is true for a lot of people. So why is there still so much bad software around?
I think there isn't enough attention to detail and design around how to build it. And very often people think that you can just build something and have people use it. They don't take the same rigor in measuring it and improving it and trying to make it better. So to me it's the discipline and lack of focus on iteration and design that often leads to such bad software.
Okay, yeah, I love the idea that you've got to focus on the details. Make sure that you are being disciplined through design and tracking. I think we're going to get into a lot of this in more depth. But before we get to that, I'm curious — because AI has made developing software a lot easier, you can do things a lot faster, is that going to improve software quality or make it worse?
Kind of a nuanced answer to that. Do I think that in the right system it could improve software quality? Sure. I've talked to companies over the last few weeks — they're like, we have not zero bugs, but almost no bugs. The second a bug is found, we have some agent pull it, then we actually fix it. I think in certain mature areas that could be correct. But if you're just creating and generating pure AI slop without a lot of focus and rigor around how good is it, how is it being used, what's the architecture of it — no, I think you run the risk of actually having more bad software.
I think the other risk you have is you have a number of people who are not professional software creators creating software. Imagine if you had a large group of professional musicians or professional anything just creating something without experience, formal training, any training on what good looks like. If you don't know what good looks like, it is hard to inspect. And that's the thing that a lot of folks worry about — how do we help all these people, our new builders using these amazing solutions, make sure that what they're building is of high quality, that it scales, that users can actually use it successfully.
What are the patterns that we use? This is a discipline, a practice that's existed for decades now. There's a lot of learnings from those decades. How do we embody it all? The coding agents have some of those learnings built in, but it's not perfect and there's still a lot of honing to get a great software product out.
Absolutely, yeah. So you mentioned quite a few things there — the idea of having a good system, a good framework for building stuff, having the right training. Because I guess if you're Miles Davis you can improvise a great tune, not everyone is the Miles Davis of software. So maybe we'll start with the systems — what does this good system for software development involve? What are the components?
You're providing a lot of context to the coding agents that you're starting to believe. What does context mean? Context means good information about the problem domain, good information about the users — who are the users, almost like a persona, what do they care about, what are their challenges. I think you need to think about what sort of data you supply to the coding agents so that they're intelligent — what are people doing today in your application, where are they getting stuck, what are people using today versus not using.
Let's say you have one of your highest use pages and workflows in your product and the coding agent decides just to change the entire thing because it gets some requirement that it has to make and that's where it decides to put it. That's pretty high risk, right? Versus if you told the coding agent, hey, this is our most heavily used thing, this is our most important workflow, it probably on its own wouldn't decide to put things there. So that's a good example of what context does — by giving it more information, you may make a different decision, just like if you're a human being, if you had more data, you're going to make a different decision. The quality of the data being provided is what you need to start thinking about first — what data do you need to make good decisions on what to build next.
Okay, yeah, I love that. Just think about the popularity of a page — if you've got an agent that's going to update your corporate homepage, that's a very different impact than it just updating some minor page or a single product page. So since you mentioned data, one of my favorite things and one of the favorite things of our audience — talk me through what kind of data is important for providing context. You gave the example of popularity of things. Is there anything else?
Yeah, look, I think we look at journey-level data. Where are people flowing? Where are they going? Where's drop-off in your application? If you have an area where you're trying to get someone from point A to point B, where do they get stuck, where do they drop off? I think journey-level data is super interesting. With the rise of all this agentic software, there's a whole new set of data that has emerged — how are people interacting with agents, what are they asking for, how is the agent responding, is the user getting frustrated. When they go to have agentic conversations, is it invoking certain MCP tooling or not?
Starting to sort of unpack how humans and agents communicate and interact is a whole new growing data set which I think becomes really important for shaping the future. If you believe that a lot of our future is going to be interacting with agents, getting a really rich data set on what that interaction means, how good it is, where there's opportunities for improvement — I think that's going to be a very interesting area. There's really no other way to get that information to know what a user wants an agent to do other than instrumenting and capturing full telemetry on all the conversations and how things are working. So that's definitely another interesting area for me as well.
Absolutely. Yes. So I love the idea of tracking what users are doing — there's going to be a lot of click data and things like that. More broadly, for people who aren't familiar with this product, how do you collect these data sets? How do you go about finding out what your customer wants from your product?
Yeah, look, we're a part of every application. You embed Pendo within your application. Once we're embedded, we're actually capturing all the behavior — everything, all the traditional clicks, the journey-level information, what order people are operating in. But also we can use the same install to capture all the agentic conversations — the human-to-agent coordination and even the agent-to-agent coordination. Think of us as having something within your application that we're constantly monitoring everything that's happening inside that application.
We also complement that with the ability to capture qualitative data — everything from basic surveys and polls to full-on what features do you want, potentially even capturing things like voting on features. We want to provide a more holistic view of how users are interacting and what they want out of software, both quantitative and qualitative. That's essentially how we work.
Okay, yeah. So it's a mix of automatically collected data sets and then maybe going through user interviews and things like that for more qualitative things. How do you get data if you've got agents as users? There's a lot of agents using software. Is that a different way of collecting data? Do you collect different things? How do you know what an agent wants?
Yeah, look, there are some ways to capture when agents are using software and there are different interactions. Some of it is agents are calling APIs, or they're calling command-line tools, or they're calling applications via MCP tooling. So I think there's ways in which we can instrument those calls to capture analytics — when are agents, how are agents interacting with software versus how are humans. At some level it's the system-level communication, people are calling it headless. If you think about headless systems, those are the ones where I think capturing how agents are interacting via APIs is going to be a good part of the future. At the end of the day, as a software provider, as long as you're using our data and using our systems, I don't know if we care so much if it's headless or non-headless, as long as we are essentially this system for whatever task it is that we're solving.
Okay, yeah. So this idea of headless software access seems to be increasingly popular where you don't need to bother actually using the software yourself — you just use your favorite LLM and get that to ask the software for you. Do you want to talk me through how that changes software development and design, if you're designing it to be used in a headless way?
Yeah. Well, I guess if you step back, the first thing you have to think about is what makes your software valuable or what makes your application valuable. For many folks, it's probably not the UI, it's probably not how you interact with the software — it's what your software is accomplishing. I'll give you an example. We're an analytics provider. Our job is to give you data on how to make your software smarter. Whether you log into our product and look at a dashboard to do it or whether you use Claude to get the answer to your question — I don't care. I care that my data is accurate. I care that you use it, that you're ultimately able to take it and take action upon it, because that's actually where you're going to get the most value out of our products.
So if I think about that, a lot of it is how do we surface enough data that people can take action on it. We want to make sure that in a headless environment we're able to translate what the user is asking for — essentially the question — to good, accurate answers, to the point where the LLM knows which tools to call for what purposes so there isn't confusion. Frankly, the last thing we want is the LLM to call the wrong tool, present the data incorrectly so that it leads to a false conclusion or something to that degree. So we're trying to focus on accuracy relative to the questions being asked. That's a big part of the design.
And then what do people want to do, what other actions do people want to take via a headless environment? A lot of the way we're thinking is anything you can do in our UI you should be able to do headless. But some activities are visual where a visual interaction is actually better — I want to see something. In that case, what does that look like? That could be providing a screenshot, that could be something else — again a different way of delivering the same value but consumed in a different manner.
Okay, yeah. So I love that you think about the important question there — what do users actually want, and are we actually providing them with the right answer, is it accurate. We talked a bit before about collecting bits of data to find out what users really want. The second part is maybe harder — making sure that you're providing the right answer. This is a notorious problem with AI — sometimes it makes stuff up. What techniques have you got to make sure that you do give the right answer when you're developing software that way?
Well, there's a lot of techniques. Some of the basic techniques that people talk about are things like evals — essentially think of it as an automated testing suite, but think of it more as a grading suite where AI is grading the answers of AI to see the success. You can get scores on that and make a ton of improvements with evals. But evals alone do not make products better, because there are situations where the eval may score something well but the actual user has a different experience.
So I think there's no replacement for really measuring how users interact with software. If they're operating in a headless environment, how many back-and-forths, are they happy with the answering. When people are complaining around the collaboration with the agent — so we have this concept within our products of rage prompts. When are people getting pissed off at interacting with an agent because it's like, to me the users know it's wrong but it keeps giving back the wrong answers. To me those are good areas to go deeper on.
I mean I love this idea of a rage prompt. Can you give me an example of this and what do you do about it?
Yeah, I mean rage prompts are everything from exclamation points to capital letters to profane language to sharp language, repeatedly asking the same question without changing it. These are all areas where the users are clearly pissed off, and we try to measure that on a per-use-case basis to understand in what scenarios are people being driven to rage. And that becomes a signal-rich area where you can then hyperfocus almost on improving those types of interactions.
We know that when someone — I'll use a fictitious example of a travel site. If we know that rebooking something, changing an existing booking, leads to a lot of rage, maybe we invest a lot of R&D on smoothing over that process, really going deep in the problem so that we address it, because that's what we're seeing is causing people frustration and we don't want people to be frustrated.
Absolutely. I think that sounds like almost the worst possible user interaction — they're getting so mad at your agent, your chatbot, they start swearing at it, they start cursing at it or something like that. So that's absolutely what you don't want. In terms of figuring out what's going wrong, do you have to do this manually? Is it like a human just going through chats full of curses, or can you do this automatically?
Yeah, I mean we have a way to do it automatically where we're capturing it all and sort of labeling the conversations as such. So now we can surface those insights right to you so that you can actually fix them.
Okay, that sounds useful if you're not having to trawl through all these chats. So let's go back to the headless environment — what if you are interacting with those headless and you ask, give me the top three areas that's causing people massive amounts of rage, and then we return those things, and then you ask, let's diagnose the application to understand how we can actually address and fix these issues. That's actually a real-world scenario where someone bringing this data and this context goes back to one of the earlier questions — how do we make sure that whatever we build is of high quality? If you're saying use this data to then improve this application, you're going to build a better software application. But it's all about the context you're provided, the data you have, and using it to make sure you're actually iterating and improving things.
