Okay. Hey everybody, welcome to the first episode of The Agents with Amelia Lerut, Chief AI Officer of SaaStr, and me, Chief AI Evangelist at SaaStr. And this is a bit of an experiment that we hope will work. But those of you who've been following us for the last 10 months on our agentic journey have consumed a lot of our content. It's exploded. Our page views are up 5x. Our attendance at SaaStr Annual this year will be up 40%. Folks are very interested in our agentic journey and we're constantly asked to do consulting calls and meetings. Literally in the last week I've done it with two CEOs of public companies. They want to know about our agents, how they work.
We are not saints. We are not better than you. It's just we're like 6 to 9 to 12 months ahead of a lot of folks. That's it. And so many folks ask me, ask us, probably 50 people a week if we can help them think about their agents. We can't do that. But what we thought we would do is have a little fun and maybe once a week share everything that's happening with our agents on The Agents, all the issues and bumps we have. And if you're on this journey, whether it's just rolling out your first AI SDR, whether you want to build your own AI VP of marketing or customer success like we have, whether you want to think about adding AI for support or whatever, maybe if you listen to this every week or when you have time, we can help you avoid some issues, think through things you wouldn't otherwise think, and accelerate your success in the agentic journey. So, that's the goal. If we have a tagline for this, it's too long, but it's going to be accelerating your success on the agentic journey. And as part of that, we are going to talk at least on the first one about some of our bumps because if you hear about our bumps, I think you can avoid them. And so we'll do like six to 10 things that we've seen this week. Some of them may be very mundane, some of them may be topical. But hopefully it's helpful. The first one I wanted to talk about and then get a lot of Amelia's thoughts on, which is: okay, now you can build it, but who maintains it? Now you can build it, who maintains it?
If you've been following our agent journey since this last summer on the leading vibe code platforms, and we love them. We love Replit. We love Lovable. We love Vercel v0. They're all exciting. You know, at first when I started, it was tough to get an application out the door that worked in so-called into production. And I struggled at first, even having co-founded a SaaS company that went from 0 to 200 million in revenue, I still struggled. As 2025 went on, the platforms got better, the models got better. It got easier and easier. And then toward the end of last year, it got great. And if you're not a developer, but you're on social media or you talk to your own teams, you'll see everything kind of exploded when Claude Sonnet and Opus 4.5 came out in December or so of last year.
Productivity exploded everywhere. It was just magical. You can see it because Anthropic went from 9 billion to 30 billion revenue already this year. It exploded the quality. But even if you're not a developer, if you're using Replit, Lovable, Vercel, v0 and others, you're benefiting from those models because they got better. And so what happened now is it's not that you can magically in one prompt build your own CRM in 5 seconds. Like that was always marketing over reality, but now you really can finish your apps. And I kind of handed the baton to Amelia at the end of last year. I built, rebuilt SaaStr AI. I started our new SaaStr website. I built our AI valuation calculator for startups and used a million. I did all that and kind of struggled and went through the bumps. Amelia took over probably in January, Amelia took over. Yeah. And built our AI VP of marketing on her own, which is very, very good.
And then our AI VP of customer success, QB, which is even better. So, we're going to talk about 10K and QB like every week, all the bumps we have because they're part of our team. They check in with our Slack every day. They are part of our team and so you'll hear us. But my slightly long setup is now if you're persistent, you can build a lot of B2B and AI apps in these vibe coded apps without an engineer. You really can. You really can get them out for real. And the meta question is who maintains them? And this is what is under discussed. And I want to talk about three things that have happened this week for us. And it's just something you're going to think about that a lot of folks, vendors, don't really discuss it. People don't really talk about it. Getting an app into production, building your own app, vibing your own app is the, you know, just like sales, when you close a sale, it's the start of a customer journey. Just like when you push an app out that finally works, that's finally pretty good. Every week you have to maintain it. And who on your team is going to do it? Who on your team's going to do it? And I want Amelia to talk about a few of the things we had, but I think this is the meta answer to the meme on social media of, oh, we're going to kill Salesforce because I can vibe it. Okay, even if you could, like which you can't, you can't do a Salesforce or a Workday today, but even if you could, who the hell's going to maintain it? But you can buy a lot of these apps for like 5 to 100 bucks a month. Amelia and I are worth a lot more than $5 to $100 an hour. So we're going to share our bumps maintaining it just so you can think about how you will, if you choose to build it versus buy it, how you're going to maintain it.
So, first one just to talk about, and this happens, but just today on several of our apps, the preview instance had database issues. They didn't connect. They didn't connect. It's going to get fixed. Bugs happen. Issues happen. It happens with lots of vendors. But just from an end user perspective, Amelia, just describe in general kind of the issue and what ran through your mind. I mean, I literally was exasperated at the agent.
I think it's tough, you know, when one thing goes down or your site goes down, it's frustrating. And of course, if you haven't built it yourself, you automatically go to the person, right? Like, oh, if my Salesforce isn't working, I'm going to go to Salesforce. Is my website down? If maybe you have a Squarespace or WordPress, like you're going to go to those folks first when one of those things are down. But then when you vibe something or if you have an agent that is slightly custom that you've done kind of yourself or you have your own hookups yourself, it's like, well shoot, who do I go to? Like I tried to problem solve at first because I was like, well, I guess I can go to the kind of root agent and see if it's something I can just fix in the platform. And interestingly, the agent couldn't resolve it. Sometimes when you're going to have bumps you're going to have to maintain, right?
And to be clear, this all happened on the preview environment. And for folks that haven't built software, you know, I'm oversimplifying, but there's basically preview environments. There's staging, we'll put it aside, and development and production environments. Production, we had no issues on the production environments, but preview is where you tweak your app, you iterate your app until you push it into production. And that we weren't able to access. It was unavailable for hours, right? And not only was it down, but if that happened at Shopify or Salesforce, right, or even Squarespace, it's on them, right? The difference here is that there's millions and millions of unique apps. So, whose fault is it? Who is the database administrator?
I mean, in this case, I was like, it's all of us, right? Like, I literally told you I slapped you right away. I was like, oh, I think some of the agents are down. Like, first I thought it was just me. I was like, oh, it must just be me. Maybe you can still access it. But then when we learned it was affecting you too, it wasn't just me. Then yeah, you know, it's the Spider-Man meme where you all point at each other and you're like, well, whose fault is it, right? Like is it the customer's fault? Like is it something that they... I kind of assumed it was our fault at first. I was like, okay, well, we built this. I'll just assume it's our fault and I should try and fix it. Then I asked the agent what to do. It started doing some lines of code, but then I realized in that process that some of my other agents were down, right, in the same way. Okay. So I was like, okay, it's actually not just this one instance. So that was super interesting, too. So then I went to our vendor that we use for this.
But again, it's one of those instances where it brings up yet like who maintains this? And also if you're not as online as we are, right? We're trying to keep up with our agents 24/7. Like I notice this pretty much right away. I feel like other folks, if you don't have a database manager or somebody who's in charge of all the agents, you may not notice. It may lapse. Like you may not notice for a couple days, especially if the front end is live, that some of your agents on the back end have broken. And so I think it's an important nuance now to start to think about who owns all this in perpetuity, not just again to your point earlier, the hard part does seem like getting it, you know, fully baked and specced out the way we like it to do. We like to always use our agents for processes that already work. Like that is a lot of run-up and lead-up of work to get it already to that point. But then you have this whole other beast once you get your agents actually into production. We're like, 'Oh shoot, what do I do now?'
Yep. So Amelia has now surpassed me in terms of her ability to vibe code.
No, mostly she has. But in terms of this particular issue, it's interesting. I still have some domain expertise. Again, having at least co-founded an app, I still understand some issues out there better than she does. Without me, her initial conclusion of the issue was wrong. As good as she is, her initial assessment of what was wrong. Then we asked the AI agent and it's pretty fun. When I started doing this, when I started vibe coding, you'd ask the agent, it would hallucinate the answer. It would make, oh, the answer is your DNS. It would just make it up. Now the agents, and we'll talk about this in a different context with a little bump we had with Clay next, but the agents are pretty self-aware. They can analyze all the issues and they can help you debug like 99%. So when you go into an app like Lovable or Replit or v0, whatever, and there'll be that little prompt, that little agent, not only can you come up with ideas on how to build the site and features, it can also help you debug issues, but it's not perfect and it doesn't have access to something. And so it tries, but all agents are goal seeking. They want to make you happy. So at the end they blame somebody else. They'll try to fix it. They always try to fix it now and then they're going to blame Claude or Lovable, blame some apps I had hooked up to it. Right. It was like it must be Qualified. Salesforce, they blamed Qualified, which was not the issue. Immediately they blamed Qualified as why it couldn't come up right. And Amelia's like, well, we didn't change the code. We've been using Qualified for four months.
I was like, are you sure, AJ? Because I was like, I haven't even, like, you know, when I make updates to Qualified, I make them in Qualified. It doesn't really change how it integrates with this particular agent. And I knew they had just integrated with Salesforce this week. So I was like, maybe there's an outside shot they changed something on the back end. Like that was the only thing I could think of. But then I was like, okay, let's for a second try and turn Qualified off and see if that fixes it. But of course that didn't fix it. Then it started to blame other third party tools. I was like, 'Yeah, it's not Qualified.' I think it just blamed that one because it's one of our beefier agents that this one hooks into. So, it was like, 'Let me just find the most common denominator and blame that instead of maybe looking for the actual problem.'
Yeah. When we talk to folks there, I don't think you have to be technical to create your own amazing vibe coded app today. You do not need to be technical, but you need someone pretty damn product savvy to maintain it if it's at all complicated. And we talked to a lot of folks and maybe this is the biggest meta takeaway. Either they don't really plan on anybody maintaining a lot of agents or vibe coded apps, right? Or they push it to someone who's maybe very good at a functional line. They're a decent marketer. They're a decent sales ops person, but they have no intuitive sense of how software is built or made. I don't think they could fix most of the issues we see. I don't think they would understand how to fix these production issues because they wouldn't be able to describe or identify them to the agent, right?
I think for somebody like myself, right, like you've built multiple companies into success, and for me having not had that background, right, like came up through more of a marketing and sales go-to-market motion, for somebody who's more so in go-to-market. When this agent started failing this morning, I was like, 'Okay, if it's not our third party tools, which I don't think it is that broke it, which we determined with the agent it wasn't. If it's not the agent itself, I'm like, maybe it was me. Maybe I did mess this up, right?' I was like...
Or it could have been me because I did a little release on this app without telling you. I could have inadvertently... It could have been a regression, right?
Yeah, it could have been a regression.
I built a Friends of Jason little page that reused some of Amelia's discounts on my own little page. Now, how could that possibly break the entire site? It shouldn't. But when you get a little deeper on how software works, you understand that regressions, when one thing can break another that don't even seem connected, right? It just is something you got to get zen about. Maybe that was the issue. Maybe Amelia broke it. Maybe I broke it. Maybe the agent broke it. Maybe the vendor broke it. And you got to... someone's got to own this.
Okay. Two other ones on this. They sound minor. I would say for the most part related to this. Generally speaking, hallucinations in our app are relatively minor compared to the old days. The old days, I would say of summer of last year, they're pretty minor. When we started using a lot of these vibe coding and agentic products, they would use more immature models, the 3s and Claude, and it would hallucinate a lot. It would just make everything up and it was very frustrating and it might have been one of the toughest reasons to finish anything, whether it was to get an AI SDR to work because it would hallucinate or to vibe an app because it would hallucinate. But they're still there. And I'll tell you just two stories from this week that were super frustrating. One is on our AI VP of marketing, 10K.
So to describe what it does, Amelia took 5 years of our revenue data for attendees and sponsors for all of SaaStr Annual, 100 million worth of data, every data point, and gave it and created an AI VP of marketing. So it has all these hard data points. It can compare year-over-year performance, week-over-week performance across any metric that you want. It's proactive now. It tells us every day where we're behind and what we should do. It has beautiful graphs and analytics. It's great. We've demoed it and we'll keep showing more.
But I built this fun fact feature where it's kind of push. It has a fun fact each day. It pushes into Slack and it sends us an email of a fun fact. It keeps getting confused what year it is and what data it's comparing. So yesterday it said we were 44% ahead of plan and then this morning it said 11. And I asked the agent what happened. It's like, oh yeah, actually I'm comparing it to the wrong year. And because I didn't have the right year, I made up the data.
Yeah. This happened before with dates, right? Like we had another agent before where it was telling folks still about London and London had already passed because it didn't know what day it was. But this is making up a corner case.
And then I congratulated myself on SaaStr a couple weeks ago is I built, only Amelia and I can use it, but I built not just 10K work, but I built a little bot, like a little Fin, like Intercom type thing where we can talk, I can talk to it. I can ask it questions and it started to hallucinate too because it didn't have a full and updated answer to the question. So, not only are hallucinations still an issue, I guess the point of the story on this, and I'll add one more and then we'll move on to the next topic, is this is something we have to fix every day. Our AI VP of marketing is very good, but it's not as good as our VP of customer success that Amelia built after, for a number of reasons. And it has these micro hallucinations every day. And for a while, Amelia built it and I didn't really touch it and I think she gave up on it, whatever. Like, it's still directionally correct, but I really wanted to go deep. I wanted answers to questions every single day. So I would push it and these micro hallucinations would creep up every day. And so now I'm kind of maintaining our AI VP of marketing for maybe 15 minutes a day, but I didn't even do it two weeks ago. So Amelia's got her maintenance now. I have these micro hallucinations for 15 minutes a day. It's not massive, but if no one on your team is doing it, you see drift. The agents can drift more and more from reality and from your data inadvertently. You think they shouldn't, but they do. So, this was my little issue where I kept comparing year-over-year data to the wrong years. And literally in the same day it said we were 44% over last year and then 11%. I'm like, this morning I wrote, which one is it? Like it can't be both, right?
I think this is why, you know, some folks are a fan of letting their agents do stuff on their own. Like, okay, like I'm going to, you know, just queue up a bunch of stuff for my agent. I'm going to let it run. I actually don't like doing that. Like, it makes me nervous if I don't see exactly what the agent is doing again because of goal seeking. I never... all these things we've built, I've built, that I'm now in like all of our 20 plus apps we have for agents, whether it's in Replit or Salesforce or Slackbot or anywhere else, or Artisan, I don't let any of them run without me looking at it for its initial execution. Like now it obviously can send some emails and do some other stuff autonomously with a lot of guard rails on its own, but when it actually comes to me deploying like new features or fixing something in the app with the agents or putting in a new sequence in Artisan, like I don't let the agents do that autonomously at all. And I know a lot of people are a fan of the dispatch you can use on Claude or like OpenClaw. Like that's why I don't use OpenClaw. This is why I don't really use dispatch unless I can watch it on my phone closely with Claude because I want to see exactly what it's doing because a few times to your point it has started to drift and go off the rails to start to do something I didn't want it to. And if I hadn't been watching the timeline, we wouldn't have seen it.
Okay. Let me just give you the third example from this week. This was a learning for me. This is a slightly unexpected one. It's model-based regressions. And what I mean by this is, if you follow us, we have a whole suite of tools to help founders with fundraising. Go to saastr.ai, click on AIVC. We've done almost a million valuations of startups and we've done like a thousand intros to VCs. And probably our most popular tool though is you can upload your pitch deck and we analyze it. We tell you how good it is, how well you're doing compared to all your peers and competitors, and we tell you what your odds of getting funded are. Okay? It's called our pitch deck analyzer. We just crossed 4,000 startup pitch decks. It's very cool.
Maybe it would be simpler if I built it today. I built this a while ago, but it's pretty complicated. It requires two different passes through Claude. Okay. And then after each pass, it has to do a lot of error and data correction, data analysis, extraction. And it's the most complicated analysis that we do. And it took me like a month of uploading deck after deck after deck and manually looking at it and manually seeing, and then I personally graded and made sure we were in alignment and they got all the numbers right and it was pretty stable for a long time until maybe January and then again in February and then again like a week ago I started seeing errors. And what it kept saying is it kept saying all these startups had $100,000 in revenue growing 500% again and again and again. I would get notified and startups wanted to be... once again, you know, whatever AI we're doing, 100,000 going 500, I'm like, this can't be right. And what I figured out is when there was an update to Claude, and it probably wasn't just like 4.7 or 4.5, it was probably a dot release or a silent release, all of a sudden hallucinations crept in and the models decided that when it didn't have data, it would just decide every startup was at $100,000 or a million dollars in ARR growing 500%. And it drove me nuts. And I kept fixing it and it would break again and again and again. And now I have to constantly work on this again. And when I left it alone for a while and it worked, but then it broke to Amelia's point. But also this wasn't because I added a new feature or changed the code. The code did not change. It was interestingly probably because of subtle changes to the model introduced in this complex workflow hallucination. So again, 4,000 decks, but it went from working perfectly to maybe 5% of them producing completely anomalous results. And so, who's going to manage that, right? Who's going to manage that? So, the three examples of why set and forget does not work.
And I don't mean to be too critical, but I want to talk about something related to this. Not only do our own apps require maintenance, right, and have these micro hallucinations and drift, we had an issue with Clay. And listen, we are big fans of Clay for the most part. We use it for what a lot of you do. We use it for enrichment and to track where our attendees and folks have gone. And we blew through all our credits and we're power users of Clay. We support it. But its agent lied to us. Its agent lied to us. So there's a meta story. So what happened?
Yeah, I'll tell you guys the story and it's all, listen, it's all resolved now because this was a couple weeks ago. So I'll say, you know, all is well that ends well. But in the moment, I was so mad. Like, I was mad and frustrated at a point where I literally told the Clay team like, 'This is the first time I've wanted to rage quit an agent, which has never happened with all the 20 plus.' I mean, I've tried more than 20. We have 20 now in deployment, right? Like, I was like, 'This is the first time I've legitimately wanted to rage quit because the answer that the AI agent gave me was so unacceptable, was so bad.' And when I reached out to the humans for the first time, I couldn't get the help I wanted. I was like, that's a double handicap.
Okay. What happened? So walk back from what did the agent... So Clay announced cheaper, better pricing for me. They just made the pricing more complex and confusing, but they claimed it was cheaper for many use cases, right? What did the agent tell you? So in other words, you went on Clay. It has basically a support agent, right? Sculptor. To their point.
Yeah, they said this to me afterwards and made this point. It's still in beta, but I was using the Sculptor because I'm used to using agents in, you know, the built-in agents in all my tools. I'll give you guys the full story. So, for the last couple weeks, we have these VIP summits we have for SaaStr Annual. I was using it to do lookalikes. I've talked about this before on Workshop Wednesday. I think Clay is a really good place if you're trying to scale up your AI SDRs or anything like that where you can make really solid lookalikes with Clay. It works really well for this and we were seeing success with it. And so I was using it for that, doing the same thing I had always done. It was kind of late at night on this Sunday. I was building a new list of, you know, VIPs to go out to for the summit and suddenly Clay wanted to charge me about 5x to run this list than it did the week previous. And so I asked...
Hold on just for a second just so that... Clay wanted to, or its agent told you you had to pay five... Its agent told you you had to pay five times as much. Confusing, Clay. The software application didn't necessarily charge you five times as much. Its AI agent told you you had to pay five times as much. Right.
Correct. Yes. It told me I was going to have to pay five times as much. And so when I pushed the Sculptor agent, I said, 'How can that be?' I just did this on a very similar size list, a very similar enrichment, basically almost the same parameters, and it was, I don't know, let's say something like 2,500 credits, but now it's trying to charge me 11,000, which is a huge spike. I was like, it has to be wrong, right? It just has to be wrong because that doesn't make any sense. And so I asked it a few questions and first I caught that the agent related to their price increase changed how it built, but it also defaulted to the most expensive action versus a cheaper action that would get me to the same outcome.
Let me... maybe I misunderstood what happened. Did the... I thought the agent wasn't properly trained on the new pricing of its own app. It was both. Okay.
It's both. Yeah. Yeah. It was both. So, like as I was talking to it, I was like, 'Okay, one, I caught that it was trying to use this higher priced, you know, model to get to the same goal as last week.' And so, I said, 'Hey, use this other action instead. It's cheaper.' And so, that lowered it by about half. Then, it went down to like 5,000 credits to run the same thing. I was like, 'Okay, we're halfway there.' And that part of the issue I thought was problematic because most people won't do that. Most people are going to look at the 11,000 that the agent tells you and say, 'That's too expensive,' and they would give up right there.
Issues. The first issue was the agent told you to use the most expensive model, the most expensive... correct. So the upsell, the agent told you you had to use the most expensive version of the product that you didn't have to use.
Correct. I didn't have to use it. So I called him out on that.
The agent told you you had to, right?
He recommended... Yeah. Yeah. He was like, 'Well, if you want to achieve that outcome, this is how you should do it. Are you ready to deploy and run the waterfall enrichment?' I was like, 'No, not for 11,000 credits. There's a better way to do this.'
You think that was intentional or you think that was poorly training the agent? You think that was intentional to monetize the base better or do you think it was bad training?
I don't know because it honestly happened the same time they announced this price increase. So, I was like, I don't know. I don't know if somebody as a human at Clay unintentionally, like that was the consequence of them putting in this release for new pricing that the agent took it and ran with it, right? Like I can totally see that as it wasn't malicious. It's just something that happened with their agent because they loaded in all the training for the new pricing. I was talking to the agent and the agent was like, 'Well, let's use this super expensive luxurious model versus like the normal, you know, Prius model you were using.'
Yeah. Because I have a rough... I know and listen, we do use Clay and it's a great product, but when they announce all this new pricing, I have a rule, a classic SaaStr rule that I think is almost always true. When a company introduces more complicated pricing, even if they say it's a better deal, it's always a hidden price increase. I've never seen it be like... there are rare occasions where companies literally say, 'Hey, listen. We've just decided to go cheaper.' But it's usually when there's something brutally competitive, but usually when pricing gets more complicated, it's a way to disguise a price increase. Even if sometimes not everybody pays the higher price, but by adding complexity to pricing, you can essentially make segments of your users pay more. So it feels like that's what the agent was trying to get you to do. It tried to get you to pay two to five times more than you had to.
At least it seemed like it, right? That was the first...
Yeah, it led to the second issue actually. So once I caught it in this like trying to use a more expensive model, trying to burn through all my... at the time I was like, I don't know, you know, I have like 3,000 credits a month. You're going to burn through my...
Five months. Five months of credit.
Yeah. Basically, I was like, 'No freaking way.' So, then I got it down to about half once I caught it in the agent. But then I could really only get it down to half and I was like, 'You know what? I'm so tired. Heck it.' Like, 'I'm just going to, even though this is more expensive than it was a week ago, I'm just going to push it through because I need to get this done.' I was like, 'I just can't keep...' you know, I had already been arguing with the agent for like half an hour at like 11 o'clock on a Sunday. I was like, I just got to get it done. And so I asked the agent to, you know, push the enrichment live with the 2x credits because I just had to get it done. And it said, 'Okay, you're not going to have enough credits to finish. So you're going to have to upgrade.' Because this is, again, this is around the time they announced the price increase. So at that point, I was like, 'You've got to be kidding me.' So I just hit the upgrade button. Again, I'm tired. It shouldn't be on the customer to figure out tired or not. It shouldn't be on me to figure out that the new plan was going to be worse for this outcome and I should have just stayed on my legacy plan at the time and just bought more credits, which in hindsight, yes, of course, I would have done. But again, when the agent was suggesting to me, hey, if you just upgrade your plan, you'll have enough credits, I'm just going to click that button. I literally, that's what I did. I clicked the button from the agent. I clicked one of the new plans or whatever, and then I finished the enrichment, and I went about my day, and I loaded up with my AI SDR. So, it was a double handicap.
Yeah, I get that. But just to... then we can move on to the next topic. But I thought the second issue when you reached out to the Clay team was that, and I think this is something we all need to learn about agents, was they had actually not properly trained their beta agent on how their new pricing worked. Like they had not uploaded the documentation, the pricing. And we've seen this ourselves. If you don't keep, you know, digital Jason, the digital version of me, it broke for four months. The old version, it didn't upload anything for four months. I didn't know it silently failed. So if you don't upload, if you don't constantly train your agents, they will give your customers wrong answers. So I think the second part was Clay changed their pricing. They didn't train their agent with the new pricing. So that was one of the reasons they gave you bad answers, right?
Yeah. Because when I flagged this to the Clay team, I was like, 'Hey, just so you know, like obviously we use a lot of agents. I'm a super fan. I use this a lot. Very in London. Like we're fans, but here's why I almost rage quit, right?' And so I sent it to all the folks I know there and was like, you know, just this double handicap of it trying to charge me more, use a more pricier model, then you raising prices, then the agent telling me I had to upgrade to get something done, which wasn't necessarily even true, right? It just told me, it just made that recommendation. It's also telling me to use credits I didn't have to use. I was like, you may want to look into this because I don't know if other people would be as patient. I don't know if it'd say, hey, this is mission critical. I got to get it done. I'm going to use the credits and pay the extra cost. Like I feel as a customer like I stuck with it, but I think other folks would be fairly frustrated. And so it's just a reminder for the folks listening, right? Like I don't want you guys to ever... and we're sometimes in the situation too, to Jason's point, like we're sometimes on the reverse side of this where with our agents it is a lot of maintenance, it is a lot of training to keep it up to date. But because they hadn't trained their Sculptor fully through every scenario, which I know is tedious, right? It's totally tedious. Like that's why... but that's the job. But that's why it was trying to get me down this upsell upgrade path when it didn't need to.
Yeah. Another example, then let's move on to the next topic. I was using the AI agent on HubSpot's website on the homepage, right? Massive traffic, right? Trying to get a pricing quote for what HubSpot would cost at our scale. I just could not get an intelligent answer.
Oh, yeah. I remember this.
I don't know whether it was... Now, it might have been because they really want you to talk to a human. I think that was part of it, but it could have been better trained. And so all these stories are that some of the best folks, Clay here to a smaller extent, HubSpot, others, like AI support is great, AI agents are great, but someone, you got to read a segment of these chats every day. Like forever you have to read these interactions with the agent because they're going to drift. You're going to forget to train them. They're not going to ingest documents properly or you're going to think they say things that they don't or your testing worked fine and in reality it didn't. So, these are endless stories of agents not being as well trained as they could be that are customer facing.
On the flip side though, we had somebody this week reach out to our events team frustrated because they thought the answer they got from an AI was not good enough. And it's funny because I was like, 'Sir, actually the answer you got was from a human being.' You flew from...
Did you make a mistake or was it... Well, it was from a human, right?
It was too... they made a mistake when they bought their ticket on SaaStr and then one of our humans on our side, minor mistake, super minor mistake. When they bought their ticket, they accidentally bought two. And so that is a human thing we have to resolve on our end. I don't let the agents right now, any of our agents touch our back end of ticketing and Stripe, right? Like that's something we have to do and it should be something we do. So this was a flow where he sent it to Amelia AI. Amelia AI escalated to a human as she should have down this path. One of our humans, our events team initiated a refund as they should have, the way they should have. The human on the attendee side said, 'Hey, you refunded me but not with the ticket processing fees like fully.' It's this weird thing in Stripe. There's like a 3% ticket processing thing.
They didn't get the 3% back or...
They didn't get the 3% back because that's our standard is to usually when we do a refund. Now, this was a case where it was our fault because the website glitched and he got two tickets at the same time. We should have just refunded that 3% in the first case. The team didn't want to bother me and be like, 'Hey, can I have an exception for 3%.' And so, they just refunded him under standard. And this guy complained after the fact and said, 'Thanks for my full refund now, but if this is how your AI support is, I'm really worried about the content it's after.' And I was like, 'Sir, I understand.' I was like, 'Yes, sir. I understand the frustration.' But it was us. It was a person. It was multiple people.
It would be interesting if digital Amelia could have identified the issue of the Stripe fees that a human might not have thought through.
It might have done better here probably, right?
That's the irony. It was a mediocre... right? It is pretty...
Okay. The next topic I want to talk about, I sometimes I'm slow, guys. It sometimes takes me a little while to figure things out. I want to talk about... I want to share a story from this week, but a lot of folks read what we write, especially on LinkedIn where people really like to be Yahoos and they're like, 'Oh, SaaStr's results with its SDRs and GTM agents. Those don't work at my company. Those are unique to SaaStr. SaaStr is such a strong brand.' Or 'SaaStr does events and tickets, it's not the same for me.' Or 'SaaStr is exaggerating' or whatever. We're just trying to help. But I...
See this, I see this all the time, and what I basically tell people is like, we're just doing the same thing you should be doing. And you might get better results than us, you might get worse, but try it, right? It's working for us. But I didn't completely get why it works for us until I had a meeting this week. Even after all these months, even after Agentforce, Monaco, Artisan, Qualified, Momentum, all the why does it work? But I was meeting this week with the CEO and CRO of a great public company with thousands of sellers, thousands of sellers. And we were talking about agents and AI. And you know, some folks do get a one-on-one version of this show. And the first question the CEO and the CRO had is said, 'What do you think we should do first?' And they had some agentic products, but they said, 'What do you think you should do first in order?' And I went through a series of what I thought they should do first. First, do something so that everyone can instantly get great quality questions answered on your website, going to what we've just talked about, right? Make sure they can instantly get an appointment set with sales, right? And I kept going through different things, and then I realized I went through all the things. I said, 'You know what? You know why all of our agents work in go-to-market sales? It's because we lead. There's no lead left behind.'
This is the flip side. We talk about how agents give you more coverage. We talk about how agents don't mind the B leads, but it's simpler than that. If you deploy some of just some of what we do in AI agents, and then Amelia can share some details better than I can. Here's what you know: anyone that has an issue like with that 3% Stripe fee gets to talk to an agent and then a human. Anyone that wants to know if they can get a discount, whether they get one or not, can find out in real time. Any sponsor that clearly doesn't have enough budget to be at SaaStr can still find out if they can sponsor. But a human might be like, 'They raised $2 million at demo day. Like, we don't want them.' But the agent doesn't judge bootstrap, the agent doesn't judge any issue. And then, you know, and then for our use of Agentforce or others, you know, we were inspired by Marc Benioff to go back to our prior leads and prior ones that no one had touched, right? And so clearly we have evidence that in many cases our agents do better than a mediocre human. I know it's a triggering way to say it, but it is true. But even if it's not, it may be simpler than that. It may be just if you touch every single lead, every prospect, every customer the way they want to be interacted with in real time, you have to do better. That's all these agents have to do to add value to the organization is that there's no lead, no prospect, no customer left behind, whether it's in pre-sales, sales, customer success. We can talk more about QB or AI, but no one gets left behind. I think it's that simple. What do you think, Amelia? I know I'm slow, but this was my aha moment. And then with this public company, we just talked about all the holes in their funnel, and as well as they're doing, there's so many ways to not get touched.
