Thanks for coming. We'll let folks stream in for a minute, but if you've been into the bunch of sessions today, we're going to have some fun.
So, most of this all, as you guys know, is built on Replit. So, thank you for building this platform. It has been quite a journey. So much has changed. If you guys have been through this half day to kick off, I did this 101 on the first agent we ever built 42 months ago on Delphi. Amelia talked about building your own 10K on Replit. Now, I want to take advantage of Amjad, who co-founded Replit back in 2016. AI was a twinkle in your eye back then, right?
Was it even a twinkling in your eye in 2016?
It was. There was a brief, people forget, but there was a brief language model, it wasn't a large language model, but there was a brief language model hype where, you know, Facebook had this like Messenger M product. There was a bunch of companies that were doing like email bots. Everyone was doing human in the loop. Everyone was doing like, you send a message and we'll try with AI. It's usually just people behind the scene. And so, it didn't last long, but there was a feeling in the air that language models are going to be significant. And there was a group of us that understood that language models are going to have a huge significance on software.
So, if we have time, we'll go back to the beginning getting Replit here, but I want to talk about the present. There aren't that many better futurists than Amjad, but I don't want to talk about 2028. I want to talk about the agents we built on Replit and maybe a little bit into the future.
I think Jason, you live in the future. And yeah, I'm not saying that lightly. Like, if you think about the recent agent hype started around December 2025, you've been in it for like 3 or 4 months, already understood the prospect of it. I think you and a lot of other Replit users understood that agents have arrived since agent 3, I would say September, October, that oh these things can run for an hour and like refactor my entire app and do all that. Engineers actually figured out a little later. And you know there was a tweet by Elad recently, Elad Gil, saying oh people at the labs are 6 months ahead of Silicon Valley. Silicon Valley is 6 months ahead of New York. In reality actually Replit users were like 6 months ahead of everyone else even people in the labs because when we launch these agents a lot of times we get researchers messaging us it was like oh we didn't know Claude was capable of running for you know 10 hours at a time and like we're learning a lot from you.
Yeah. Yeah, it is. We're living in the future and I think if people, it's a good point, if people take anything out of it is that you if you want you can live in the future too.
I actually think engineers are laggards. Not only engineers but like a lot of engineers are because they're kind of tied to their workflows and I think you see it now with 10K agents like 10K. Really no one is building agents like that.
That are actually employee-like and actually and I've heard you're going to have someone reporting to them pretty soon.
The goal, that's the goal, yeah.
But yeah everyone talks about agents. The only agents that work are software agents. Marketing agents, sales agents, all that haven't really worked very well and so 10K is an example of something that works.
You're listening to the official Saastr AI podcast brought to you by... Hey everybody starting a business can get expensive fast. One place here, email somewhere else, business phone with another provider. Northwest Registered Agent gives you a complete business identity in one place with free tools, resources and built-in privacy from day one. Get more at northwestregisteredagent.com/sastrfree. That's northwestregisteredagent.com/sastrfree. Saastr free. Starting a tech company is thrilling. It's also really messy. Just got to ask anyone who's done it or anyone who's tried. Better yet, ask EY. They've seen startups in their best moments. Go to ey.com/techstartups to learn more.
So, I want to, there's a bunch of stuff I want to chat with you, but first I want to get some of your reactions as one of the creators to all of this to a few things. So, just a reminder, I know you know a little bit of, we built two, well, we've actually built a third you haven't heard about. So, 10K was the first one. Becoming autonomous marketing, QB's customer success handled everyone out there better than a human I want to show you. Then we have a third one. I know you're not going to like this. That manifested itself, that created itself on Replit, which I'll show you.
So, but I want to get your, we've talked a lot about this, but I want to get your perspective and go deep. So, first cheat code I want to is that the beauty to it is it knows us cold. Amelia I think touched on this prior session, but literally every piece of data you can find, some of it is pushed into the Replit database and to Neon or whatever it is. A lot of it is pulled in real time out of APIs, Salesforce, niche apps you've never heard of like Bizible and Marketo and others. But, the amount of data the agent can access and process is more than any human.
Yes. I think we've already crossed the threshold for which the context links is larger than any context a human can have.
And I think that's huge because I mean, just think back 2 years ago we had 16K context length. Right now we have over 1 million. It's a massive difference.
And how, and I want to show you what, why this chart is sort of interesting, but it's a million. Help me understand. I just want to get a little nerdy for a second. When I'm running the agent in Replit perpetually, not refreshing it, not, which is my passion. Never, never, in the old days when we started in Replit, you like rebooted the agent three times a day. Now I mean I just go with never, right? It's compacting the data. You're doing it, Claude's doing some, you're doing some. How long can that context window last at a practical level? Forever or and where's it going?
Practically indefinitely. Now, it depends on the compacting algorithm and how good it is. We actually think ours is much better than Claude and many others on the market.
Are there two or are you, are they both running or only you're only running? So you're not using any compaction from Claude. It's just Replit's compaction.
Ours is better. And I think the thing about the primitives that the labs build, they tend to be very generic. And so I think every app will need to, you could start with a generic primitives, but for it to perform really well on your benchmark, on your users, every use case is going to be different. And so we care a lot about like, you know, certain pieces of data around the architecture of the app. There are certain things that absolutely need to remember. Like if it forgot that it has a database, like that's catastrophic, right? So there are things that we really care about preserving. But then there are a lot of things that are like bugs and things that you solved. Actually, if you keep them in context, agent performs worse because it is confused by the history of it. So you need to be very prudent about knowing what to delete and knowing what to keep. In the background, we have sort of a graph-like structure of understanding that the different memories. And we do compaction on, actually, there's like multiple layers. There's compaction, but there's also writing to long-term memory, and long-term memory is today is markdown files. So, raffle.md, I'm sure you look at it sometimes. Raffle.md is an example of that, but there are other files that the agent can write to. And you could also prompt it to write its own long-term memory. You can say, you know, use this knowledge base, and you can also have the knowledge base shared between you and the agent, so you can always kind of keep track of what it's doing.
Now, you said, okay, let me get your insight. You said sometimes with the long context window, the agent will make more mistakes because it has more bugs to put. But let me give you, we haven't really talked about it much. If you go to saastr.ai, we probably have 10 apps running in one codebase and one URL. We have a website. We have a fun tool to help you value your startups. Maybe folks have used it. It's been used over a million times, which is crazy. We have an AI that grades startup pitch decks 4,500 times. We now have a one that grades all how agent-friendly APIs are. Okay, all these different apps. When I go to build a new one, it pulls, it remembers. He's like, oh, we did that with the pitch deck grader. It didn't work that well. Let me do this new app a different way. Maybe it gets confused on bugs, but the power of the agent remembering how I built other apps to build a new one, I find it kind of amazing.
My point is bugs that it fixed should be removed from context.
Because then it will get confused, but architectural decisions on how it built things in the past is very important to stay in context or stay in long-term memory and be able to pull it in very easily. But you bring up a very good point. With agent four, going from agent three to four, one of the big architectural decisions that we made with agent four is we made it into a mono repo. What you described is a mono repo. Like if you go work at Google or Facebook, usually everything is one mono repo. And the reason they do it is because they want you to be able to use packages across Gmail, across Google, or Facebook and Messenger. And actually this is doubly important with agents. So, when we embarked on Agent 4, the thing that about Replit that felt kind of a little clunky is that for you to make a deck, you need to go to a different Replit, create a new project. For you to make a mobile app, you need to create an entirely different project. We're like, 'No, everything needs to be in one place. You need to be able to run your entire company in one place.' Ideally, you need to be able to see everything laid out on the canvas. I don't know how much you use the canvas.
Like the idea, I mean how the other day I was talking to a company fully bootstrapped started on Replit. Five, six people now. They started with a couple people, but then they've grown. And they have everything laid out on a canvas on Replit. They have their mobile app, their web app, their back end, their admin, their back office, their automations. I'm like, 'Man, I wish I could run Replit like you know?' Because you can see everything in one place. And now with the new collaboration primitives that we have, where you can start new tasks, new threads, and so people can work simultaneously on the same projects, I think this idea of a mono repo will become more and more important. You want the agent to have access to global context as much as possible. Not necessarily all in the context window at any given moment, but it will know it will have enough pointers that it will know to go search the files and bring in the right context at the right time. So, I recommend everyone here, whether you're using Replit or not, really think about providing these mono repos with all the information you want. We have a data agent at Replit built on Replit. But because data at Replit is so big right now, we have terabytes of data across BigQuery and Databricks and all the other things. And all the schemas are different. And so, we created this repo with that has all the information about all the fields and everything the agent learned about our data because, you know, there's a problem now when we talked about it earlier where agents can write back queries that end up costing you millions of dollars in BigQuery or Databricks or Salesforce back ends cost. So, it's important, especially when it comes to data, to write out the information, have the agent continuously learn how to better query the database.
Yeah, the, that maybe just to summarize it at very lay level, the mono repo, if you're building on something like Replit, maybe it's a hack today. Maybe it won't be a hack later in the year. The more you put in a single app, the more power you're going to get rather than breaking it up. It's very tempting to break things up into 20 apps, right? But I found and there are powers to it because in some ways you get the benefit of a clean database, separate data and yeah. New clerk instance or but you lose, but there's so much power if you guys, Amelia talked about building 10K and we have 10K and QB, we'll chat about before we end our AIVP because they're different. It'd be great, in some ways it'd be great if they were the same app, right? They communicate through APIs.
But they can't learn from each other the same way it...
They can't share context.
Yeah, they can't quite do it, right?
But just on this and maybe we don't have, we can move on, but this is all of our social media numbers. 1.27 million social media followers for Saastr. I can see it all over time. This is all built, 10K built this for me. We used to have an admin who spent 10 to 15 hours a week building this.
And she would go to, there's a lot to, a lot of these aren't even exposed APIs. She would go manually put it in in a Google Sheet, collect it, forget to do it. Now 10K can do all of it between scraping and APIs, and then I can talk to this data. It's a massive amount of data it's collecting. This isn't even all people that are here 10 years of attendees. This is just social media data that was unable to be acted upon until it was inside of an agent and inside of an application.
Well, as I understand it that like when you were running SaaStr whatever 5 years ago, you had like 20 people...
and now you run it with two people and multiple agents.
Is that the rough numbers like an order of magnitude less headcount?
Yeah, but you can do more.
Yes. But this one and then this person just quit after 5 years. I couldn't, she couldn't stand doing these Google Sheets of how many Twitter followers I had after 5 years. It was that, you know, the agents don't mind mind-numbing work.
Right. I mean that's that don't get discussed in the job displacement discussion is that there's a lot of jobs that suck that agents are just better at.
People don't want to do them, right? All right, so let me show you a little narrative to get your thoughts, right? As founder and CEO of Replit. So, here's 10K on the side. This is me talking and again, you can tell me whether this is a hack or not.
10K is an application and it has a chat window and it can be accessed on the web, but we run it almost full-time in Replit and have the Replit agent talk to the app.
Right? It's smarter. And I asked them, 10K, like you might a VP of marketing, are you sure all the VCs that came to SaaStr last year, are you sure this is all the VCs that came last year that aren't coming this year for new email campaign? Sounds like fair concern. I screwed up. I missed some titles. I found 137 VCs that came before I forgot to invite, right? Pretty interesting that, and is this weird us using the Replit agent to talk to because we're staying in dev mode.
This doesn't work in production, does it? Unless I'm missing something. Replit doesn't think it works in production. I don't think it works in, so we're hacking the Replit. I know I'm not supposed to personify it. I know it's an editor now, but we talk to the Replit agent to make the application work better.
Yeah. Yeah, so this is amazing and this is why I say Jason is living in the future. A big part of the next iteration of Replit, if you guys are following what we do every 6 months, release we kind of a massive jump in capabilities between agent 1, 2, 3 and 4, everyone is almost an entirely like new product, new set of capabilities. I'm really trying to track, we're trying to be at the edge of what's really possible in AI today. And I think the next thing is to platformize Replit even more to make it so that you can deploy Replit agent to production. Now, the reason this works really well is because, you know, 10K is an app sitting inside of an application platform and it has access to the database and things like that. But, the cool thing about Replit agent is has access to all its skills. All the skills in the world, we can download any skill once you got to kind of ask a question it'll go download the right skill. It's access to all of the connection. Any connection is like one click away. Salesforce, what have you. It has access to the file system. This is huge. Agents really like file systems. They like them better than SQL.
They're really trained on doing grep and doing kind of search in the Unix command line. And so, because of that power of a general purpose coding agent, you find even more success of talking to 10K through Replit Agent because Replit Agent can, you know, do a lot of other things and query a lot of other sources of information before it kind of relays the information back to 10K. So, but that's what we want to do for everyone here. They don't have to hack it. We're going to be able to deploy a Replit Agent into a production environment and have it be served publicly to your users.
Okay, so let me show you what happened next. So, I with Tocaly Asian, it said, 'You're right, I'm wrong. We forgot to invite these 137 VCs here.' And then, here's something I don't really understand, Amjad. So, we've been playing with 10K for months and we have it draft emails, okay? And it's using the application that we vibed and it's using some cloud and using whatever is and the emails were okay, okay? That it drafted. Then, I said, this is literally last week because everything's getting better, but I don't always know why. Then, I said, 'Okay, draft me an email to these 137 VCs.' And it drafted one that one of your investors, okay? And it was like, 'To Bloomberg Beta.' I think they're one of your seed investors. And I said, 'Write James isn't coming back. Write him an email.' This is I told I just said like you would say to a human, 'Why the hell isn't James here this year?' You know, he's been part of SaaStr, why isn't he? Look at this email he wrote. It's next week, not pitching you. Replit's here in Series 4. Now, think about the data point. Here are 25 people coming from Replit. Amjad, Michelle...
Look at, look how good this email is. All these people and everyone else, the competitors, the adjacent folks, they're all here.
Look at what a crappy SDR AI email was 12 months ago. This is so good.
A little hard for you to see, but how does it so good?
This, if you did this like a couple months ago, it wouldn't have...
Yeah, I'm not even talking about December and four or five. I'm literally saying last, when I showed this to Amelia, I did this last week. Like, what is this? May 8th. My jaw dropped because I'd never seen an email this good.
Right? Look, I think I'm, I got a lot of feedback from folks that got different versions email. I've, that's the best one I've ever seen. But why? What changed or is it just getting better at because it's just pulling data, right? And but it's so, this is such a good email. Why is it so good? No human could write an email, no human would even have the patience to pull every single person of Replit is here, scan 8,000 registrants, figure out who they are, who's like them, create like, how, what happened 2 weeks ago, Amjad? What is 10K's AI? Is it OpenAI? Is it, what is running 10K's AI? Not the Replit agent, is it?
No, okay. I get confused. It can do it at once. A lot of it ironically just runs on Haiku and Mini.
It's just Haiku and Mini, right? And so it's not even running, it's not even running on Sonnet or anything, right? But this was also the agent.
In the Replit agent working with 10K to this email.
So I'll, what I'll tell you how we improve Replit on a day-to-day and week-to-week basis. Maybe that'll give you an idea. Evals are incredibly difficult. It's really hard to know, it's really hard to build an Eval benchmark. It's a research problem. We actually did put out a research paper called Vibe Bench that we're going to be talking about soon. We worked on this for months in order to just like create one single Eval. Now, to evaluate everything that Replit does is like semi-impossible. What we do is a lot of AB testing. And what are the parameters that we AB test on? Yeah. You. One very important one is sentiment. People are incredibly honest with the feedback they give Replit agents. It often they're profane and they'll curse at it and some people say it works better when you do it so. I pass no judgement. So, we will look at sentiment, we'll look at deploy rate. Is any change that we're doing to prompting, underlying model, to underlying collection different models because we use a lot of different models. You know the architect? That's GPT-5.5, right? It used to be 5.4 like couple weeks ago.
I didn't even know that. Yeah.
And we also moved to 4.7. We're also constantly testing the reasoning, how much reasoning we're doing. We're constantly testing our searching capability because that's also a different model. We used a Gemini Flash for a long time. I think we use one of the minis now. And so we're constantly evolving the prompts. I'll tell you the crazy thing that happened recently is we closed the loop and we have an agent internally at Replit that is constantly evolving Replit agent. What it does every night, it will look at all the traces of everyone that's interacting with Replit, analyzes them, finds things that are broken, sentiment issues, errors, things like that. Generates a pull request with prompt changes and sends it to production. That gets merged, that becomes an AB test. We test whether that change is done and then we loop back. We either release it or we kind of edit it. And that's autonomous now. So now we have a self-improving loop. This is the crazy thing that everyone needs to understand today is that we are on the precipice of unlocking self-improving agents. Self-improving agents is like the stuff of sci-fi or nightmares depending on what kind of sci-fi you read. But you know, a lot of sci-fi talks about the moment we get to singularity is the moment than an AI just is improving itself. And we're actually almost there. It's not improving its weights, but it's improving its context, which is as important as its weights. And so, that's all running itself autonomous loop. So, honestly, I couldn't tell you what changed because there's so many changes happening, but we track whether it's improving or not and there's a constant improvement.
Well, look, and I don't know if folks agree if you can see the screen and we'll share all the slides, but until May 8th, I don't believe that 10K together with Replit could build a better email than anybody. But this is a, I know you're not, I know you're a little bit more of a developer and engineer by training. This is better than any human can do.
Yeah, I mean, what you said about telling James Replit is there and all of that. I mean, yeah. Yeah, it would take a good creative person, sales guy, to actually figure out how to...
And they wouldn't send it.
Right. It's still better because it would be typo-ridden if it was a...
Okay, so then it went, then it did the whole campaign to all those 136 VCs, including James. And it came back to us. I just wrapped it up. Here's what I did. We, he emailed 331 investors, inviting them back. Zero send failures for now. And then he put it all into different groups. This is, he did the campaign on his own. 10K, the 331 emails. I mean, it's pretty good. Okay, I want to show you one other data, okay? I know for you guys this is a little hard to see. I apologize. I'm color blind, which is not an excuse for the chart. But let me show you what's going on.
First of all, I could never build a dashboard. Now I get any dashboard I want.
It's really a big deal. So, this is day-by-day ticket sales, free and paid to this event. And the dashed line at the top is when 10K took over marketing.
The lighter dash at the bottom is when poor Amelia had to do it last year. Poor agent.
And here's what's interesting. I don't know why the gap, and you see how the gap grew toward the end?
Is this because 10K got better? Is it because Amelia was too busy last year building this and she couldn't do it?
Is it because 10K sits idle all day waiting for work, but look at how much better.
