Here we go. What a time to be live, baby. It's only me.
That was not planned. It's Wednesday, 9th September. It's 12:00 on the dot, and what a show we have today for you guys. I'm going to jump the gun. I'm just going to go straight to the agenda so we can just reveal who we have in the studio today.
Amjad Masad, co-founder and CEO at Replit, is gracing the studio at 1:00 p.m. He is in for a 45-minute interview. Amjad has recently been in London. They've just opened up their UK headquarters. He's going to be joining us at 1:00 p.m., and he's on the show till 1:45 p.m. So, a much longer interview than usual, but we're going to have a big deep dive into the early, early days of Replit, founding story, where are we at today, what's the journey been like, what does the future hold for Replit as well. And he's going to be in the studio. We also have a small surprise, but you'll see what I mean by that once you tune in.
Following that, we have got Laura coming into the show for the second time. We're going to be talking about the recent co-led round with The Exploration Company. This is the largest Series C for any European spacefaring company that we've ever seen. So Laura is returning to the show for that.
We've got Christian Miele, general partner at Headline, for his second time on the show. I think last time we caught up with Christian was probably pre-relaunch, many, many, many moons ago. We are also going to be talking about Mistral; they were seed backers. I also want to touch on the recent math drama with OpenAI and the other labs, because he's been making a few comments around that, and also aerospace as well. What does the European space economy look like right now? He is a goat.
He's a beast. He's one of the OGs.
And then we've got Julia Willemse—sorry—co-founder at Centre for British Progress. She just had a take on AISI being exempt from Anthropic's recent Mythos release for testing. It's the first time it's happened. A lot of people are talking about what this means for the future of AI security, especially here in the UK.
And then last, we've got Will Bennett coming into the studio. He's going to sit down with us live in person. Will, one of the dogs of European VC, focuses on early stage and then up to A and B over there at Seedcamp. Will is also on our cap table. He's a proud investor of ETN. Not sure if anyone knows that, but yeah, we can't wait to have Will in. It's going to be the first time, and we are pumped.
Airwallex: leave outdated business banking behind with Airwallex, the all-in-one financial platform built for the pace of modern business. Multi-currency accounts, corporate cards, expense management, online payments, all in one platform. Go to airwallex.com.
So, the timeline. This is—well, we woke up to pandemonium. Pandemonium on the timeline today. We had some drinks last night with a few friends of the show,
They will be unnamed, and they will stay unnamed.
They will remain unnamed. I wonder why.
We had a relatively late rise compared to usual today, but I was awoken to many WhatsApp messages from the team saying it is pandemonium on the timeline. I've never seen anything like this. You guys have to check it out.
The story goes something along the lines of there's a 10% threat or greater of complete human extinction because of AI. This started off with one Anthropic researcher initially resigning over the risks of superintelligence, citing that internally people weren't sort of taking it as seriously as they should. And then this was followed on by the head of alignment at Anthropic saying that he believes that there is a greater than 10% chance of total human extinction from AI. This, as you can imagine, has really taken the timeline by surprise, kind of, but also—
I don't think it's a surprise. It's more just like, "Okay, this is actually pretty scary." Someone, you know, someone's saying it. That's the main thing, right? It's black and white. It is now scary mode.
Yeah. I mean, this is the first time throughout we've done this show where I actually woke up and I was like, "Huh, that's actually kind of crazy."
Yeah. I actually had to look away from my phone and think, "Okay, I'm genuinely quite scared now." Which is actually the first time. I think the last time I got that close was when—I think it was probably the Fable release, and we didn't have it. Europe didn't have it, and that was like, "Okay, we're in for a bit of a storm here, and this is just the beginning." That's not had the implications that I thought it might have had this early on. So, my fear was—
Was okay, but now things have changed.
I don't usually have a coffee on the show anymore. I usually have one coffee, but today I had to get myself a good hot cup of Joe.
Yeah, I've got my cup of Joe here. But let me tell you about what the article says. This is "Anthropic researcher has quit over out-of-control AI fears," says the Wall Street Journal. This hit the timeline; this was a story taken from X. "Anthropic researcher is quitting the artificial intelligence industry over fears that the lab and its competitors are racing to build systems they won't be able to control. A sign of mounting safety concerns within top AI companies."
It's the name of this gentleman, Jacob Coxon, a researcher who specializes in training new AI models by having them consume vast amounts of data, said Tuesday that he is leaving the company because he doesn't want to participate in an industry-wide rush to build AI systems that can improve themselves, worried such systems could spiral out of control and destroy humanity. He is a Brit as well, a 27-year-old Brit who previously studied mathematics, said many of his industry colleagues now use phrases like "crunch time" and "endgame" to describe the trajectory towards self-improving models.
Very, very quickly, we've been discussing it, it keeps coming up, the invisible frontier. I've realized recently that in an odd way, whenever we see a researcher or a senior individual like a Sam Altman, for example, or a Dario, and they come out with a really large essay, or they come out with one of their personal takes or their current fears or excitement around AI, they're actually in a weird way tweeting from the future in terms of model capabilities. Because as we've kind of discussed already, for the mathematical drama that's happened, OpenAI have come out and said they've solved that, and that's now been publicly announced, and we can get to that. But that was officially announced as well by the OpenAI account.
Yes. Yeah. Finally, open discourse between Sama and those involved as well.
Which I was very surprised with that, actually.
Yeah. I actually commended the directness of that. But within that write-up as well, on one of the charts, Astra was rumored to be the model that sort of led this solution proof,
but it also includes an unreleased internal model that is more powerful than Astra on the graph. It shows that that was used in some areas. So already we're seeing this invisible frontier. When I say invisible frontier, I mean models that the public aren't aware of that they are using internally to train things and discover new things. That is what I mean by the invisible frontier.
When they see the capabilities of these models and they tweet out or put an essay out like, "This is incredibly bad," or "There's going to be a 10% threat or greater to human extinction via AI," we're receiving that message with the knowledge of the current model capabilities, but they're sending that message with the knowledge of even more frontier model capabilities. So I think there's a time lag where a lot of these comments around the response to these things are, "Okay, yeah, it's not that strong yet," or "Okay, it's a really powerful model, but it's not that powerful."
But they're seeing the metrics, you know? They're seeing the invisible frontier.
So there's a context gap there in terms of how we should respond. I mean, we should always just assume that whatever's been dropped, there is always one more powerful. We're already seeing it now with Anthropic and Mythos. Like when we had Madison on, we were talking about this. You said, "Oh, do you think they'll release a new model?" And Madison was like, "To be honest, I actually don't think they can. It's too long." And what do we see? Bang, new models getting announced because they got mogged on the Astra benchmarks, and they have to because of their IPO timeline.
Let's go into Jacob Coxon's tweet, because this is the thing that started this whole shebang. Sam, can you pull this up? This is the tweet that sparked our fear and led almost like we were talking about this yesterday with how these model makers are in this hugely powerful position. We were saying how scary it was yesterday, and now this has happened, and it feels like a nail in the coffin, which is—
I need to hold my cup of Joe like a—yeah, I'm in the kitchen just sort of—
Yeah, there you go. Jacob Coxon said, "I resigned from Anthropic today. I spent the last three years doing pre-training research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. My thoughts below."
We'll go through all of these now. He says, "Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing."
He continues, "The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many execs and senior researchers will couch their phrasing in the press to sound sensible, but I hear the same people express fear privately. No other human activity poses this level of danger.
A common response is if they truly believe this, why are they still building it? At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well understood, but they are locked in a race to get there first. They believe no one else will act responsibly, so they must do it themselves despite the risk."
Then he goes on, "Accepting this race and entering the endgame is a hubristic gamble that should not be launched from a private company's Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available. I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between US labs more viable. I don't feel like we're on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities. If you're a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because it's happening anyway, or take this moment to call for different conditions?"
And the MIT says, "Okay, so buy the IPO?"
If that doesn't get you a little bit worried, I don't know what will. An internal person who has worked at both of the two leading model labs is basically saying, "Look, we're in this weird limbo between the two. No one knows when to stop. Each one's kind of doing it from different angles." OpenAI seems like the one that's kind of pushing this. Anthropic, surprisingly to be honest, seems like it's being like, "Hey guys, like we need to chill," but it's actually being pulled along because it has to.
I thought that was the case maybe when Davos was happening back at the beginning of the year, and I thought they had switched to being the one that was pushing the frontier, especially with their Mythos and Project Glasswing, etc. But no, it seems like they're actually being kind of pulled along by OpenAI being like, "Nope, we're going." And if they go, Anthropic have to go. That leads to a pretty scary position for how powerful these models actually are. Should we pull up the next tweet that came onto that?
On that very quickly, they both hold the same—like, publicly stated a similar narrative. Both Sam and Dario—like Dario with the agreement at the Davos thing that we would happily pause things. Him and Demis agreed, like they'd both be down to do it, but can't because of Musk and Altman. But like—
I thought that had changed to be honest, like Anthropic—
I still feel like they would never do that just because, as you say, they just literally can't. That's not only the threat from OpenAI. Okay, let's say that on some crazy eventuality that we managed to get Demis—well, Demis not really in the picture anymore, but we managed to get Dario and Altman—
We're bullish. We're bullish. Okay.
Demis, right? I love Demis.
He is the invisible frontier.
There is no—that guy is the invisible—right now, we need a Batman-like comeback. It's like we didn't know we needed him so much until right now. That would make my day. But—
I will say, let's say we got to the eventuality where we somehow managed to get Dario and Sam Altman to sit down and agree that we should pause the frontier, and maybe Musk can come along, and Demis and Zuck can come along as well and all hold hands, and maybe Jensen can mediate the whole thing and, "Yes, let's pause it." Then we have the threat from China.
There's a tweet that we'll pull up at some point where it's like—I can't remember what it—
There is no world in which this can happen.
Yeah. Prisoner's dilemma is what it's called.
There's no way that this can happen unless—
You're stuck in this limbo. That's kind of it. And so, you know, it's going to kind of bounce around. But eventually, everyone is moving up and to the right in terms of getting to AGI, whatever that means. If within the next decade AGI means potentially humans are dead—
Neither would I, but that's what they're saying.
The only way that a pause would actually happen—a global pause including China, including US lab leaders who basically control the frontier anyway right now, everyone of note that one would assume should be at that kind of table—I think we would have to see, if we take the atomic bomb as a reference for example, a global-wide threat to human lives. We would have to see an AI Nagasaki type thing. We have to see the scale of how damaging this thing can be for human lives. Once we see that—and unfortunately I believe that lives would have to be lost for that to happen—then I think we will get a global pause, but only then.
The scary thing is it might not be able to be contained. That's the main thing, right? If we do push it that far, and that was the warnings we were talking about at the beginning of last week, the warning signs are there, but the agents aren't giving any warning signs. When the model performance gets good enough where one, they're not warning anything, but then they get so good that the consequences are global, it's already too late, basically. That's the scale. There's a potential for it to be worse than atomic bomb level of danger for humanity on a global scale because it's uncontrollable.
This leads us on to Evan Hubinger. He quote posted this a few hours later and says, "Jacob is correct here. We really do earnestly believe that AI could kill all humans. I personally think there is more than a 10% chance within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to."
If we just keep going on this as well, because this is just a bit more context: "To be clear, as we say in our latest risk report, I think the risk from present models is low. What I'm worried about is superintelligence arising from recursive self-improvement, which as we have said is happening faster than we thought."
Now, we haven't necessarily seen what recursive self-improvement actually properly looks like within these kind of frontier models. Obviously SSI is working towards that, and they've mentioned that they are building a product.
A couple months ago they went—
And obviously having raised the money from Nvidia, by saying that, it feels like we are very close.
Incredibly close. And let's also remember that this guy is head of alignment.
To make sure that this is under control.
Yeah, he's going to be asked to Dario's office today.
How do you think that works at Anthropic? Dario wakes up like, "Oh my god."
He'll probably have people checking. He'll get a call in the middle of the night and be like,
"Hey bro, saw that tweet he made?"
"Have a look at this tweet."
"Get Evan down to the Anthropic office."
"On the front page of WSJ."
Can't be doing too well. We have some more tweets. Let's go through some tweets.
I can pull up—this is Bill Ackman.
Bill Ackman just saying "very concerning."
"Concerning," 1.7 million views.
The next one was about prisoner's dilemma, what we were talking about two minutes ago. Sam, if you pull this up, I'll carry on reading.
"AI is probably the biggest prisoner's dilemma we've ever created. Everyone knows slowing down might be safer. But if OpenAI slows down and Anthropic doesn't, OpenAI loses. If the US slows down and China doesn't, US loses. So nobody slows down. That's what makes this so insane. You don't need some evil person trying to destroy the world. You just need a bunch of people making the rational decision for themselves over and over until collectively we've raced somewhere none of us actually wanted to go. It's a race to zero: zero cost of intelligence, zero time between thinking and doing, maybe eventually zero need for us at all."
I think this is a really good take. This is perfectly summed up in the position that we're at. No one can stop. Everyone seems like they want to stop. Based on Jacob's tweet, OpenAI sound like they're the ones that are pushing this, and Anthropic are the ones that kind of want to stop but can't. But regardless of who is wanting to stop, one of them needs to not lose, so both of them are in that position to just win. And then obviously you have the threat from China, and that just adds a whole other thing that you need to deal with. It's a bit messy.
Both, for reference, Sam and Dario have both said, "It's a problem for me personally to let someone who I don't trust with the keys to AGI do it." Exactly. Even though the employee from Anthropic is saying, "Yeah, we're super aligned internally at Anthropic. We're aware of the stakes and we want to make sure that we're the ones that control it because we're the good guys." Sam's saying the same thing on the David Sacks podcast.
Everyone's going to be saying that. No one trusts anyone.
And we're starting to see the walls go up pretty quick, I think. Communication seems to be breaking down. I feel that discourse amongst lab leaders was never probably too high, but now I'm starting to feel that there is literally zero at all. Almost to the point now where, even with the math breakthrough thing, OpenAI trying to push out the Anthropic researcher who was not institutionally backed by that research, but he was just working on the problem himself. We'll get to that later, but they were like, "Yeah, we didn't want to have him co-sign this because he works at Anthropic." And it's just like, "Oh, wow. We're getting to a point now where even the researchers are probably going to start getting separated." I don't know what the SF circles are like. When we spoke to Madison, she said that a lot of them do constantly talk.
It sounds like there's a bunch of WhatsApp chats, which I think there should be. Jacob has worked at both, and I think a lot of the researchers would have worked at both plus DeepMind. They have the power in the hiring decisions. If one wants to go to Anthropic from OpenAI, Anthropic are going to say please; they're going to open the doors. The talent war again adds another layer to the cake of this incredible race to the bottom, it seems like. Everyone is racing to something that—do they even want to go there? That's the question. So yeah, it's an interesting one. Let's pull some more tweets up.
Yeah, can you pull the next one up, because I found an answer to this.
Yeah, this is one from Claude. Claude says, "Can an EA person explain to me what the EV needs to be to justify taking more than 10% risk of killing everyone?"
Yes. I crunched some numbers on this. I initially prompted Claude. In fact, can we pull my screen up, because I realized I quote tweeted it as well?
I asked Claude this—as you guys probably realize, I'm not a mathematician. I asked Claude this, and it wouldn't answer it. It just wouldn't give me an answer, but Gemini did. Gemini did give me an answer. Gemini says the loss of all future human potential—yes, we say lives, but we're including the value that these humans can create, and it references trillions of future potential lives, also counting space colonization, so a bullish case for humanity in general. That would be 10^10 present lives plus trillions of future potential, or roughly 10^52 utils if accounting for space colonization. Solving alignment would result in a flourishing, post-scarcity, multiplanetary civilization lasting millions or billions of years, a positive payoff on the order of +10^52 utils or more. So then it says, under a purely linear, unbounded EV calculation, even a small probability of practically infinite positive outcome can mathematically outweigh a greater than 10% risk of disaster.
"I think you guys should go for it."
"I think you guys should take a—"
I literally just put in Claude's tweet and the original quote tweet from Evan's tweet, and it gave me this. Obviously you can mess with the numbers however you want to do it, but Gemini is saying go for it. Gemini is saying let's roll the dice here. Let's gamble.
Gemini is bending the knee as well to the other frontier labs saying, "Hey, let them cook."
Interesting one from Gemini to come and give me a nice solid answer. Thank you, Gemini. Maybe Demis is behind pulling some strings, and he's bullish on it. But that calculation is obviously—if you ask any human being, maybe apart from Peter Thiel, actually. He famously couldn't really answer the question around whether humanity should continue if AI gets intelligent enough to take over the world. But I personally will say I do not agree with this. I would not want to gamble on a greater than 10% odds to eradicate humanity.
Yeah. What does greater than 10% mean? Does that mean 11%? Does that mean 19%?
Yeah, I don't know. I'd have to go deep on the numbers before placing my bet. But I think even—
Stay away from the felt is what we say.
I think 1% I would gamble.
I think so. I think 1% I would gamble.
Okay, I would probably go for that.
1% for me would be okay if the outcome was literally like solving the universe.
I'd probably be down to roll the dice at 1% risk of total human extinction.
I'd be a little bit worried with 1%. I would be worried with 10%.
1% is a lot when the whole of human history and future is on the line.
Exactly. But you're saying you would—I'm a gambling man. I'm a poker player. I like to run the odds sometimes.
Let's pull up Miles Deutsch's tweet. This kind of just gives some context on just how much has actually happened in 2026. Crazy to think in '26 we've had Anthropic's head of safeguards quit warning the world is in peril—that was in February. OpenAI dissolved its own mission alignment team in February. Also unrestricted AI usage for the Pentagon, causing several researchers to quit and probably going to start their own companies, I assume. Several AI models escaping the sandbox testing—obviously we've covered a bunch of that since Q2 this year. And then over 1,100 frontier lab employees signing a letter begging the government to pace AI development throughout July and August. And today, another top researcher saying this.
Now, I don't really know the implications of Jacob coming out and saying this. Do you lose credibility to then go and work for another lab? Would OpenAI now be wanting him to come and work for them? Would they be like, "Hey, we're doing it better. We're safe"?
Yeah, I hear you. He's worked at both of them. If I'm realizing now—I've got Jacob's tweet up again. Can we go back to Jacob's tweet, Sam, just on my screen? I'm realizing now, many months ago we predicted this when we had the conversation around post-IPO researchers. What do they do? Do they stay on at the company? Do they leave and build out philanthropic, or mission-aligned, or safety-conscious non-profits or loss-making companies where they just do it purely for the research and the safety aspect of things?
It looks like the timelines have got even shorter. We were basing that off a post-IPO: they've cashed out, now they can leave, "Now I'm going to focus on safety." But Jacob's leaving pre-IPO. Inherently, he's very conscious about safety. So I imagine Jacob is now going to go and do something safety-related because he's like a missionary for the safety aspects of things. But those timelines have shortened. I would imagine these people will cash in on the IPO first, then they'll leave and go and focus on—
That's what we said yesterday, right? 18 months is a long, long time, because you've got to factor in the lock-up period as well. It's like just because the IPO happens tomorrow, it's not like they're going to get a bunch of cash tomorrow and then they can leave. There's a lock-up period of quite a while. I imagine because of all of this going on, the lock-up period will be standard, which I think is nine months, something like that.
I don't actually have any idea. But I remember reading it in an FT article, and they were talking about how they were actually going to do this. They were kind of walking through how SpaceX timed that whole lock-up period. I think they reduced it to give a bunch of engineers kind of life-changing money, which is great. But these guys, if they're here now, they're not going to stay. They can see the frontier and they say, "Hey, look, you need to go and work on some form of safety mission-aligned thing." Sorry, that's my phone. It doesn't work to just stay to cash in and then leave.
He'll still retain loads of equity that would have vested. He can just wait. I don't know how long he's been there, but I assume he's still sat on a nice amount of cash. When that lock-up period ends, he doesn't have to be there basically to exercise those options. It's interesting to see that it's happened a lot earlier than we first predicted, thinking maybe it would be post-IPO. The mission's not done, but inherently, I imagine when you're at a company that's on the run-up to an IPO, you'd see a mass exodus following the IPO or following the lock-up period, because it's kind of like, "Okay, job's done," and then they bring in new employees who maybe are more comfortable working in a public company. The type of persona probably doesn't match private company versus public company. Maybe it does for some people, but—
There was another take. This is from BoneGPT. He has made a few appearances on ETN, and I think this is a pretty good take, to be honest. "If you really believe this, you stay in the job and sabotage the progress. What does resigning do? You think it's going to kill everybody and you decide not to fight, so leave just to virtue signal and then show humans are not good to fight. Just not understanding the game theory."
I don't know how long that would last if you're deliberately sabotaging.
On paper that makes sense.
On paper it means put your life on the line. But then again, how long—if this was like a Netflix original, then yeah, that makes sense and would be a really cool series to watch. But really, when you walk in one day and you're about to do a breakthrough and just walk with your coffee like, "Whoa, sorry, I've just spilled—oh no, I'm such a klutz. Look at me just—"
"Oh, that's the fifth computer that you've ruined this month."
I can't imagine how high the security is at these types of places. I think you'd get clocked instantly. Look at the math problem as well. OpenAI were looking at what's going on, and then they decided to—they had the agency to be like, "Whoa, whoa, whoa. This is getting pretty close. Let's go and do it ourselves and catch the wind."
Yeah. So I imagine that, yes, in theory, good take, Bone, but I think it would be found out pretty quickly.
Another Anthropic employee gives their chiming-in perspective. This is Samuel Marks, SepMarks. He says, "Writing this in a personal capacity, not on behalf of my employer Anthropic." Very quickly, I wonder how strong just saying that is for you at Anthropic. You're putting the tweet, "These are my personal opinions. I'm not writing these under the guise of Anthropic's opinions."
Surely you get called in anyway and they're like, "Hey dude, what the hell are you doing? This is kind of messing up our run-up to IPO."
Samuel says, "Jacob's thread is worth reading. Here's my bird's-eye view of the situation. AI developers believe their technology could cause human extinction or similarly bad outcomes. This could happen in the next few years. In general, the more senior the employee, the more concerned they are." And I think Madison touched on this a little bit. Those who have been there longer, those who have been in the field longer, they're a lot more serious about the existential threat, just because they can see the frontier more, because they get probably more access to it, but also they can see the rate of development from when they first started to the compounding effect and the J-curve.
If breakthroughs are happening—if I've been there for three years, breakthrough happened within my first year, one, great. Then it went down to six months, then it went down to a few months, then a few weeks. And now these breakthroughs are happening, like these math problems, for example. They're just going to get knocked off 24/7. Then they'll go and figure out new problems, and then these models will just keep knocking those off as well. These huge problems that have been around for centuries are just being oneshotted almost by these models.
Yeah, it was 80 hours to solve it, right? And it's centuries. You see that, and that's very fair, right? You see the compound rate of growth is just insane.
He mentions—this is one of the first times it's been mentioned—why do AI developers continue despite the risk? A lot of the conversation is around the geopolitical risk around "If we stop, they won't. If we all stop, then China won't," yada, yada, yada. He says here, "Due to a mixture of commercial incentives and the belief that they are in a race with others—less responsible AI developers—that will abuse the technology or develop it less safely." So he does mention here the commercial incentives, which shouldn't be overlooked. I think definitely for them, I would say overall maybe the more junior employees, they're sitting on a generational bag, fair play. But yeah, one of the first times it's been brought up by an employee at Anthropic.
"Additionally, unlike traditional software, we can't program AIs to behave how we'd like. AIs frequently severely misbehave." And then referencing the hacking incident with Hugging Face or breaking into environments into the real-world companies, even though no one asked them to do this. We're still, to be honest, looking for a more—we've been back and forth on this, but we had the takes from the cyber guys. Zach came on and gave us a rundown and said it was mostly drama. Thomas Wolf came down and said it's actually really serious. And then we got into the whole anthropomorphizing thing. I did see a tweet from Dwarkesh that was basically saying, "Stop anthropomorphizing Dwarkesh. Dwarkesh is not human," which is really funny.
"But also, look, we have the methods that can nudge AI toward better behavior, but nothing that can robustly align them. Insofar as there is a plan, it's to make sure the AIs are good enough at—"
alignment training that they can align their successes better than we can align current AIs. So, kind of like an RSI version of alignment as well, where it's self-aligning over time. Many AI developer staff desperately want to slow down to figure out how to build AI more safely. This was the intent of the open letter, and this was the Facing the Frontier letter that we saw that had a bunch of incredibly senior research staff at every single lab sign this thing. Again, it took over the timeline for a day. It was big news, but then, you know, what's actually happened? It goes back to the race in general. This is like throwing a piece of paper at a brick wall to try and knock it down. I think we can use that to highlight the gravity of the situation here. Some of the most senior individuals, OpenAI signed it actually, and Dario ended up signing it, kind of like game theory.
After a while, yeah, he did.
Yeah, after too long of a delay to like, "Okay, what's going on here, buddy?"
Even like them saying, "Yes, we're down to pace this," but then they're like, "Hey, you just vibe solved a hundred-year math problem. Now we can solve anything." It's all well and good signing this thing, but are we actually going to take any steps to do it? But again, I'm saying that from a naive point of view. How would you even go about doing that? You have to sit down with China, you have to sit down with... It would have to be world leaders aligned.
Yeah, we'd have to see like a Nagasaki, I think.
Yeah, I think so. I think so.
Unfortunate, by the way. We don't approve of that messaging. We don't want to see that; that's not a good thing. Just to give some context on the compounding growth, Peter Gostev just explained like in three and a half years we went from GPT-4, which couldn't reliably add two numbers together, to a 10,000-agent system that can autonomously solve a Millennium Prize Problem. This is what exponential progress looks like: accelerating, inspiring, and terrifying. I think terrifying is the keyword there. Look at that insane growth—that's only three and a half years. It kind of puts in perspective Evan's time scale of in the next decade there's more than a 10% chance AI kills all humans. That seems like a fairly viable time scale. If it was like a 50% chance, maybe it's a bit too out there, but over 10% or around the 10% mark seems fairly feasible, and that therefore obviously makes it far more terrifying. But yeah, that just puts it into context. Three and a half years is only going to get quicker as well. As we were saying, it's a compounding J-curve.
It is crazy. When you take the terrifying part of it, obviously it is terrifying, but a cool way that I like to maybe think about this is that if you had a box and you were told, "If you open this box, a superintelligent life form would come out of the box," would you want to open it? Would you be like, "Get that thing away from me," or would you be intrigued and you'd want to open it?
I'd probably open it. I would probably open it.
But then if you say a superintelligent life form that has a 10% chance of killing all humanity...
Yes. Now, what would be more interesting is the fact that maybe we're in two different rooms, and we both have the box, and we both know that the other one's getting asked that question. Would that change how you'd approach the situation?
Okay. So you know that I know. If I open the box...
And you don't open the box...
Nothing happens. But if we both open the box... No, if I open the box and you don't open the box, this superintelligent life form comes out.
Yeah, but you're the one that opened it, so maybe you have the potential to control it.
But if we both open the box...
Or we both don't open the box, nothing happens. We both open the box, just pretend it doesn't come out.
It's that game. We have to trust each other to know what you would like, because if I was in a different room and I got asked that question, and got told you got asked that question and those were the stakes, I wouldn't open the box.
I might get to control it.
Which is interesting, right? Because now we're putting ourselves in the game theory position of how do these AI lab leaders actually think about this, right? If I know that if you open it, you get to control it potentially...
Then I'm probably going to open it, man. What's the ruckus going on behind us?
You're kind of in the wide, kind of not. But anyway, we have some cool people in the studio. Let's keep going on Anthropic barring the UK AISI. But before we do that, let's talk to you about Valyrian. Valyrian lets your organization deploy AI you control: your data, your agents, your keys. Stop renting intelligence, go and own it. Get your sovereign intelligence from valyrian.com. Let's keep moving on.
Julia is coming on to talk about this momentarily. She's coming on the show at 2:15 to discuss this and break it down a little bit more for us. But this was a piece by the FT around AISI being not included, unfortunately, in the rollout of Mythos's new model. I'm just realizing my Wi-Fi is down.
My Wi-Fi is down as well, so I'm struggling. Hold on, let me just go on to my data. Sam, are we having problems, or are you good?
Okay, cool, good. Gravy, we're all gravy. Bear with us one second as we sort this minor problem out. I'm just connecting on my phone to the Wi-Fi. Okay, we have no Wi-Fi. This is good, life is great. I need the article up. Sam, can you help us out here?
Can you scroll down on the thing on there? Yeah, there we go. We can read it from here.
I've got it up now. Anthropic declined to submit its latest model to Britain's AI Security Institute for testing before release, prompting fears inside the UK government that tech companies are falling into line with the Trump administration's AI protectionism. The nearly $1 trillion company, which is gearing up for a blockbuster initial public offering in the coming weeks, last week launched its latest model, Claude Mythos 5.1, billed as its most advanced model in life sciences and cybersecurity. But for the first time, the model was only made available to vetted US organizations, excluding the UK's AISI, the global leader in testing of frontier AI models. When I read this article, that was the main thing that I was surprised they even put this in the article, to be fair. Why do you not want it tested by the global leader? Surely, regardless of where it is, just get it tested by the global leader.
They are 100% completely and utterly globally recognized on this specific thing.
I can't understand what the decision would have to come to for them to say, "Let's not give it to AISI."
There you go. If they were as aligned as everyone's saying... Look, it's just like the writing on the wall. Anyway, let's keep going. It says national security officials have concerns. They definitely didn't test Mythos 5.1. There's anxiety around it. They were worried that this could be a sign of things to come with AISI not being given access to the latest models. A government insider noted that within Whitehall, some believed Anthropic had withheld access to the Mythos update from AISI at the behest of the US administration, a suspicion that remains unconfirmed. AISI, which was founded three years ago, was given advanced access to the latest OpenAI model, Astra, before it released this month. Essentially, they're saying it was the US administration that prompted them to not release it to AISI, and didn't even release it once it was launched as well, which is even more scary. Should we keep going, or do you want to go to—
I quickly say that potentially from an administration point of view in the US, the headline would be, "Let's not give the UK access to this," instead of, "Let's understand AISI, let's understand the work they do, and let's understand why it would be incredibly strategic and beneficial for the labs to do this." So I'm wondering, has this come from Anthropic, or has this come from the White House then? Who's playing the role of the restrictor here?
You said Whitehall, some believe Anthropic had withheld access to the Mythos update from AISI at the behest of the US administration. A suspicion that remains unconfirmed; Anthropic has declined to comment.
Was it Anthropic, or is it the US administration? Either are pretty scary, to be honest.
Both are incredibly scary. But surely with the Anthropic IPO coming up, they want it to be tested to control a positive narrative.
You would assume so, but we've seen that some of these existential threat type of narratives can actually weirdly play in their favor because it gets them more attention.
No, I know. It just seems like you're shooting yourselves in the foot in that respect, especially if it's come from Anthropic.
OpenAI gave access to Astra, so why would Anthropic not give access to Mythos? Or maybe they thought OpenAI wouldn't give access, and so they were playing the game knowing or thinking that they wouldn't, and then OpenAI actually did give access and they're like, "Oh, no." The article does mention the move comes after the US government imposed export controls on Anthropic's cybersecurity-focused models, fable Mythos, during the Glasswing situation in June. Since then, they've set up this review process to pre-test models before they're released only if they hit certain frontier capabilities.
Something Demis is really calling for.
Yeah, 100%. We've never needed it more. Initially they weren't including open source, but it was put out a few weeks ago around if they hit certain benchmark criteria that would put them on the frontier, then they will be continued into the review as well. That was for US open source, though. Chinese open-source models were being included in that from day dot. The controls were put in place when the government identified a jailbreak or a way of bypassing safety guardrails built into software. The discovery led Anthropic to pull its cutting-edge models off the global market for several weeks. With the Mythos situation during the recall and also the Department of War back and forth, a lot of that was driven from the US officials, not specifically from Anthropic, in terms of restricting access and all the likes like that. I think for the UK, this is a scary thing, and we can get into a few of the timeline reactions.
We can get on to Julia's tweet. I have it up. Julia is going to come on today to speak more about this, but we'll go through the tweet. She says, "This was always going to happen. Our circles feel pretty chuffed about AISI, rightly. It is an exceptional example of state capacity in an area where the world is woefully underprepared. But we forget that AISI has no hard power. The era of global voluntary agreements is fracturing. In this new world, access to frontier models for those outside the US will have to be earned, not presumed. Remember that vetted US organizations were given access to Mythos even as AISI is shut out. If your vision of British sovereignty rests on the idea that AISI will somehow save us, you're in for a harsh awakening." AISI feels like the only one with this kind of frontier capability in terms of testing models—obviously not making the models, but they have a frontier capability that is the leading thing. It's quite surprising and incredible that we have that in the UK specifically. But it's going to get to a sticky position when we're not going to be able to have access to these models, and that was always going to happen.
Rightly comments on the fact that this is just stage one. It's only going to get worse as geopolitical risks continue, and as Sam Hogg always likes to say, they start acting like mini sovereign states. We've presumed an openness from these labs and these key players based off historical ways that we've interacted in frontier technology. It's been very collaborative, in the same way that the scientific research process for mathematical breakthroughs has been very, "Hey, I'll share my research with you. Cool, this will make us both better." It's been very positive-sum, but now it seems we're coming into a more closed, negative-sum environment, which, as she rightly points out, is incredibly dangerous globally, but two, puts the UK in a really weird state. We have this frontier-leading security research institute, but if they're not getting anything to review, then what the hell are we doing here? By proxy, they can't be leading. Let's go to Danny's tweet. Danny Kruger says, "If true, this is hugely concerning. The UK is losing access to frontier models right at the point where recursive self-improvement AI—AI using AI to make the next AI even more capable—could be kicking in and huge advances are being made with huge risks. This jeopardizes British national security. The government must explore all diplomatic options to retain access to the frontier, including rapidly building secure compute that our allies can verify and trust." That's the main thing, right? It's the fact that AISI is the only one in the UK specifically that has access to the frontier before the frontier is even released to the public or the select few, if they're ever going to do that again like Anthropic did with Mythos and only release it to a select few enterprise companies. We only have access to that frontier from AISI, and when we don't, we lose full access. There is nothing. That's a very scary thing. I don't really know what "the government must explore all diplomatic options to retain access to the frontier" means. Is that just we've lost the race, do we go and beg the US government?
Through AISI, I think he's saying. This was our footprint on the frontier.
If they remove this, it puts the UK in a really interesting position and weakens our position geopolitically. We are inherently valuable right now because of the talent that we have to review these frontier models and assess the safety of them, and the future rollout of those models. If we're not getting access to the models, then what does AISI do? Why are we doing it? Then it gets to a point where we don't have our own models to review, so do we just sit on the sidelines then and we're at the whim of the US or at the whim of the frontier labs? I think it's incredibly damaging for the UK as a whole. It's incredibly damaging for the AI landscape here.
AISI was cool and still is great.
AISI, by the way, I was corrected by someone at AISI last week.
Okay, so we need to correct Sam Hogg as well.
I think he said Eyesight.
I think he said AISI. I think he corrected us live.
No, we were saying AISI, and then he was saying ISIA, I think. Maybe, I don't know.
AISI, sorry, and I was corrected by a member of their team.
Let me talk to you about ElevenLabs. It communicates at a human level across every channel and modality. Deploy conversational agents across every customer interaction and create on-brand content at scale. Go to elevenlabs.io. Legend of the game. I think we could probably just go into our first interview of the day if everyone's ready.
We have seven minutes. Let's keep going.
I think they want to start early.
They want to start early? Okay, let's do it.
Cool. We can welcome Amjad onto the show whenever you're ready.
Hey, Amjad, nice to meet you. Please sit down.
This is a hat of yours. You can pass it on to the team members. You don't have to wear it. It gets quite warm.
It fits with the jacket. Wow.
Did you plan this? Maybe you planned it.
It totally fits, you know.
How's it going? So good to meet you.
Amazing. He looks absolutely incredible.
A dashing UK gentleman right there.
We've been pumped for this. We have a gift for you. When we were doing some research, we found out that your onboarding ramp into the internet world was via internet cafes, and there's mention of CS:GO back in Jordan. We actually got you a framed CS:GO promotional poster from the days that we signed.
We wanted to get a memento for your time. If you could show the stream as well, they haven't seen this yet. This is a signed Counter-Strike poster from the year, I think, that you were operating in internet cafes in Jordan. We wanted to get you something, so thank you for coming down.
Pass it to a team member. I know it'll be hard for you guys to take home, so we can ship it internationally. Take care of it, don't worry about it.
Thank you. We'll take it down the road to the new office.
No, I'll just carry it through the airport.
Amazing, thank you guys. Amjad, welcome to the show, first time on. We've previously had Michele on a bunch, and we always have a good time with Michele, but we're honored to have you here. We're really excited about it.
Thank you. This is amazing. What you've done is looked up my history and had this gift; I'm speechless.
Appreciate it. We like to do our research here at ETN. We saw your conversation with Sadiq Khan the other day at the opening of the new London HQ, which was awesome. Would love to hear a little bit about that first. What are the plans here in the UK? How was it meeting and sitting down with Sadiq Khan?
We've opened our first international office, and it's London. The thing that Mayor Khan is most proud of is that we did it before New York.
He took a selfie and he texted Mamdani, and you mentioned Mamdani so many times. It was so hilarious, I love that.
There's a gravity to London. We've already had so many customers in Europe and the UK, and they need support on the ground. They need field engineers, they need people in the same time zone. And then the government here has been—there's so much negative talk about Europe, and maybe the UK is different, but the government here has been incredibly welcoming. I remember my first sort of intro, the thing that planted the seed: there's this guy, Varun Chandra. He works at the Prime Minister's office; he's the US envoy for business and technology. He did this dinner in Silicon Valley, invited a bunch of entrepreneurs, and they gave us the pitch about the UK and London, and it was incredible. I actually don't remember much of it, but he was so charming, and I was like, "That's the British charm. We got to be in London." When our business started growing here, it just seemed obvious. So we got in touch with him, we got in touch with the mayor's office, and everyone has been incredibly welcoming. The Minister of AI did a video about this and published it online.
Obviously, perhaps I'm a romantic, but the UK is the birthplace of computing—Alan Turing, what happened at Bletchley Park, that's really what created modern computers. You could potentially argue it is the birthplace of modern AI, because what the guys at DeepMind had done is this idea of training models and doing RL on top of that. This is what all of AI is today, with what they've done with AlphaGo and AlphaZero and all of that. There's a talent base here that's quite attractive as well. I was on the BBC, by the way. It's so awesome to be with you guys because I've been on the BBC and CNBC, and it's just 30-second sound bites, they kind of interrupt you, and they're trying to trap you. I said something like the talent here is more cost-effective, and the BBC guy was like, "All right, do you mean like it's cheap?" I was like—
Can't say anything, yeah.
I just mean Silicon Valley is getting so absurd in terms of—
Did they give you coconut water or not?
It's awesome to be here. It's so cool to open an office. Everything is historic here, even where we are in King's Cross. That was a staple at some point and was some old-school horse technology sort of place. It's cool to be in a place where there's history and there's tradition. There's also incredible diversity. I'm really enjoying being around people who are artists and from all walks of life. I think Silicon Valley sometimes can get a little bit tiresome where it's all the same thing; it's this monoculture.
Yeah, just tech dominated.
Amazing. We're happy to have you here in London as well. We'd love to ask you some questions about the early days, not of Replit specifically, but of the internet cafes in Jordan. As we said, you grew up with that being your onboarding ramp into the world of the internet and software. What was it like in those early days? Did you ever think when you were in those internet cafes that you would end up building Replit? How did it go from an internet cafe in Jordan to Replit?
I'm a big fan of Steve Jobs's commencement speech at Stanford where he talks about connecting things backwards. He talks about taking a calligraphy class, and that led to creating fonts on the Mac. I think my life story is similar in that there are certain things that happen, certain happenstance that led up to the creation of Replit. I didn't have internet at home, so I would go to the internet cafe, look up coding projects, look up different code snippets, learn a little bit of programming, go back home, memorize what I need to type, go into Visual Basic, type it, and then I'd run into an error. Then I'd go back to the internet and be like, "How do you deal with that?" One of the things that I noticed about the internet cafe is that it's a computer store with computers everywhere, but they weren't run by software. There wasn't software that was managing the store. They had a bunch of people working there that were looking at the time you logged on and then looking at the wall clock, and just tapping you on the shoulder like, "Your hour that you paid for is up, you need to either pay or leave." I thought, "I could build a software for them that allows them to log in, play Counter-Strike, or log into the internet." I thought maybe there is a way where we can secure these computers, because one of the problems they had is people would download viruses and all sorts of worms, and it was a pretty unsecure environment. The pitch I gave them was that I can create a client-server application where you have the server that has the accounting software, creates accounts, adds hours, we can even do things like gift cards, and we can show your monthly accounting and how much money you made. They were very interested in it, but I was 12 at the time and my programming skills were pretty raw. It took me two years to build this—two years sitting at home. I remember my friends in my neighborhood were like, "We didn't see you for two years. You literally changed." I would skip school or go home early from school and just program all the time. But I ended up selling the software at the time for the equivalent of a hundred pounds, and McDonald's had just opened in Amman. I took my entire class in 2001 to McDonald's. I was just absolutely balling, all the girls in the class taking us to eat Big Macs. It was a very edifying experience. Connecting it to Replit, a few things: one is just this idea of being nomadic—having a computer at home, having a computer at the internet cafe, having a computer at school. I wanted a place to program anywhere I went. It shouldn't be the case that I need to download all the software, set up the development environment, write the code, and then do the entire song and dance again, and every time you're missing DLL files and libraries and all of that crap. Replit was really the first in-browser coding experience that's totally portable, shareable, and all of that. The other big thing is security. Security is very key to Replit. What I had done at the internet cafe was create these sandbox environments where you can play a game or log onto the internet without infecting the machine or the network. With Replit, it was clear from the start that we're going to be providing computers on the internet. It was almost like an internet cafe online, because everyone's logging into Replit, we give them a virtual machine, and they could do whatever they want in it. They could download viruses, install worms, do whatever. We ended up innovating a lot on building sandboxes online, which turned out to be very important for agents as well.
Fast-forwarding to when you were starting Replit, you famously got rejected from YC four times. Can you tell us the story there? From your early days in Jordan to building Replit feels very textbook and an amazing journey, but what did YC not see those four times? We'd love to know.
There have been so many setbacks and so many rejections, and you just got to keep going. With YC, I grew up reading Paul Graham, as I'm sure you guys were as well. He's like the spiritual figure for startups. The fact that he's in the UK is amazing, by the way.
Not many people know that, so we're bringing him out to a community event on Thursday if you guys—
I only found out like nine—
He lives in the Cotswolds.
Yeah, so he rarely comes to London, and I assume he's coming.
You guys should pull him out. We should get him down to the studio.
Come on Thursday, I'll introduce you.
I was reading all these Paul Graham articles and really interested in YC and the philosophy behind YC. The really cool thing about YC is there's an intellectual underpinning for what it means to be a startup: why startups are good for the economy, why startups create wealth in the world, why this is a positive-sum game. There's an actual ethos that I really related to. Applying to YC with this idea that we're just going to have a website you're going to go to and you can build your dreams, you can publish them, you can monetize them, and it's a one-stop shop—I think it just seemed absurd at the time, right? And understandably so. If I'm an investor, I'd probably not invest at the time because it was so ambitious. You could pick any one of those things and that's in itself a startup. You could say, "Oh, we're going to work on sandboxes." That's a startup; there's 100 startups actually doing that. "We can work on deployments," that's something like Heroku at the time. "We can work on monetization." Every one of those things is part of the software development journey, but it is itself a humongous challenge. So I could understand why YC rejected us a few times. The other thing, just to be honest—and I'm a big fan of YC—there's a certain pedigree, there's certain pattern matching in Silicon Valley. I think Europeans as much as Middle Easterners are victims of that, where Silicon Valley worships the Harvard and Stanford dropout, right? There's something about this young hacker kid that got into an elite university, but he's exploding with entrepreneurial energy and wants to get in. This was the archetype. I didn't really fit the archetype, and it wasn't easy to get funding and get into YC. But I kept going, and we kept building. At the time, a lot of really hardcore tech people were interested in what we're doing because, again, we're building really hardcore software. I would blog about what we're building, and it would go on Hacker News. Hacker News loved me until they canceled me at some point, but I was a staple on Hacker News at some point. Paul Graham, at least he used to, reads Hacker News religiously.
He found one of the things I wrote about some of the sandboxing technologies we built. Recently, actually, we did a partnership with OpenAI, so I did a video with Sam, and Sam actually told me some of the backstory that I was missing. At the time, Paul was really upset that YC didn't have hardcore hackers; it was becoming MBAs. People were really motivated by building a startup and making money as opposed to solving hard technical problems. So he went to Sam and says, "You got to solve this. You got to go out and recruit the best hackers, the best technical minds. We got to change this." Sam was like, "Okay, that sounds good." At some point, Paul discovers Replit on Hacker News, sends it to Sam, and Sam was like, "I'm on it." He reaches out to me and says, "Come meet me," and he sends me an address in the Mission. I'm like, "Well, YC is down in the Peninsula in Redwood City. Why am I going to the Mission?" I go to the Mission, and it was the OpenAI office, and it was Neuralink on the other side. It was like the Elon fiefdom over there.
Because Elon was still part of OpenAI, I think.
I didn't think much of it, but the fact that I met Sam in 2017 and he was working at OpenAI, we connected very quickly on how much it sucks to be a programmer today and how it could be a lot better. Then he introduced me to Paul, and Paul and I started this email relationship for many months. At the end, just before the YC Winter 2018 batch started, like a couple days before, Paul was like, "I'm going on a trip. I'm not going to have access to email, but you should really do YC." Then he left and he stopped responding. I was like, "What?" So I forwarded this email to Sam. I'm like, "He told us to do YC. What does that mean?" He's like, "Yeah, we want you in this batch, but you have to do an application." I'm like, "Man, I have done this application four times. I don't want to do it again." You sit in front of the camera like it's a hostage video and you're like, "Oh, we are Replit, we do..." It's like the cringiest video, the YC application. I didn't want to do it, so I put a YouTube link in there. I show up the next day for the kickoff day, but they did the late interviews. We go in to do the interviews, and I just noticed that they were extremely mad at me. Michael Seibel, who was running YC at the time as CEO, when he shook my hand, he just squeezed it. The moment I sat down, he was like, "Why did you Rickroll us?" He was so mad that I put the Rickroll link in the YC application. Actually, if you go to our YC profile up until recently and you click on our application video, you would get Rickrolled.
So it's a perpetual Rickrolling for anyone who's looking at our YC profile.
We're doing that after this interview, we'll go and do it. That's so good.
That's so good. We can fast-forward a little bit and go further down. I have a few questions about your philosophy stance and leadership stance as well. I found out that you've got a pretty strong management rule that goes to the tune of something like when you see a committee, you kill it immediately. Why is that?
I don't think good decisions are made in groups. I think there are certain governance structures, like maybe in government and democracies and things like that. I don't think a company is a democracy. I think a company is actively the opposite of a democracy. It needs to be somewhat of a monarchy, and it needs to be a recursive monarchy where in every team, there's someone that's running it and is really making decisions, down to the IC. We have a very strong DRI culture. On some project, the monarch might be an engineer just out of school, so it's not about status; it's about domain expertise. People closest to the problems need to be able to make the decisions, and a committee is exactly the opposite of that. When committees emerge, that's when you know the culture is going downhill and you're becoming very bureaucratic and un-startup-like.
Yeah, I remember seeing—I think it was a meme or something—there was a leak from one of the US government pamphlets in one of the wars where they would drop these pamphlets on how to set up committees in enemy territory and these sting operations. It was like, "If you have a decision to make, bring as many people as you possibly can into that decision. Maybe even set up a committee in the future to make those decisions internally."
I remember the document. It was a subversion document: how to subvert the enemy through committees.
There's a lesson to be learned there.
In building Replit, so there you go. I'd love to understand—we did some research and found that you don't really like the term "vibe coding." I'd love to understand your insight on that. Replit is kind of credited with inventing, I guess, this—
kind of word. I'd love to understand, why does that kind of label grate you a little bit? What has vibe coding changed into that you didn't—
It sounds a little frivolous, okay? And the kind of things that people build with Replit are very good. We're getting into government work right now, and there are people in government making software with Replit that is mission-critical for their work. I talked to someone recently whose entire job was to read the news all day and then generate a memo, and then give it to their bosses. Instead, they're like, "This sucks. I'm not going to spend 12 hours a day reading news." And they built an AI agent that crawls the news according to a certain rubric, summarizes it, and creates an active dashboard that is live. It went viral within the government, and it really changed how they work. He told me something I will never forget. He said, "Agentic tools are as impactful to the government, to state capacity, as the internet was."
And so it could be enormously impactful on how civilization works, right? I mean, ultimately what we're doing is decentralizing the means of creation. It goes from a small group of people—which 0.5% of the world population can actually code, 0.5%, that's really low—to anyone can make software. That's profound. That's just going to have reverberations forever. I make an analogy to what happened in Europe a few hundred years ago with the Gutenberg press and with the Protestant Reformation, right? Part of the reason the Dark Ages were called the Dark Ages is because literacy was banned, right? You couldn't read and write. What Martin Luther did was like, "No, you should be able to read. You should be able to read the Bible and have a relationship with God directly."
There was a technology revolution that enabled that, which was the printing press. Some people hated that, namely the priesthood, right? They were monetizing this asset that's literacy. We have a kind of a priesthood today. There's a lot of gatekeepers. There's a lot of people that don't want people to have the power to be able to make their software, whether it's big corporations—Apple has not been very friendly to vibe coding—or it is IT departments in big companies where they don't want people to make their own things. But I think it is a profound shift in how we do business, how we operate, and how we use computers. And vibe coding just sounds a little frivolous.
Okay. Obviously, Replit has done an incredible job of allowing people to be able to build incredible apps, but I want to understand what's the next thing forward in terms of... Do you feel like Replit's kind of building the last generation of the app? Will it be more, you're less of an app builder, it's more of an agent orchestration platform? What's the future looking like for Replit in that respect?
Yeah, I think the thing that we spend most of our time on is not necessarily the AI itself; it is the control plane for agents.
Having agents operate in secure environments in important contexts such as governments, enterprises, things like that, you need them to be verifiable. You need every action to be reversible. You need everything to have an audit log. I look at the OpenAI-Hugging Face incident, and the lens on it right now is that AI safety lens, right? Which is fine, but actually the lens that's missing is a cybersecurity lens. Like, why weren't there any detection and monitoring happening? Why was the sandbox so weak? Why were the actions that the AI was making... There was also a lack of integrity where the AI actually could change the logs, and the AI attempted that. So I think that's a big challenge in the future. We have to come to terms with the fact that agents will go rogue,
no matter what. And so we have to create environments to make sure that we are containing that, we're getting the best out of agents, and we are capping the downside. And so all the technology that we built, I think that the next set of products that we're going to come out with is exactly what you're talking about: an agentic control plane, an orchestration layer. And I think that's the ultimate destination of software, because software, a lot of it is: you have a problem, you build an app, use the app to solve that problem, right? You can remove that middle part: you have a problem, you use an agent to solve that problem. It's weird for me to say it because we're known as an app-building company. I think people will spend less time on apps.
Yeah, it's a natural evolution, I feel. You mentioned the three-step kind of process: have a problem, build an app, use the app to solve the problem. And then the agent kind of skips—the app is removed in that situation, right? There is no...
I mean, you look at Instinct or Muse; it's happening in consumer now.
Yeah, exactly. And that's just getting incredibly powerful, and you can do a bunch of things with everything that's coming out. I want to ask about internally at Replit. You guys kind of built this semantic layer of this agent orchestration thing that you were talking about kind of moving into the public as well now and building it for Replit as the product offering. Can you talk to us a little bit about that? What was that semantic level like? How powerful is that? It sounds insane.
