Welcome everyone to a special edition of The Information TV. My name is Akash Basria. It is Thursday, September 24th. Today on the show, we are on the ground at The Information's AI Agenda Live conference. I'm bringing on the CEO of Replit. We're talking about safety and cyber security. We've got the CRO of Nebus coming on to talk about dynamic pricing in the AI environment. Got a venture capitalist coming on to talk about the future of AI, small models and valuations. We also got a former OpenAI researcher who is going it on his own, launching his own new lab. It is a busy show and a fun one. So without further ado, let's get right on into it.
I'm here with Amjad Masad, the co-founder and CEO of Replit. Amjad, welcome to the show. It's great to have you here.
Thank you. Yeah, it's exciting.
So, there's always so much news to talk about, so we might as well talk about the models out this week. We've got new models actually. Well, today there was new Gemini models, but I think the Claude and OpenAI models are really what people are talking about. You know, I wonder what this week's model releases tell you about the direction that models are going and what trends you're seeing there.
I mean the main observation is that we're not stopping anytime soon. All the benchmarks are coming up. They're coming up with new benchmarks and we're quickly kind of saturating those. You know, ARC-AGI which is supposed to be the AGI benchmark is saturated I think now like 80 plus or something like that. And so I think this scaling moment is a lot smoother. You know, with pre-training, we had GPT-2 to 3 to 3.5 and 4 and it sort of stopped there, right? And then there was a lull for a while and then we entered the reasoning era. But it wasn't as smooth — like every three months we'd get a big jump, like the Opus release in November last year. Now it's continuous. Now every two or three weeks we get a new model reliably better than the old model.
And they're cheaper. I mean these ones this week, they're cheaper.
Who knows if they're actually cheaper or the open source pressure on the labs is making them cheaper.
And so you think maybe this is not about token efficiency. This is actually a conscious decision from the labs.
Oh no, per token price is going down — they slashed per token price, right. So token efficiency would be same token price, which is actually getting better. But the model is emitting less tokens per task. Both of these things are happening. So overall cost is sort of going down, which is positive, but people are using a lot more tokens. So no one's noticing that.
I want to ask you — we had the CEO of Clay on the show a couple days ago. One of the things that he said was, you know, he does scenario planning as a CEO about, okay, we have this debate — are we going to reach RSI? Is RSI a spectrum? What does my company look like in a post-RSI world? What are the different versions of Replit that you have scenario planned along this axis of when we're going to reach RSI? What happens if we do get there? Help me understand that.
Yeah, one of the things we sort of embraced early on is that Replit, you know, Agent — September 2024 was sort of the starting gun for coding agents. Really, there was no coding agent on the market you could try. Vibe coding wasn't coined. Replit was the first kind of fully natural language interface. Pretty soon after that, we got a bunch of copycats, and initially you're sort of competing on all these details. What we've realized is our value proposition has moved from code generation, which ultimately is a commodity — it's commodifying really, really quickly actually. On stage I said eventually your fridge will make you a website, really anything can make you a website because everything is a coding agent these days — your Muse agent is coding, your Instinct agents, your Claw, everything has coding capabilities. It became a fundamental infrastructure technology as opposed to a product. It's a given.
So where Replit is sort of differentiating — it's very topical given what's happening with the Hugging Face incident, with all the different incidents — is the infrastructure around the harness. You know, I'm kind of coining this name, outer harness perhaps. But you know, we had the harness sort of sitting on top of the model, but the sandbox around it, the database and memory and context around it, the identity, the authorization, the network security, the deployment and managing of code in production, deployment of agents — all of this stuff Replit has figured out because we had to. There was nothing on the market. Meaning how to be secure, how to manage agents at scale, how to manage memory and context at scale, how to do agent authorization identity, how to do MCP and connections securely — and all of that are enterprise features that become more and more crucial over time. And so when we go to enterprises now, the question has become a little bit less about AI adoption. Obviously, there are some tardy enterprises, but for the most part it's like, help us manage the chaos. Code is everywhere.
What I hear you saying is basically, RSI or not, the playbook will be the same for you, which is prioritizing the safety and security that you already have — safety, security, management, discovery, right? There's all these things. So, take for granted that coding agents are everywhere. Everything is generating code. What's managing that code? What's deploying it? What's giving you leverage over cost? So speaking of safety then, what is your view on AI safety regulation? Should the government have a role in regulating AI safety? Should it be an independent body?
It's not obvious to me right now that the discussion needs to be about safety. The discussion needs to be about cyber security. So for example, if you look at four of the five hacks — the OpenAI, Hugging Face hack on the side, you look at the Meta, Anthropic, Gemini hacks — I think three out of four, they all kind of use the same sandbox contractor called Invisible, right? And there was a misconfiguration in the sandbox that caused these attacks to happen. No one's talking about that. It is really rarely covered. There's like some actually pretty cool online new magazine called Effort that kind of covered some of it. But the reality is there's a cyber security discussion that is not getting enough airtime. And so we've always had software, we've had software worms, we've had viruses that sort of escaped containment and did all sorts of destructive things. This is obviously that on steroids, but the way we've handled the previous generation of malware is by betting on cyber security, by innovating on cyber security.
But so should the government have a role in this cyber security equation?
I think they should enforce existing laws, which is if your agent's committing crime, someone should be liable, right? And what about additional policy around pacing the frontier or agreements like that? I think the labs need to slow down to harden their systems, because — and we're happy to help them by the way, they haven't called us yet — but their systems are clearly immature, right? Like if you read the logs from these incidents, like there was no monitoring, there was no kind of safety red button, like, you know, shut down everything. And so I think they need to pace themselves. And should the government do it for them? It's almost like they're asking, hey, make the decision for us. But I think the responsible thing for them to do is to pace and slow down because otherwise they're going to be criminally liable for hacking core internet infrastructure — like they hacked Ruby gems, they hacked Hugging Face, they hacked PyPI. I mean that's really unacceptable.
Well, so then let me ask you this. Is it practical to slow down? Do you think it will actually happen?
I think yes. I think again, if the government is enforcing existing laws — like you shouldn't be able to hack Hugging Face and Ruby gems and just be okay with that — I think there needs to be something there, and that will force some kind of slow down. I saw something — maybe you guys reported on it — but I think there's a deal between OpenAI and Anthropic to like test each other's models, some kind of rival adversarial threat team, which is a really cool idea. So I think there needs to be some kind of configuration where there's some accountability. It seems like it's premature to ask the government to do that, and we run the risk of regulatory capture obviously.
Right. Let me ask you one more question before you go. You recently opened an office in London, spent some time in Europe. How are they approaching this exact conversation differently in Europe? How is the conversation about AI different broadly speaking over there?
I think they're eating the bait much more than our administration here, right? Like I think recently both in Europe and the UK, they banned superintelligence. What does that mean? By the way, superintelligence — you can't define it anywhere. You can't put it on paper. Yeah. I mean, by the way, according to our president now, AI is SI, right? So already we already achieved superintelligence in the US. But you know, I think the politicians there are taking debate from the big labs much more readily than the politicians here. And I think part of the positive things about the current administration is that we have a lot of tech people there that understand the technology, that understand the incentives, have been in VC, understand the incentive of regulatory capture, that sort of thing. But on the other hand, what I've seen on the ground — like I met with Mayor Sadiq Khan, I met with the prime minister's office in the UK — I see also tremendous optimism. Like in New York City they actually banned AI in schools, in London they're actually very open. We're actually collaborating with them on some training and upskilling programs. They're open to implementing AI in the core foundational aspects of their society.
So that resonates with you more than the way the US is approaching.
Yeah. I mean Mayor Khan, he called himself a realist optimist, right? And I think that's the right approach — that there are obviously tremendous risks involved.
Sounds like a venture capitalist.
Yeah, but I think there are a lot of venture capitalists that are purely acceleration optimists, which don't recognize any safety concerns. I think like most things, there's a kind of a nuanced middle position that doesn't get you amazing headlines, but it's sort of the reasonable common sense position.
Great. Well, Amjad, I want to thank you for joining us. That is Amjad Masad, the co-founder and CEO of Replit here on TITV.
I'm here with Jerry Twerk, the co-founder and CEO of Core Automation. Jerry, welcome to TITV. It's great to have you here.
Happy to be here. Thank you very much.
You left OpenAI in January. You've been on your own now for — what are we? We're almost at nine months. How's it going?
It's definitely a journey.
Tell me about what is Core Automation. What you've been working on?
In the world of AI, everything moves so quickly. It's almost like blinding lights in your eyes. But I can tell you it's a very special journey in this day and age to be able to form a company in the way that you want to, to take on research bets that you think are the right ones to take, and that's generally what we are trying to do. I think AI is in a place which will change our world. It will change our lives, and in some ways each one of us trying to build it, we can shape it and shape that future a little bit. What with Core Automation we are trying to do — we are focusing on agents and on research, we are trying to do fundamental deep learning research.
Building your own models?
Agents, building our own models from scratch to build better agents. Ideally agents that learn at test time, agents that learn on our own data, and we're trying to use agents as much as we can in doing that research.
So you've been playing around with Meta's Muse I take it quite a bit, and Instinct?
Actually not at all, but we are mostly using OpenAI Codex. We are programmers, we are researchers, so we are using coding models. But the consumer products is not yet something I've been doing much.
Why couldn't you build the models that you want to build within OpenAI?
I think the big labs today — they are very successful companies and they already have an existing structure of how they train models and what they do. They bet very big on transformers. They scale pre-training of transformers and reinforcement learning of transformers, which even reinforcement learning is a relatively new addition to this stack. But I do think and I do believe that there are different ways to train models. We can change that recipe. And in some way the bigger the company and the more successful it is, the harder it is to change that stack and the harder it is to do big sweeping changes. There are multiple teams that need to coordinate and there are multiple ways to do it. I think just from a practical perspective, it is much easier to do it in a small company that moves quickly. We try to redo all the stack and try to train them all in a very different way than it would be in a much larger organization.
So given that you are so close to the research, I want to really get into some of these big questions around AI research that everyone is talking about right now. We'll get to the safety conversation in a minute here, but let's go back to the Navier-Stokes debate, I guess, when it happened — the model solved, I guess, this math problem that couldn't have been solved for many decades. There was a whole discussion about whether OpenAI leveraged work that other people had done using the model or not. What did that whole conversation tell you about the current state of AI research, and what did you reflect on it?
Well, I am already pretty tapped in very closely to the state of AI research. So, I don't think I was surprised in any meaningful way. The people who worked on those solutions, on those proofs, were my former very recent co-workers. So, I know them well and I know what the models are capable of. I think the announcement and the way how the result was shared with the world was a little bit tricky, and I think it was on both sides. I think it could have been done better in many ways.
What do you wish both sides had done differently?
I think there have been some notion of disagreement on both sides who did what — did OpenAI use other people's input or not — which, you know, I don't know precisely, but I think it's very unlikely that they did. But I think we are entering very clearly a world where the models are doing more and more stuff that we thought machines are not capable of. Maths is becoming more and more used in frontier AI — AI is being used more and more in frontier mathematical research. And what we can see in the story is that even the mathematicians that were trying to solve this problem — what they were doing, they were talking with GPT and Claude about their research, they weren't doing it completely separate from the AI. The question is kind of like, who uses the AI? How do we credit any future discovery? Is the prompt the thing that leads to the discovery, or is the reasoning chain and the model and the training of that model that actually leads to the discovery?
But I guess the broader question that I have is — I mean, this happens often, right? Anytime there is any kind of an achievement for the model, it feels like we have some flavor of this conversation over and over again, which is, well, what did the model use in the background to get to this achievement? And so playing this forward, the models are going to continue to get better, they're going to continue to be able to do different things. I mean, it would be kind of frustrating if we have to have this exact same conversation for the next five years saying, well, you know, we don't really know if the model leveraged somebody's work. So I guess the broader question I have for you is, is this the conversation that's going to keep happening for four or five years, or is there a solution to this that you think can be implemented to kind of make everyone feel a little more comfortable about this?
It's a great question. What is the model trained on? The model is trained on all of our data — all the tokens, all of the texts we have written, we have published, is written on all the scientific works of human researchers. The models are not trained on some data from the sky. They are trained on all collected human intelligence and thoughts that we have aggregated on the internet. And in some way, I think for the positive future, we just need to get a little bit culturally used to the fact that the models will be doing research. The models are trained on our own data in some way. What the models are doing — they are learning from our work and our own thoughts, and they are just applying them in some way. A model is just reusing the thinking process of all the mathematicians that ever contributed anything on the internet, and now we can scale it up. We can have the same thing kind of available in large quantities whenever we need to solve a very hard problem in a technological sense. This is a very useful and a very powerful thing. And of course there would be people who argue and say, yes, this is the technology we've created, but it's not necessarily beneficial to people who like to protect their work. You could say — there's always people who will make that argument.
Yes, like, I think one thing we should probably just from an empathy perspective — give people guarantees, whoever doesn't want the model to be trained on their data, to not train on their data obviously, and not use that. Why would we want to do that? At the same time, very clearly just from a perspective of civilizational progress, we should at some moment accept that having models that can automate a lot of the things that we've been doing before is actually a good thing, and maybe letting go a little bit of this attachment to how things were done in the past and embracing that AI will be doing more and more things for us in the future is actually a good thing, and it frees us up.
What about safety? How should we regulate AI? How do we ensure it's safe? Who is the right body to regulate safety in AI?
Great question. I don't think there exists a right body right now today. The good thing about safety — and I think we already see it — no one in the world wants to destroy the world, plainly speaking. Obviously everyone developing AI and the researchers doing it are getting worried because they want the world to go well.
And we should make clear — are you concerned? Are you—
I am, I'm concerned somewhat. I'm generally a technological optimist. I call myself a cautious optimist in a way, as in we cannot just YOLO what we are doing. We have to make sure that every step we are taking is a measured step and we are aware of the consequences, and those consequences are not something we regret at any moment. At the same time, my take is that the deployment of AI until today — it's been extremely successful and extremely positive for the world. There always are certain externalities, but they were much smaller than the benefits they bring. I do think that this technological revolution will be different than any other before — that it will move quicker and change the world deeper. In some ways, whenever technology changed in the past, the world did change, the lives of people did change, but it happened very gradually and very slowly.
I want to go back to — so you said the perfect group does not exist right now insofar as who should regulate the safety component of AI. So if they don't exist now, how would you design this group? What's your idea of a completely impartial group? And does the government have a role in that?
Here are my personal take — which is I don't really believe that much in having independent oversight groups. Those oversight groups are never independent and they are never actually doing the things efficiently in the way we want. What I mostly believe is kind of like, can we achieve the market in a place where the participants watch over each other and police each other, which I think—
I think one thing we have seen is that Anthropic and OpenAI are agreeing to check each other's models. And in some way, if you have multiple model providers, it is for example beneficial for provider A to say, hey, provider B has an unsafe model, don't use it, because then they'll grow their market share. So in some way, testing each other's models and checking what could be the unintended consequences can be done fully by positive market forces. What I would like — two things to happen. One thing, I think we should have multiple labs in the world — the centralization of labs is not very good. The more we have, generally the better. Ideally I would like the world where we also would not concentrate compute too much, while we try to achieve the world where those multiple labs have roughly spread compute equally in some way.
Do you think we should go as far as putting a cap on how much compute?
This is a good question. It doesn't feel very much free market, but I think this kind of can protect us from some kind of runaway scenario where one company gets AI that is good enough that it can outthink and outsmart everyone else, which I don't think anyone wants. In that case, making sure that we encourage various market participants to be checking each other and verifying safety of their products and their deployments is a good thing. And suddenly what we want to achieve is at any moment in time we need to make sure that what we can call the good guys — but people who care about humanity flourishing and not their own self-interest in many ways — have more compute than the rest. That's what we need to do. And I think we need to believe that companies are a lot of people, company leadership changes, people who are making decisions change. We cannot ever bet — even if we think this person is good or that person is good — those people will never be always in charge. We already solved a lot of those problems with choosing how we run our countries and how democracy is run. So I think the plurality of players and spreading compute—
Which is where you come in. I mean this is—
I'm trying to do my part.
So let's go to Core Automation then. I mean look, these labs have so much capital. I mean, you were at OpenAI for a long time and I don't need to share with you how expensive it is to do the work that you're trying to do. So what makes you think that you can take on your former employer, which has — they've raised hundreds of billions of dollars in capital?
The story is, part of it works. For example, for Anthropic as a company — which you could ask, and I was asking myself for many years — how could they compete with OpenAI that already has this healthy revenue stream and already has those hundreds of billions of dollars of capital? And the reality is you can, if you grow quickly enough and get on the flywheel of deploying good models, building your own revenue stream, and then growing your compute. I think the world is catching up on this idea that it's better to have more AI labs rather than less.
But I mean, tell us about your own progress. Where are you at with your model development?
Yeah. So the whole bet of what I am trying to do — which is you cannot out-compute bigger players, you cannot out-data other players. The only thing you can do, which is the equation of building models — is compute plus data plus algorithms. So what we can do is do better algorithms than others. And if there is any chance that we can, for example, get 10 times more compute efficiency than others — if you can get a model that has some unique properties and has some unique selling points that no one else has — doing original research, then we have a chance to have a place in the market and start building our own compute base and start building the next lab out there. So that's kind of like — we are a little bit of a group of — you can think of us as rebels.
How many people have you gotten together for your company so far?
We are around 23 people at this moment. And the good thing is that the agents help you move pretty quickly in the current world, in the current day and age, which is really great. So my assumption — like working with researchers for many years — that a group of 20 people can do roughly what 200 people could do a year ago, right? That's my take on the speed of research and on the progress that we are doing. And yeah, we are betting on some very non-standard techniques and very untypical ideas.
Not yet time to fully disclose it and not a time to talk about it in detail, but through those we think are at least some that can provide significant efficiency and capability wins over what much better capitalized perhaps can do. I think that's kind of what the role of the new labs is. We have incumbents that are very successful and have established techniques that they use for training models. There needs to be someone who is exploring alternatives, and hopefully one of us — maybe us — will find something that actually works.
You know, you think so deeply about what agents can do and about research. Which brings us to the conversation around RSI, which is something that you likely spend a lot of time thinking about. What do you think people get wrong about the conversation about RSI right now?
I think the most thing people get wrong is that RSI is a spectrum and it's not something binary that you have or not.
You think it is a spectrum?
Okay. So we will never really know.
I think I'm one of the people in the market that maybe thought of it longer. There are probably definitely some that have been working on it as long as or longer than I do, but there are very, very few people in the world that did. So, I did very first work at OpenAI on coding models. I did work on coding and applying coding to doing AI research for the majority of my career. I did start the project of building AI scientists internally, and now I'm trying to bring it even further — with trying to think how do we build a whole company about the concept of the AI scientist. And in some ways, what is RSI? RSI is applying AI to developing better AI, and in some ways we've been doing that for years already. It's just the fact is that through the recent developments in coding agents, it's going now faster than before, faster than a year ago, because even a year ago we were applying — when you are in a lab you have AI and you use it for whatever you can. AI in many ways is an automation technology and you automate as much work as you can. But the story is the better AI gets, the more you can use. We don't have what I consider like fully autonomous or strong RSI, where the models can just improve themselves without human supervision. That's just not happening. And honestly, the model's research taste and research ideas are not very good yet in any way.
But so do you think it will get to — I mean I hear you, it's a spectrum. So getting to RSI is maybe the wrong question to ask, but do you have confidence that the labs will get to what they think to be RSI here?
Sure. Surely it will happen. The question is when. I don't think there's anything that we can say AI will never do this. AI will do everything eventually. The question is is it this year, is it next year, is it in five years? Those are very practical questions. And in reality, what I think is that more things are algorithmic bottlenecks than people realize. And with one or two good new algorithmic ideas, you can probably get there pretty quickly, which is in some ways the scary realization.
That like there can be someone in the world, somewhere in a lab, just having a good idea and running one good experiment, and that good experiment could suddenly create a very, very smart system that we didn't foresee before.
We don't know. In many ways, scaling laws are nice because they give you a predictable way of improving capabilities. But what me and my team did at OpenAI with introducing reinforcement learning and scaling in reinforcement learning — it brought completely new abilities that no scaling law would be able to predict for you and no scaling law could factor in. If there is any next algorithmic improvement, if there is any new step change in how we do it, then the further you get, the scarier those get, because those capabilities are something that we don't really know how to protect and control.
I want to end with something that you wrote. You said the road to singularity will be paved with BlackBerries of our generation. And I presume you're talking about the BlackBerry, the phone.
What are the BlackBerries of this moment right now that we are seeing?
I don't think I know, and I don't think I can call those out in any way.
But you know, take a guess. I mean, here's one — like, so OpenClaw, you know, let's look, it took Silicon Valley by storm. It inspired many of these agents right now that people are getting very excited about. But people are not as excited right now about using OpenClaw.
And so I would go out on a limb and say I think OpenClaw itself is a BlackBerry of our generation. So you know, I mean think about that. What are the tools right now that you think we might not be using? Maybe the use cases, the agents — what are the BlackBerries?
Like, in some way, very likely the majority of the things we are doing today and using today will eventually get sunset. I think we are still pretty early in the deployment of AI. In every wave there are early winners and early stories of success, but the longer-term rollout and the longer-term success is something very, very different. So it's very hard for me to say, but I was generally trying to imply that there are a lot of things that are good at the moment and are valuable at the moment. But at the same time, the more time passes, the more longer-term economic trends and longer-term efficiency matters, and the competition becomes more about fundamentals rather than, for example, about who was first or who did something. It is very easy to do something new right now, something that no one did before, and that is really, really great. And you can get — if you are the first person in a new market, you usually get a lot of it. But then others show up, sometimes quicker, sometimes later. And we are getting more and more capable players in this market. And where things will end up, I think it may look very different than where it is today.
Great. Well, Jerry, I want to thank you for joining us. That is Jerry Twerk from Core Automation here on TITV.
I'm here with Gokul Rajaram, the founding partner of Marathon Management Partners. Gokul, welcome to the show. It's great to have you here.
Thanks. Great to be here.
So you do all sorts of great writing and consumer is one area that I know you've been close to throughout your career. And we have this agent out, Muse, which is taking the airwaves by storm right now. Everyone is very fascinated by it. It seems to be impressing a lot of people. What have you made of the rollout of Muse, and do you think agents are ready for prime time? How are you thinking about this?
Look, I think Muse is an extremely well-designed product. The people who built it are friends of mine and I think they've done an excellent job of going behind and exploring some of the dogma and making it a truly useful product. I use it a lot myself. I also use Instinct.
I use it as a personal assistant essentially — to remind me of things, to essentially make sure my travel, my restaurant reservations, and my chores and errands are all on track. But I don't give it access to my email yet. And so I think the number one thing — I don't know, I would not give my assistant access to my email. So I think of it as a personal assistant. I give an assistant access to my calendar but not to my email.
This must really frustrate your assistant, I have to tell you.
You know, I think email has personal stuff in it. And I think the number one thing that assistants need to transcend to really be used by — and even more useful — used by everyone, is one, trust. Which is people need to trust that it'll do the right thing with your email. Email will not — there was an ex post yesterday with Instinct where it accidentally leaked apparently someone else's messages into this person's chat with the agent.
And so that kind of stuff you just can't have happen. So I think trust is the number one thing — the psychology of human beings and trust. The second thing is, I think, bringing it to the workplace. Because half of our job — I use it for personal stuff, but how do I get it connected to my work email, to my work calendar? I have an IT team that runs this. They don't want agents in there. It was super hard to get almost any external services — hard to get AI agents like Claude and ChatGPT to connect to my work account. Now getting this set of agents connected to work account is another barrier. And in particular in regulated industries — in finance, legal, healthcare — these agents getting access to data that might be accidentally leaked, or just comfort with that — I think that's the barrier to entry, really. Trust.
So if trust is the issue, then on one hand we have a conversation about the models approaching RSI and the models being dangerously good, and then we have these incidents where they just seem like mistakes that any model should be able to avoid. So what's the solution here to generating trust? Is it that the models still are not good enough? Because we have people saying they're dangerously good. What's going on?
I think it's — the fundamental root cause is the lack of guardrails. And I think models assume trust versus earning it. And I think a good example is — I think on the medical scribe section, we've never heard — I think there have been issues, but there are companies like Abridge and others who are now used by doctors in very sensitive cases, in sensitive situations, to basically scribe the voice memos. They all started by first getting the doctors to approve every single thing they transcribed. So 90% of things are approved, slowly they moved to only having 20, 10% of things being approved. So you assume that it's untrustworthy and you slowly get every single thing reviewed by humans, but you move to a situation where very, very few things are reviewed by humans. I think right now Muse, Instinct, all of them need to earn my trust.
