The fundamental idea here is to build AI that improves itself. And I think we're getting there.
For the first time, we could swap the optimizer as the optimizee. And this leads to a reinvestment of cognitive improvement into itself.
You can think of RSI at many different levels of the stack: prompts, harness, weights, and you can go all the way to basically chips. Our goal here is to optimize all the levels of the stack. Our approach to RSI is that we have a multi-agent system working together that's been recursively improved.
I'm so impressed by your belief that, like, this idea of a model improving itself...
It is a dramatic statement, which is, I think, the first time I've heard you say it.
Because we have been so thorough, we have not released things until we're super sure it's correct.
Do you envision a system where the amount of energy that goes into it somehow decreases over time relative to the improvement?
We can't share too much, but one of the very important things that we've tried to do...
Welcome, both of you, for joining. We're going to be talking about everybody's favorite topic, which is RSI, recursive self-improvement. Maybe to start, it'd be great to just introduce yourselves and your backgrounds. Then we'll talk a little bit about what you're working on now, and then we'll dig into the content.
Yeah, I think of myself as, like, a physicist of some kind. I studied that. Then I did the PhD in reinforcement learning at the MIT Media Lab. I really like building things, and I'm very excited to be building a cool startup with Owen and Grace.
My name is Owen. I was first studying math, and then one day I realized that all mathematicians are proving theorems. And I was taking a class called formal language and logic, and they were showing how you can build a theorem prover. And I was like, oh, that would be cool, I can just build a machine that proves theorems for me. That's how I got into AI.
And then for my PhD, I worked on continual learning and meta-learning, then worked for a few years at DeepMind as a research scientist. Now I'm working on RSI.
I see a lot of pitches come in, and a lot of them use the term RSI. And I'll be like, 'Oh, so what do you do?' And they'll say, 'We use AI to create AI.' That sounds very interesting. So what do you do? 'We use AI to create GPU kernels.' Which, that doesn't, to me, kind of match the definition of what I would consider to be RSI. So maybe can we start by you defining what you think, or what you would include in RSI?
I think RSI is definitely an overloaded term today. Many people, when they say RSI, they mean different things. So I think the important thing when you are differentiating them is to ask the question: what is the optimizer and what is the optimizee? Like, when you're self-improving, who is improving? Right. So there are people who are saying, okay, we're doing RSI, we have humans using AI to develop better AI. But that, to us, is just autocatalysis, in the sense that the improver is still human with tools, even though these tools are getting better and better. But the idea and the improvement come from human brains.
Right. And just, I don't mean to interrupt, but to be clear, most technical epochs have had autocatalytic effects, right? Like, there's no way you can create a computer chip without a computer, right? You could argue that the internet development was sped up by the fact that people that weren't geographically located could coordinate, right? I mean, the steam engine, you could build better steam engines because you could get coal faster, etc. So this notion of, like, using the thing to build the thing faster is not new. And so you're saying something different.
Yes. I think for us, in the history right over civilization, we have this autocatalysis, you know, computers using computers to build better computers. But the idea for this change, for this improvement, comes from the human brain. But that human brain was never the improve—never the optimizer, right? Whereas now, for the first time, we could swap the optimizer as the optimizee. So the brain of the agents, which is the LLM, the model, can be the thing that you're optimizing. And this leads to a compound—this reinvestment of cognitive improvement into itself.
But there has to be a reason that you decided to take on this hard problem as opposed to what anybody else would do, which is, like, chew off pieces of the problem.
Yeah. I mean, I guess to answer—before I answer your more important question here. The reason I did this is I think it's a hard problem, and it's fun working on the hardest problem, and I think...
But do you have an intuition that this is the actual right answer as opposed to, like, something that's more pragmatic?
Yeah, I think a pragmatic thing would be just from a control theory perspective. If you have a gap in the system, you can't have a closed loop, and autocatalysis has no closed loop, right? So whatever you improve, there's still, like, a gap. Humans have to come in. You have to pour in other resources to make it work. But if you have a closed loop, you still have to pour in resources, but the marginal gains ideally get either constant or larger over time. And that's the fundamental difference. What's really different here can be explained pretty easily. So if you think of capability...
What we're trying to do here is we're trying to improve the rate of improvement of capability, right? So, like, the first derivative of capability with respect to time is the thing that everybody is trying to improve.
And now you can put that to be equal—driven by two things: the numerator and the denominator. The numerator would be performance.
Yeah. Which is what everybody is trying to improve.
And then the denominator is some kind of cost. Let's say capital, right? Or difficulty. And this one is pretty—pretty set, because the higher the capability, the higher the difficulty of the next step. Right. Now the numerator is the interesting part. For things like steam engines and whatnot, that was fixed.
Right. So then you end up with a very simple differential equation where you basically plateau.
Right. Now, if you have a system where the thing being improved is also the thing that's doing the improvement, you now have a new way to think about it. And the simplest case would be that the numerator is also correlated with capability, which makes sense, right? Because the better you are, the better you are at also improving. And if you do that, the math cancels out, and you basically get constant improvement over time. That's—that could be one level of RSI, right? And then there's one more case, which is now that the numerator could be correlated with a higher power of capability.
And then you get exponential takeoff.
Yeah. Can we dig in a little bit more between RSIs having closed the loop and actual convergence properties? Like, for some reason, when people see a graph where there's, like, a circle with an arrow to another circle and it goes back, and they look at that closed control system, then they say takeoff... [laughter] Right. Which, I mean, if this is a nonlinear control system, it's very difficult to know what the hell happens. By the way, most people don't seem to understand this distinction, because we very quickly go to, like, takeoff and foom and all that stuff when we talk about these things. Well, as a network guy, you know that it depends on the exponent of, you know, your function that's self-referencing.
And it's—what we found is that it's almost not useful to speculate on what we think could happen, because this is really an empirical problem. And the real way to know what's happening is to test things. And testing things is really, really hard. Like, reward hacking is one of, you know, many problems that we have to deal with. But the question you're asking is very empirical, and for some...
Interesting. So, but, so I'm a networker, that's how I would have answered. I would have thought math people would have been like, 'We've got this, we've got this differential equation, and I took out my pencil and I wrote down the answer.'
I mean, it is possible to write down the differential equation, but you still have to empirically measure the variable, the exponent.
So there's no analytical solution.
I mean, honestly, we don't even have a learning theory of deep learning. We don't know, like, fundamentally why it works, right? And then now you're, like, you know, five, six layers on top of deep learning, and so we don't have a good theory. I think the closest things we have are scaling laws, and these are empirical laws. I think one misunderstanding people have about, like, our approach to RSI is that they think the entire system has to be fully automatic and digital. But actually, if you look at our mathematical foundation, we have a multi-agent system, not just one digital agent. And in this multi-agent system, human researchers and engineers are actually a special case of these agents. And it's this hybrid system of multiple agents working together that's been recursively improved.
I see. And we don't want to leave out any component to not optimize them, because, like, if it's possible to optimize in the model, why wouldn't we do that? Because you will have more gains if you do include those. Anything that you don't optimize will later become your bottleneck.
So one observation that we have made here is there are claims of speed improving, which is actually the case, right? So if you look at, like, the model release time, it is reducing. And so on one hand you can be like, okay, so clearly we're in some sort of a, you know, rapidly improving phase. But if you divide by the actual amount of capital raised by these companies, that's no longer the case, right? And so you're like, well, maybe they just raised a whole bunch of money and then, like, AI allows you to apply that money. So how would you—this is not for yourself, but how would you encourage people that are either evaluating claims about this or even working on them to think about not fooling yourself by throwing more money at the problem, like you're actually showing that you're making an improvement here that's meaningful with the denominator?
I think for us it's pretty straightforward. It's because if you get improvement and they just come at more and more resources being poured in, it's not RSI. You have to be able to back your claim of how much improvement you're getting. But do include, like, all the man-hours and experimentation that went into that, because I feel like at some level, like, they'll talk about, like, the actual FLOPs that go in and money that go in, but the reality is this is, like, the 23rd experiment, and it's actually a human burning a lot of mental capital to get to that final configuration. Like, do you actually include that in the denominator?
We do, and I think it's good company discipline to do that.
Yeah. And we've tried, again, to very intentionally from the beginning have a culture of tracking where things go, cost, time, all that kind of stuff. I'll share a funny thing: internally we have this thing—people talk about leaderboards, you know, which is like score. You know what I'm about to say. We have something called a bleederboard, where we keep track of the cost of experiments of everybody who's running experiments.
You know, and everybody—it's the company—everybody any day can look at, like, who's high up on the bleederboard.
I can also give an example from AI. I don't know if you know this nanoGPT benchmark they are running. It's basically asking you to retrain GPT-2, which when it just came out is a very expensive model to train and takes weeks, I think, on multiple GPUs. And right now people can train on a single GPU within hours, and that number has been shrinking constantly.
But that's also totally separate, right? Like, that's just, like, whatever, Moore's law, like...
No, no, it's the same GPU. It's the same hardware.
Using the same hardware, and then you improve the techniques of pre-training.
Oh, interesting.
Yeah. What has improved is, like, algorithmic techniques, data techniques, kernels, all of that kind of stuff.
Do you know how much it cost to train GPT-2?
I think it's a few hundred.
$100. I just—oh, wow. So the same GPU...
And this is just algorithmic. And do those come from humans or do those come from...
Agents. A lot of them are auto-research results.
But that's definitely the denominator. That's not the numerator. Like, what you want to see is, like, you can do that with GPT-5, which we're not seeing. [laughter] Right. I mean, like, the cost to these things is going up dramatically.
I think one of the most closely guarded secrets in the industry is exactly this thing, which is how much compute you can get out of this.
Because in some sense, even if all you were improving was your denominator, you could get faster training because now you can get more rollouts.
Right. And that means now you can sample—what you were describing, like, the manifold of the space of language, more extensively, and then you can get better trajectories, which means you can get smarter models when you use that to train.
For sure, improving the denominator is good.
