Hi everyone. It's great to be back in my alma mater at Berkeley. Today I know this is about agents, but I've been focused a lot more lately on recursive self-improvement. So I'll try to give you a little bit of my own perspective on it. Okay. Anyways, so I'll start talking, but the slides hopefully come up. But I think one of the things that at DeepMind we worked for a very long time is agents actually, and it's kind of interesting to see the evolution of what agents have meant to be over the years, especially when you have a company that has been building around reinforcement learning, which defines what an agent is, for over a decade now.
So my first slide, if it appears, would show the usual traditional reinforcement learning loop where you have an agent, you have an environment, and there's observations and actions being interchanged. And generally those are like the agent is powered by a neural net and so on. But then the observations and actions are generally fixed and the environment at the time, and it makes sense, that's my slides. The environment at the time we used video games, which hopefully makes a lot of sense today. The arguments we were making at the time, people were like saying, hey, games are not real world. I mean, this is not perhaps going to be as impactful the technology as you make us believe. Obviously, hindsight tells us this technology has been and is going to be very impactful. But video games have this very nice property of being sandboxed even more properly than the current sandboxes we have. So we were testing algorithms and ideas on the agent capabilities on Go, you know, StarCraft, and all sorts of different games until of course games were like we ran out of games that we had tackled the most complex games and so on. I'll get back to that point a bit later about games.
But you know, the preliminary era agents, the graph was fairly simple. If you look now at what agents mean, and we've heard so many times in the talks in the morning and now, I don't want to go too deep into this, but there's a bit of a reshuffle. I think the one that's most profound is that the environment is hard to disentangle from the agent. So I put the environment inside of the agent, right? The environment is like a machine, possibly with access to network, internet, and so on, in which the agent harness that has a large language model as a brain. Not that different from that other picture that was at the time powered by a neural network trained specifically for like a game or a domain. But this sort of interaction is so important and so critical that it's very hard to disentangle. And then unlike before, we have a user that is interacting with this agent. You know, perhaps coding or asking it hard questions that might take a while to run. And the environment in general can be shared with the user. It might be that's my machine. There's also some machines that can, virtual machines that can spawn on the cloud and so on and so forth.
So I'm going to describe a bit what I think, you know, a very practical definition of self-improvement and recursive self-improvement is through this lens. I think it's not that different actually, that's why I'm bringing it up in the agents section, what SI and RSI are. So self-improvement is kind of a bit of a nuance over what agents are doing for us today. I would say probably the user is not as interactive with the system. I mean, self-improvement means that the system is operating by itself mostly without much input. So a user might give us a specification and then this process might run for hours and hours or days and days rather than maybe being more interactive. And then critically, there's a fitness function or whatever the improvement target is or might be. I think in the morning there was a Tetris, right, as the target or the fitness that then the system was trying to optimize. But it's not that different from an agent, it's kind of almost a detail.
And then if you go one step further, the recursivity, the simplest way to think about it is like, well, wait a second, the agent harness itself, the code that runs the agent, and even the brains behind the agent actions, the LLM or set of LLMs, now can also be modified, right? So these are trivially can be seen as part of the environment, right? The code that runs your harness and the model, maybe it's an object that you can sort of upload, you know, to your server that's serving tokens and so on. And so recursive, the recursive part is just allowing this environment and agent harness to modify itself or the model. And it's useful to distinguish self-improvement in the harness space, right? This is more like the code, the how you do like compaction, all sorts of different things that you need to implement for agents, you know, what tools does it have available and so on. And then of course also the model weights themselves, right, which could be rewritten from scratch, could be fine-tuned. And I mean, the action space for the recursivity part is anything that modifies that object, right? So this is fairly simple. I don't think we've seen very impactful actual recursive self-improvement examples yet. Certainly many self-improvement examples. Those are more commonly seen. But of course this is coming and it has fairly deep consequences into how we do our own research to develop then the harness and the LLMs, which of course is at the core of what we at many frontier labs are doing.
So unpacking a bit what are, you know, how would you expect this agent with the brain to do recursive self-improvement, it's quite useful to put a bit of my researcher hat. I mean, we all, or maybe not, but many of us have written papers, have thought about ideas to change the field and so on. So the process in which you would expect a fully capable recursive self-improvement system to do is not that different from the process that, you know, many of us have been playing, you know, as a game for years to publish papers and trying to get our techniques adopted widely, right? So this can be decomposed in so many different ways, but I think it's useful to think about it in these four ways. More detail can be, you can zoom in on one of these and so on, maybe that's a good point for discussion. But very critically, there's an ideation phase where you take all the previous work, all that you know about the field, and you select which ideas are worth pursuing fundamentally, right? So this is a kind of a very important reasoning step which makes you go into a next stage of then implementing said idea, right? Like you have to implement it. It might be complex, it might be simple. After which you maybe have an experimentation phase, you run the code, there's bugs, it doesn't work, maybe you insist, maybe you give up and go back to another idea and so on and so forth. And last but not least, you evaluate what happened and whether the hypothesis maybe that you formed on the ideation phase is verified or not. And then you might obviously traditionally write a paper. In a recursive self-improvement system, you might just go back and loop around, right?
So what's critical about this, and my observation is currently a lot of effort is done in the middle two boxes, implementation. I mean, we know how powerful coding agents are these days. So I think there's certainly a lot of progress and enthusiasm behind implementation and experimentation as well, like running, compiling, you know, debugging. Also, I think we've seen, perhaps because it's the most interactive with the user, so more similar to self-improvement than recursive self-improvement, we've seen a lot of progress there as well. But the other two not so much. So I'm going to zoom in a little bit on to kind of some maybe thought-provoking thoughts on what might happen and what needs to happen for us to really achieve this loop meaningfully with full automation.
So evaluation is always a very important topic. I find as a community evaluation and data have never been perhaps highlighted as much as they should have. Even, you know, different papers that propose a new dataset might have been regarded as an application and not worthy of publication in one of the bigger conferences certainly years ago. So I want to insist on evaluation because it's so critical. So how do we measure progress in a more like scientific way as we are developing better models and better harnesses, right? The first way is kind of the obvious way. We take some capabilities we believe are important in this and some benchmarks. So SWE-bench, ProBench, perhaps perplexity on held-out data in pre-training is commonly what most people are optimizing for these days. And then you just hill climb, right? You just say, look, we're going to get better at all this. And hopefully by doing this, recursive self-improvement might emerge. And certainly there are pros for this approach. I mean, this is exactly how we've been training LLMs for a few years now. Evaluation is reasonably cheap. We know how to do it. It's well defined. But I mean, the cons, I would say the one that is the worst is that it's a very indirect measurement of actually a system that might truly develop recursive self-improvement. And of course, there's all sorts of issues with overfitting, cheating, and also the current evals, at least that we have, are focusing, as I said, mostly on those two middle boxes, implementation and experimentation.
Now, a much better way would be to evaluate literally for what we care to improve the capability of, which is in this case recursive self-improvement. There are some benchmarks that are appearing because of course this is gaining traction in the field. Post-train-bench is one example, but Don was telling me about another one. So we're going to see this popping over the next few weeks and months for sure. And of course, the advantage is that we measure what we care for, right? So we are improving the self-improvement capability if we set the evaluation in a way that is like, hey, like is the inner loop itself is like, hey, evaluate some other metric, go for a few tries and see how much improvement you get, right? That's your evaluation. It's a bit meta, but it's more powerful. Of course, there's a few cons. This is expensive right now to evaluate. You literally need to run an agent for a while to improve some set metric. And then the inner loop still will be, you know, I guess improving Tetris or something that is not perhaps the actual metric that in production we'll care to evaluate, which would be, hey, I want to automate my whole frontier lab, create the best model in the world, right? And I mean, these evaluations might be a bit out of distribution as they stand today.
And then perhaps a modification of this, very relevant for what's been, you know, the discussion of the day and certainly what I expect to increasingly happen in the field, is to tweak the previous one by also not only evaluating for, hey, how much can I, you know, better play Tetris over, you know, three hours of compute that I give this agent, but also how do I achieve that improvement, right? I don't want to see cheating. I want to see creativity. I want to see certain aspects of intelligence and evaluate them with, you know, some powerful reward models that are a bit more fuzzily evaluated but still very important capabilities that we expect this model to exhibit. So an example, just to give it a concrete and maybe to use ideation as I think a fairly underexplored area in general in LLM training. So well, how do we evaluate ideas, right? Well, there's many aspects. If you are a reviewer in a conference, right, you're looking for research taste, novelty, you know, efficiency, elegancy, is this going to stand the test of time, this kind of technique and so on and so forth, right? So you can imagine trying to evaluate the behavior of an agent doing recursive self-improvement on that ideation stage based on a few of these that we can write rules and try to automate a little bit. But it's very hard at the moment. And these are very understudied. So I think this is going to take us some time to figure out to advance these capabilities and the reasoning, the automatic reasoning that might be required for us to be able to optimize with, you know, reinforcement learning on top of this system. And I would argue even as humans are not that good. I mean, we have very expensive systems, conferences, review processes and so on. And transformers actually took many years to become mainstream. Distillation, which is getting more famous now, was actually rejected from the conference we submitted at, and it took many years and some sort of random process for it to reemerge as, hey, this might be actually useful. And I mean, all of you probably have your own sour rejected paper story of a great idea that in hindsight should have been accepted by this automated and expensive process that we set up. So automating this truly in an agentic way might take some effort, but it's worth pursuing to achieve RSI.
Of course, we've seen, and I bring back the games here. This is not a surprise that our agents are sort of doing things a little bit differently than what we expect. So we had very famous games that are exploited in some different ways. The other video doesn't work, but it's also quite funny. So this is many years that we've seen this coming. And then to wrap up, right, we need to obviously, it's an incredible time to be alive, right? It's definitely clear that RSI will increase perhaps 10x or more the speed of progress of certain in engineering and science tasks for sure. But you know, be mindful of how we set the goals, misalignment, resource reallocation, how the labs are run might change. So we need to be quite careful about how to take different steps in this. And as I was arguing, it might not come as fast as we thought it would because first, evaluation is very hard and ideation is also very hard. So these are two important components. Also, some process might be just limited by physical constraints. I mean, the chips cannot go faster than what they're designed to be. Of course, also like speed of light and so on, right? To transmit data around like the chip and so on has also certain limitations. And then last but not least, performance that we had obtained as humans might be close to an asymptote. So by adding recursive self-improvement, we might anyways not see a drastic improvement, right? So how good is AlphaGo with respect to the, you know, perfect Go game? Unclear. Or another example is sometimes it's kind of annoying how well these models write and you have to make it look a little bit worse. So they got too good at writing, for example, right? So there's some of these domains certainly where better doesn't mean better necessarily. And with these thought-provoking thoughts, hopefully we can have a good panel later, but looking forward to the rest of the session. Thank you very much. Thanks.
