Welcome to "Google DeepMind-- the Podcast." I'm Professor Hannah Fry. Right, agents, they are here, or almost here, and they are probably all anyone is going to be talking about in 2025. But they are definitely not new. My guest on today's episode is someone who last came on the podcast in 2019 to talk to me about the multi-agent system he was working on. It could beat the professional StarCraft players at their own game, and eventually went on to achieve Grandmaster status. But how have agents evolved since then? What can they do now? How have the advances in language models and multimodal AI change things? And how do you possibly go about building something that can make autonomous decisions on behalf of its user? Now, I should tell you, if you want a primer on agents, you can watch our episode with Frederic Besse that we recorded over the summer. But for now, Oriol Vinyals is Vice President of Drastic Research and co-tech-lead of Gemini. And it's fair to say we've got quite a lot to catch up on. Oriol, welcome back to the podcast.
Hi, thank you for having me.
What is Drastic Research?
Well, I keep telling my team they have to think drastic, meaning don't just do the incremental stuff that everyone is thinking about. Try to drastically think what will happen in a few years time, and then try to backport those ideas and then execute today with that mindset in mind. So that's what drastic means. But yeah, it's a word that I use a lot.
I think when I last got to see you, you had been working on an agent that could use a keyboard and a mouse to do things draw pictures in Paint or play StarCraft. And, well, things have moved on quite a bit since then.
So those agents, at the time, you took sort of a very generic set of principles, very simple principles in the field of machine learning, and you would basically specialize a model on one task. And what we were doing at the time is have a curriculum of tasks that were more and more and more difficult, right? So when we last spoke, for instance, in video games, we were looking at StarCraft, which is one of the most complex modern strategy games out there. And of course, DeepMind is notorious for having started the trend with Atari, which is a fairly simple game of left, right, hit the paddle and hit the ball and off you go. So that is what the algorithms themselves, we tried to push for them to be very general so we can keep climbing this ladder of difficulty, curriculum of games, and doing more and more complex things. And right now what has happened is even the models we train are broadly applicable to many more things than the models we developed back then were. So think about the process of creating this digital brain hasn't changed that much. But what that brain was able to do was reasonably narrow, although very complex, like playing StarCraft or playing Go. Right now, these models can do quite a lot more broad applications and, of course, talking to us, chatbots, et cetera, et cetera.
So back then, reinforcement learning was your main lever, I guess. How different are things now?
Yeah, so algorithmically, actually, the process of AlphaGo and actually AlphaStar, those two had the same set of sequence of algorithms applied to creating this digital brain. And it's not actually that different from how current large language models or multimodal models are created today. There's two basic steps that have been pretty constant throughout many years in many of the projects that we've worked on, which we can call the first one pretraining or imitation learning. That is, you start with random weights, you have an algorithm that will try to imitate lots of data that humans have created to either play a game or, in this case, all of the internet, all of the knowledge available to us. And in that first stage, you just adapt the weights to try to imitate that data as well as possible.
And these weights are, essentially inside each of the neurons is a series of numbers that kind of describes how it's connected to everything else?
Yeah, so basically there are units of computations that are neurons. And the connections between neurons are what actually you have as weights. So you can imagine that there's a neuron, there's a few neurons connected to it. And you're basically adding all the activations from the incoming neurons there multiplied by the weights. And those weights are the only things that move. And the inputs excite the neurons. It's pretty much how a brain works with some freedom of, yeah, creativity.
OK, if we were to do an analogy, it's almost like you've got the neurons and you're like, water is flowing through it and the weights is like the width of the pipes between the neurons?
Yeah, that's right. And then you can imagine having millions of neurons and billions or even trillions of pipes. And that is what we spend most to compute actually training these models, especially language models, is in this pretraining or imitating all the data that we have available to us.
OK, so you've now got this gigantic network with loads of pipes going between all the neurons. And that's your imitation phase done. Next bit, if you were doing, say, AlphaGo or AlphaZero, you would then get it to play itself.
Yeah, so this model now is reasonably good at playing moves that look human-like. So that means, of course, the sentences are very plausible sentences in English. Or if it was playing a game, it would click things reasonably to move pieces on the board and whatnot. But what this model hasn't done is learn that these actions yield reward. That's the bit of reinforcement learning or post-training, which is the second phase of training. So you can write a poem by just, hey, just how does a poem on the internet look like on average? But then the question is, well, I want only the good poems. So how can I further adjust these pipes based on a signal that now, having written now a whole poem, would give a score of 0 or 1, let's say. And if it's a mediocre poem, you get a 0. If it's a good poem, you get a 1. Again, for a game analogy, which is what we use reinforcement learning traditionally, if you win at the game, you get a 1. If you lose, you get a 0. And then you further adjust the weights. But now, instead of imitating humans, you're just saying forget, I want to go beyond what humans could do and try to really get all my poems to be the perfect poem or all my chess games to be the perfect game. And in language models, this second phase, which is reinforcement learning post-training, tends to be fairly short-lived because we do not have access to super clean reward as you've won the game or you lost the game, when you do self-play in traditional board games, for example.
So once that's done, that's all the stuff that goes on behind the scenes. And then you're like, hold it right there.
Stay exactly where you are, everybody. We're going to take just basically a snapshot of this entire network, and that is what you actually get to access as a user.
Yeah, so now this amazing process finished. These weights are super precious, right? So this configuration you found, you've really spent months to finesse it, to tweak everything. And now you will never move it anymore. So training is over. You're not changing the configuration anymore. You might want to make it super efficient. So say you find that, oh, look, this neuron is not that useful. It's not used for anything. You remove it, so everything becomes faster and cheaper to run it at scale. And then as a user, you just get the same weights. Everyone gets the same weights we've trained. That's what we call Gemini 1.5 Flash. That just means a set of weights that are frozen, will not change, will not further train or anything. So those two steps actually pretty much are identical from AlphaGo to AlphaStar to current large language models. And of course, there's details that matter. And the field has evolved, certainly, but the principle is pretty much unchanged, actually.
Because under the hood, as it were, there are differences between-- I don't know, I'm thinking like DQN here, which was the Atari example, or the types of algorithms that are used in AlphaGo or then again, in the large language models, the architecture is different, right?
Yeah, so there's a few components that go into then what the digital brain is. One is the architecture. So there are these neural networks. Now we have the transformers, which we certainly didn't have back in the DQN days. So there's always some sort of breakthroughs in architectures that are better at learning from the data. But then from transformers to today, it's almost all about little tweaks. Even if you look at AlphaFold, which also is fueled by a transformer, what the teams do for years sometimes is just to find little tweaks on, hey, let's remove this set of neurons, let's add another layer, let's make this a bit wider. So the brain shape changes a little bit, and that makes it or breaks it sometimes, in terms of the performance achieved.
So if these are all the things that have been achieved so far, the goal, as I understand it, is to create more agentic behavior, to get these things to make autonomous decisions. How did these help to achieve that end?
Yeah, so let's zoom in a little bit on the current trend. We call it large language models, but they're multimodal. I think we had an episode earlier covering heavily the multimodality aspect, how good it is to be able to add an image, then ask something, a follow-up question, and so on. So this score, we will still improve it, this set of weights that do these amazing inferences about the input. What's this image about? What's the user asking? Can I write a better poem? Can I just make it longer or whatever? Like all these interactions we all get to play with these days. But this is just a component. And we can think, hey, this is now our CPU. And we can add more to it, around it. What if the model could go off and do research for you, for example? One example, we were already thinking about that back in the day, I could ask a model, a language model or a visual language model to learn to play the game of StarCraft. That's a very different approach to say, create one agent that does play the game. In this other example, it could go online, watch videos about the game. It could, of course, download the game to start interacting with it to learn, oh, yeah, I know, I get it. Do research online, go to forums, read the forums. Go play and figure out that it's weak at this thing and improve and so on. And after, literally, it could be weeks, it sends you an email that says, I now know how to play the game. Let's play. That's not a reality that's that far away. But these models all of a sudden actually do something, take some actions, and learn anything new that is available to them. And that's pretty powerful to think about. It's what pushes the generality the most. And that's what makes the AGI, as many people call it, feel closer.
So if I understand it correctly then, it's almost like the stuff that we have at the moment, the large language models, the multimodal models, whatever you want to call them, that's the central core. But the next step is that you build stuff on top of that central core, that it can go off and take off the stabilizers and go off and do its own thing.
Yeah, exactly. If it has access to all the knowledge and it can use its time to do some proper research, write hypotheses, write some code and so on, and take its time to really answer very, very, very complex questions, then, yeah, the possibilities now have broadened quite drastically. Although of course, we're not going to need that for everything. I mean, if we ask a question like, hey, I like rice. What should I prepare tonight? Probably no need to do a very deep dive into thinking or just going off for three weeks. Then you'll probably not be very happy about the waiting time, right? But I think to push the frontier, you're giving a digital body to the computer so it can not only just think and give an instruction or a word output, but it can also go off and do things online or on documents that you might upload or whatever. And ask very, very complex questions and personalized to you, et cetera, et cetera.
I like that idea, this central core and then you're giving it a digital body. You've got the electric brain, and now you're giving it a digital body.
OK, so in terms of the electric brain and in terms of this core, this processor, let me just ask you a bit about that. I guess we should be thinking about Gemini here, right?
Which is essentially what we're talking about, the multimodal model that you guys have. I know that one of the big ideas for large models was just to scale it up, to get them bigger and bigger and bigger and bigger and bigger. Do you think that the results that we've seen from scaling have plateaued by now?
Yeah, it's a very important question. We have studied how as you make the models larger, that is how many neurons literally these models have, how do they become better at certain tasks that we have clear metrics from the whole machine learning community? For example, one that is very simple to understand is machine translation. So how good the models are at translating between two languages as you scale, as you go from millions to billions to potentially trillions of neurons, you can see the performance keep improving. Now, even when you do those studies, one trick is that it looks linear, but you have to plot logarithmic axis. What that means in lay terms is that, let's say from the last three years, we had some improvement, you shouldn't expect the same improvement in the next three years. It's actually exponentially hard to get there. So that means the compute investment, which of course, also advances at the super linear rate, but perhaps not as good as these sort of trends suggest, you would just see some diminishing returns. Because simply scaling the x-axis, the number of parameters, you need to go 10x to see the same improvement. And that just creates some pressure to, hey, maybe we can't scale as much and we need to think about other ways to scale to make the models better.
The example I give to my students is like, if you've got a room that's really messy, the first 10 minutes that you spend tidying, it's going to make a massive difference. You pick up all the dirty plates, put away all the dirty washing, fine. But once you're like 7 hours in, that extra 10 minutes, it's not going to make any difference at all. And that's essentially where we are, right?
Yeah, that's exactly a very good analogy. And in fact, that analogy can even apply to then the performance of the models. Even if you have extremely good performance, if you want these models to be 100% factual, they will never make something up-- we know that if you probe them, you can make them say some things that are not real-- even that last mile also is super hard, which creates some interesting challenges to deploy them at scale.
So OK, I hear what you're saying about how there's diminishing returns in all of this. But in terms of how you make these things better, how you make these models better, is it just data, computational power, and size? Are those the only things, the levers that you have to pull?
Yeah, so certainly, if you froze the architecture, let's say for the next year, no innovation, we just scale because there's better hardware coming out--
Just make it bigger, yeah.
--make it bigger, that certainly would have a trend that would look OK. But what's happened, and certainly in Gemini, we have other innovations, other tricks, techniques, details from how to order the data that you present the model with, to the details of the architecture, to how to run the training process, how long to run it for. What kind of data do we actually present the model? How do we filter? Do we present more data that's high quality, less data that's low quality? All sorts of different what we call hyperparameters and, of course other algorithmic advances, we also investigate fairly carefully. Because the process of training a model is expensive. So we need to be extremely careful with piling up innovation so that eventually when we are ready, we have enough innovation. And also probably we'll have a better scale to run for the next iteration of models. We run it, and then we get algorithmic, not only breakthroughs through data and compute.
I guess the other thing about this scaling stuff is that there's no limit, really, to the number of nodes that you can put in. Maybe there's no limit, in theory, to the computational power that you put in. But there is a limit to the data that you can put in. There's a limit to the number of human words that are out there.
Good point. So I think there is a limit on the nodes. Because how you scale these models is, well, they don't fit on one single chip, hardware chip. So now you have a mesh of chips that are communicating. There's certain limits like speed of light, et cetera, et cetera.
So there starts to be a time where the efficiency of training such a big model is also just not worth it, even from a utilization of the hardware at your disposal, but very good point. The other bit that is critical on this pretraining, imitate all the data, is that we do not have what we call infinite data regime. There's finite data. And so as soon as the models need to-- you can think, well, let's train on all the data. If you want to train--
Everything humans have ever read--
All of the internet. So we're just starting to think, OK, we're running out of data. There are techniques like synthetic data. Can we write or rewrite existing data in many different ways? I mean, languages would be obvious ways to think, hey, you could write the internet. It's mostly in English, I mean, 60%. I don't know what's the exact percentage. But there are ways to rewrite the same knowledge in different ways. We're exploring those. That's kind of a research area that many people are starting to invest. Because if you run out of data, the scaling laws punish you even more.
So for example, then, you could get Gemini to write its own version of the internet, and then use that to train a new version of Gemini?
Is there a danger, though, that if you start feeding in the output of the same model, that you can end up creating these little, well, unhelpful feedback loops?
They certainly can do some interesting experiments to test ideas like this one you just mentioned. And indeed, that is, on the surface, not a good idea. The model suffers if you just ask it to recreate all of the internet. And indeed, a priori from information an content point of view, look, this data set has the information that it has, how could you create new information? I don't know. These ideas might help a little bit because there's machine learning deficiencies that we're not at that fundamental ability to extract all the information truly from the internet. We have good algorithms, but they're not perfect.
I guess I just want to think about that a little bit more because it's a really interesting idea. Because of course, naively, if you did it without thinking, then it's like, the new version would have the biases in it. And then the new version on top of that would be more biased. And you'd end up sort of spiraling away from the original human one. But then what you're saying is as though in the original human internet is sort of embedded these conceptual connections. And if you can extract those-- I'm thinking almost like E equals mc squared. If you can find the E equals mc squared for human concepts and then just generate new data using that alone, then that seems more realistic.
Yeah, exactly. And I think that's where you start hitting-- I mean, are these language models just repeating what's online and not being able to create anything new? Or are they learning a world model truly that you can then from the principles it extracts, possibly generalize beyond what the data has? And under the more optimistic version, which I tend to believe more, we can push the limits of data little bit more than the current limits that we have. That being said, there are some data sources that we haven't quite seen a breakthrough, like video data. There's a lot of it. And we haven't quite seen a moment of take all the video data where you probably can derive a lot of knowledge, a lot of laws of physics, a lot of how the world works, even if there are no words associated with the videos necessarily, and extract that knowledge. Even that I don't think we tapped into that source.
And it doesn't work that way?
Yeah, it feels like it should, even how we learn. There's some language learning in the early days, but we learn by also observing three dimensions and so on and so forth. So there probably is more knowledge that we haven't extracted. What obviously we've gotten pretty well at, and you can see by testing the models, is connecting the concepts present in the video. And then you can do amazing things like, hey, take this full hour video and just extract me three interesting moments. But the model itself, has it actually used that--
--information directly? Probably not.
Oh, I like this so much. We were talking to Jeff about with multimodal models, if you get these models to just watch all of the videos that have ever been created, can it quite literally extract what gravity means as a concept? But what you're describing here, if I understand it, is that at the moment, it can tell you what's in the video that it's seen, but it can't then say, and E equals mc squared. Or if you showed it pictures of the night sky, it wouldn't suddenly be able to predict the planetary motion in the same way that human astronomers did.
Yeah, exactly. The shortcut we're taking here is that the data we train when we train on images and/or videos, we almost always have a text representation associated with that modality. So it could be a caption explaining what this image has or this video has and so on and so forth. And, of course, then it's incredible, right? You can put a picture of homework and with a little drawing conceptual and it will connect and do quite a lot of good logic just based on that. But what I'm saying here is, could I just take videos, no language, and then train a model to then understand what's happening? Maybe even, in a way, derive a language-- it's obviously not going to be our language-- and extract those concepts. And that has not happened. And it probably will.
Just going back to what you said at the beginning about those two phases to basically all the models that the DeepMind have built.
The imitation phase, which is what we've been talking about right here, but then the reinforcement learning phase on top. And I know that AlphaGo and AlphaZero and many more got better by playing themselves. Does that apply here as well?
