Amazing. Great. So it's great to be back at Berkeley. There was a probabilistic trip, which would have helped me to be in the US today. But sadly, I took it very short last week. So I was in the Bay last week but not this week, so we'll have to do this online. But yeah, it's great to be virtually back. I have deep roots in Berkeley, as you might know, if you have known me for a while.
And today, I mean, this lecture really will have two parts. And it's a medal, actually, in a way. And it's also kind of, selfishly, I'm doing this also to remind, I mean, first and foremost, people in the field that lots of ideas recirculate. And so, in a way, I put together this slide deck, or I recovered the slide deck, from the AlphaStar project, which is six or so years old, and realized that many of the current issues but also solutions that we're using in modern LLMs trace back further before.
But I mean, we did encounter many of these issues ourselves when we work on a fairly, probably our most advanced, even today, especially on video games, agent that DeepMind had built at the time. And I'll get into details now in a second. But it's useful to revisit these ideas to see them with perspective and also realize what are the things that we did back then that, currently, we can't do, or we're not quite doing at the same level in LLM. So this lecture will be a bit of a mixture between AlphaStar and also Gemini and other LLM agents but with the kind of hindsight, foresight lens, which hopefully, you will find entertaining.
So let's go through a bit of basics. Now of course, I'm sure you've covered many of these topics. And you probably know them because they become quite mainstream. But indulge me into giving you my own perspective and point of view. So this is the OG slide of DeepMind, actually. The pitch slide deck had a version of this that did not have StarCraft, but it had agent environment, agent with goals, observations, and actions. That was, at the time, not that mainstream. Reinforcement learning wasn't mainstream when DeepMind was eventually acquired. And also, it serves a little bit to try to put our heads on what is an agent since this class is about agents.
And I think today if you ask people, like, is this an agent, is this an agentic behavior, you probably get quite a few different answers. Back then, things were a bit simpler, though. We were mostly connecting to single environments with, well, spect and defined observations and actions. And the agent had just, maximize reward as a goal. Of course, in games, many times, the reward was fully specified and fairly simple to compute. And then this was the loop that agents lived in forever. And their goal was to optimize reward. Now today, things are a bit different. Besides, I guess, having agents in the traditional sense, we have assistants that assist users. So then, what is the environment, what are the actions, start to be a bit more diffuse. And I think embracing that is fine.
This is probably the most accurate, at least from our side, perspective of what we mean by agents. So you might be a user of a chatbot or advanced system. And you're going to interact with something, this object. We can call this the agent. And as a user, you'll ask questions, maybe give feedback, et cetera, et cetera. And then this agent now actually is more complex. In a new way, I think it's more useful to think about the environment being part of the whole package. So you'll get an agent harness that's controlling how you call the LLMs to issue an action or issue a search or issue a terminal command, et cetera, to this very rich environment that could be a digital computer, a network of computers. It could be anything. It could be a game, of course, as well, very general. And then that's how we orchestrate the whole thing. So it doesn't quite fit the paradigm as neatly. But that actually things-- makes it more exciting.
So sometimes when I'm presenting some of the results and observations, especially from AlphaStar, it's going to come from a more simplistic world, where single environment, clearly defined, and no user, so to speak. So the world has changed. And these will keep evolving, especially as we get to an exciting topic that, I think, is the one I want to leave you, maybe with some thought-provoking ideas and observations on multi-agent systems, and what does that even mean here. So a bit of history.
The DeepMind, at the time of acquisition, had just mastered Atari. I remember fondly the NeurIPS in which Vlad presented the results. And they felt really refreshing. I believe that was the sequence-to-sequence in NeurIPS. Maybe it was the year before. I can't remember. It was a while ago. And then in the internal program for DeepMind, there was sort of a curriculum of difficulty of games that we were going to tackle. And famously, Go came a few years after Atari. And then I would say, maybe StarCraft graduated as to just master all due to its complexity. Although, of course, there's many interesting games that have slightly different complexities. But StarCraft, clearly for AI, felt like a good kind of ending note. Although even today, we are working on games still, as you might have seen with things like WOT models. And SEMA 2.0 was just released that can play 3D games and so on. But the approach is quite different.
Now, blast from the past as well. So 2011, ages ago, I used to be a grad student. And very exciting from Dan Kline's group was this project that tried to tackle solving StarCraft with machine learning. And there was a whole class set for testing, kind of maybe somewhat good, old-style-fashion AI, although some machine learning as well, interleave with just trying to play the game or play different aspects of the game. So it was recognized, of course, as a domain. And personally, of course, this resonated very much when I saw DeepMind for the first time applying ML, which I love, to video games, which I love-- I used to play StarCraft-- to eventually become the lead of the AlphaStar project. So very, very interesting domain at the time. In 2011, really, it seemed quite impossible that a single neural net could play the game. But that's basically what we did.
And for those of you who do not know StarCraft, I mean, it's very hard to give you a two-minute overview. But it's essentially a chess in which you don't see the board all the time. And things are fairly continuous, like actions. It's a real-time strategy game. So there's lots of hiding information from the enemy because it's partial, observable. And generally, the action space is quite complex because you have to control not only the strategy at the high level but also the microstrategy about how to move your units about the map. So here you have a map. It has resources. You gather resources. You get money. You build buildings. You build units. You engage the enemy. And if you destroy all the enemy buildings, that's the winning or losing condition. So it's kind of a fun game. It's been quite heavily played over the years, different versions of it. And it brought esports to life in South Korea, certainly very, very early days, in the late 1990s even. So this kind of pioneered, in a way also, certain aspects of video gaming.
Different points of view, this is the one you might have actually seen that's rendered directly from the game. This is your UI. You only see on the bottom left here, part of the map. You need to move the camera, send the units around, build buildings, issue commands. It's fairly fast-paced. So there's lots of cool subproblems that you could tackle as an AI domain. But as I was saying, what we actually end up doing-- and I'll describe bits and pieces but not in full-- we trained a single neural network, basically, that was issuing all these commands and how and it matters to LLM or current trends that we're seeing. I'll describe in the second part of the talk. And then in the third part, we'll cover multi-agent.
So achievement wise, I mean, it was fun. And I remember fondly when our agent achieved grandmaster status, which is given only to the top 100 players in any given region. So we were playing online, literally in our office. So imagine playing video games at work. That was pretty fun. And we were all sitting around, playing games against people that didn't know they were playing against AlphaStar, although many could guess. And there was lots of speculation because you had to opt in to play against possibly advanced AIs. But eventually, we reached the grandmaster milestone. And then later, we also went sort of in the wild in BlizzCon, which is a NeurIPS for StarCraft, so to speak, where we played many, many folks who just were mesmerized by playing against an AI at the time. This was, I believe, our first AI system in the wild from DeepMind, actually. So it was quite nice to just interact with people, see how they felt about the game, AI, et cetera. This was 2019, right before the pandemic hit. And so we visited this wonderful, very different style, conference, I would say than NeurIPS, although I think NeurIPS is becoming a bit like this as well lately. And it was fun.
We played a bunch of people, including Serral, who was one of the legends of the game. If you want to see the games, the five games we played, I believe YouTube links are here at your disposal. You can see here, me, a bit tense to see what the agent would do under such strong opponent. But it was all fun. And I think probably the most profound realization was a couple of weeks after, you see, November 24, Serral mentioned that he actually copied StarCraft or AlphaStar, one of the that he had played against. And of course, an obvious comment that, again, in a way, concluded this epoch we had at DeepMind on mastering games in one-on-one setting, was that we got a player learning from AlphaStar. But of course, AlphaStar had learned from several, as you will see in a second. So that was pretty fun.
So let's start to see, maybe some obvious, some less obvious, connections. And then yeah, basically, why does it all make sense? Even when we talk to reporters at the time, very early days, they didn't understand the impact of playing games in real world. So here I try to map, fairly imperfectly, the timeline of certain major projects, Alpha asterisk projects, if you will, that kind of, in a way, took inspiration, either algorithmically or learnings or whatnot, from the previous projects that had come in because games were such a good sandbox to try some ideas to scale up neural networks. And I mean, at the time, it was a bit hard to convey this to, maybe, general public. But it's great to see that this kind of plan for, really, the whole company was set about, is taking shape. And games still have a role to play, by the way, into building, pursuing AGI in the context of LLMs.
But here, there's a few projects. I didn't put all of them. Certainly, there's no Atari, for example. But certainly, as I mentioned, after Go, we decided to tackle StarCraft. And then there's no other games. But after that, we had huge inspiration to start working on coding early days on our first LLMs-- Chinchilla. Visual language models Flamingo or Gato. And AlphaFold also took inspiration of, I mean, certain aspects of transformers, distillation of data, et cetera. And today, as you probably can guess, a lot of the research and resources are now heavily invested into a single project, almost, although it's very large-- so there's many, many different components-- called Gemini, which I'm co-tech leading, together with Jeff and Noah. And of course, in there, we're just trying to build the most general system we can. So a game or playing a single game might be one of the many, many things, the incredible things that these models can do today. But it's nice to also look back and attribute a bit of ideas or where things come from.
So more specifically, let's dive into some technical details. And some have striking closeness, I would say, with how today we're creating some of these amazing LLMs. So this is from the Nature paper that was eventually published. And we'll discuss the three aspects. But in sequence, we did, basically, three things to create this single neural network. We had, first, supervised learning, as we called it in this figure, learning from human data. Then we had a step of reinforcement learning. And this reinforcement learning didn't happen in a vacuum. It happened in a massive multi-agent system, like as follows. So mapping to today's projects or work, this is what we commonly now call pre-training. This is post-training. And I don't think there's a massive amount of motivation work, although there is more and more, of course, as we get more mature in the technology for LLMs. But even at the highest level of how the different steps of creating this agent, we already could see, or we already can see that many of the problems we faced back then apply today. And you will see why in more detail now for each of these stages.
So let's start with actually something that at the time felt kind of complicated and also, in a way, obvious in hindsight, having worked on sequence models, which is, how to deal with actions, the actions that you needed to issue to activate or to operate StarCraft. So the actions were fixed. There was an API. And as an API, you basically had all sorts of different functions that you could call. And different functions required different arguments. So this is basically an API that Blizzard has and had before we started working with them already to interact or to send network messages, actually, between players in StarCraft. And we just used it as is. So you could imagine in this instance, there's this thing here that blocks this ramp. So there's an action called forcefield, which is the name of this thing. And you issue this action to this set of units. So you're saying, hey, set of units, try to issue a forcefield in these coordinates, 6, 82. And then you also say, delay from current. You can wait 10 steps, which is kind of half a second or something. And only repeat this action once. So that's how you do everything in the game. And it was a bit tricky to how would we be able to issue these commands.
So the solution we found is to just lay out an autoregressive model, a kind of a little recurrent neural network that was structured so that all these arguments were lining up with different softmaxes, that would basically say, Q, yes or no, when. And then it's a softmax between 0 and 128 steps or whatever maximum we decided, what is also softmax, a categorical softmax, over positions, et cetera, et cetera. So we handcrafted this architecture. And then we obviously learned both imitation and reinforcement learning to finesse how to issue these sequence of instructions or commands that form an API call that then we would send to the game. And this is strikingly close to tool use in general. Because in tool use in LLMs, what we have is an access to an API. There's MCP. There's so many different protocols. But essentially, there's an API. There's a way to define how to call a search engine or issue a command on a terminal, or click on the screen for that matter, all these things. And then this is defined now in natural language. But actually, lots of different problems and solutions, again, align here. And it was cool too.
At the time, I mentioned when I talk about AlphaStar, AlphaStar is like a language. You just need to issue these, literally, code calls or API calls into the game. And there are some differences that are notable as well. Here, this architecture is setting the API. So the aggressive actions are expressed through this particular architecture. As cool as it is, it's fixed. Whereas today, we know that to issue API calls using the power of generalization of text or code or combinations allows us to issue API calls to all sorts of different environments that agents might be connected to. We also have now a single transformer architecture that just writes these text or code for us that then is interpreted and sent to the environment. And then there's actually a lot of flexibility even in the formats because you can instruct the models to change the formats if you want to be empirical about which formats might be best, versus here, where there's a certain architectural rigidity then. So of course, we did ablate many decisions. And we had to retrain the models and then try which ones play the game better. But it was much more cumbersome if we need to retrain.
Now another very big problem, and I think this is actually a key debate, even today. It's still pretty hot between many folks, which are, the issue of using imitation learning, which we commonly denote as pre-training in the LLM world. So in StarCraft, unlike, let's say, Go or Atari, where the action space, however large, was literally not as large as it is in StarCraft, where it's very combinatorial, and also the time horizon was much longer, et cetera, et cetera, exploration is and was very challenging. And I'll demonstrate that kind of pretty sharply in the next video. So the problem is, random policy will not do much in this game that's useful. And so in a way, the bitter lesson discussion, which is, should you gather as much imitation learning experience to bootstrap the knowledge about the domain even in a game like StarCraft, which arguably, is not AGI complete, as complex as it is, didn't work to the best of our abilities. So for us, it was very clear we would need an imitation learning plus then an experience learning, reinforcement learning, loop on top. And that was kind of a nice merging of deep learning and reinforcement learning, which I very much welcome also in LLMs, although it can be debated whether, ultimately, pure reinforcement learning, for example, could get us all the way there. I strongly believe that answer is no, as you can probably guess.
So what happens when you apply pure reinforcement learning is that it actually discovers how to beat the enemy but in a way that is unsatisfactory. It takes all your workers. And then it just sends them to attack. This is not an uncommon technique. I mean, if you're a newbie, you might do it. Some professional gamers do it at some point with aliens, especially that they also bring all the workers to attack. This one beat the hardest AI in the game by exploiting it, basically with something that was not pre-programmed to defend very well against this kind of attack. But clearly, we just kept seeing these results. And we had to resort to essentially using human data, which luckily, we did. We open-sourced over 1 million games of StarCraft, thanks to Blizzard collecting anonymous replays online. They're still available today, hopefully. And there's a good cool way to do research on imitation learning. But basically, getting lots of human data online was felt necessary and was part of the solution.
And then something else we did which also, at the time, felt very different. But besides saying, hey, imitate every human move from these millions of games, we also said, look, hold on a second, from the whole game, we know what happened. We know how good the players were. We have some metadata. We know what the players eventually built, what units they did. So these metadata, we decided to encode it. It's a bit of a cheating information because, as you start the game, you don't know the future units that will appear, et cetera. But it was sort of a guiding signal to condition the neural network to look at these for correlations. So for instance, the neural net would, OK, I need to do-- this kind of unit will appear. It would learn the correlation between these units and then how to build it even in imitation learning setting, even though the correlations are fairly weak. And so that was kind of a powerful thing to do for many reasons. And I'll go get back to these conditioning signals again as well during the lecture.
But again, contrasting this with current LLMs, this actually was a form of prompt or instruction following. So we all start a model, and we tell it what we want it to do. And that was our way, without obviously knowing about this yet, of doing the same to a fairly complex system. We would tell this agent to either do this sequence of buildings and/or to construct these units and/or to behave like a professional gamer or a novice in advance. That was our instruction. And then by doing this, the agent was reconfigurable to play in many different styles, controlled by the user, prompted by the user. Now that's a similarity, many differences. I mean, this agent was trained on about 5 billion tokens. We know, of course, LLMs trained on over 1 trillion tokens. So the order of magnitude is quite different. These prompts were structured, were again hand-tuned to the style of vectors, binary vectors that we expected, rather than expressed through language. It was not that we were writing, hey, build an overlord and then build a. These were binary encoded, so to speak. And then these prompts, unlike in today's LLMs, were derived by just looking at the whole game. We could look at the future. We saw the whole game. We summarized it into these vectors that were given from the first frame of the game versus, of course, in current LLMs, oftentimes, we have a prompt or an instruction that requires human annotation in some form. Not always, but it's generally, hey, there's coding competition. Someone comes up with a nice coding competition problem with some instructions. And then we let the agents try to solve for that particular coding problem, for example, or a theorem proved by LLMs, et cetera. So again, some similarities, some differences here as well. But with these, we actually went from that state of degeneracy of RL to actually being able to start to see believable game states being achieved by these agents that they didn't experience the game, but they imitated the human knowledge, same as what we see with LLMs before we do any post-training step.
Now on to, I think, a very exciting area of progress, and I would say probably not the most mature even in LLMs, despite all the investment, which is post-training or using reinforcement learning. So in StarCraft, we did a few things, all of which, actually, we also see in some form or another, done for similar reasons, to be honest, in current large language models. So here is a fairly imperfect way to measure Elo scores. You might be now fairly familiar with Elo scores, thanks to LMCs, for example, which scores, of course, your favorite LLM providers against each other. But of course, I mean, chess and many other games use Elo as a system. And think of things like 40 Elo differences, not super large, but it starts to be significant. 400 is fairly significant difference. So if we train from scratch without human data, we didn't get a very strong agent. Just the supervised learning policy was at 900 Elo. So that's a fairly strong initial state to then kickstart the whole reinforcement learning multi-agent system that we had.
Now, the obvious thing is that this policy, initialize it, and then add, just use reinforcement learning vanilla, and you get a bit of a boost. But what tends to happen is, you might see some collapse of behaviors, which also is quite commonly seen in LLMs. So the diversity of the whole distribution that you've gathered over imitation learning is lost, so to speak. So we added a supervised KL penalty, which is a fancy way to say, we distill the policy, the original policy, onto the one we were training with reinforcement learning. So in a way, you don't overcommit too early to a certain particular way of playing. You keep the diversity of styles and strategies, et cetera, although that itself was not easy to do. So we'll get back to that in a second. But that helped quite a bit keep things much better. So that boosted Elo by 400. And then the last thing, which is this conditioning, explicit conditioning, on the kinds of things we want the agents to do, further boost the Elo score by almost 150 or so. So all these aspects came together. And we see obviously all these being employed as well in LLMs.
Another trick we find, which was neat, is that for the value function, which tells you whether you're winning or losing, even though it's a partial observable game, we found that the value function seeing the whole board, so to speak, was much more accurate and helped reducing variance in training, which was problematic and always using reinforcement learning. So that's another trick we did, which also, in different shapes and forms, is seen as positive when you apply it to certain domains in current large language models. So all these tricks and more are used in modern LLMs. But what are the differences between what we were doing here and LLMs? I mean, most crucially, that rewards can't be fully specified always, means StarCraft, you win, you lose. There are interesting things. StarCraft, actually, many people resign before losing. So it's not that all the data, you will get to the winning condition necessarily. So there are some fun problems there. But by and large, you can compute success or not success. Whereas, of course in LLMs, I mean, write me a nice poem, it's just good luck, getting humans to agree on that. So it's very, very tricky to reward behaviors when you're going to much more general problems.
Now, that's kind of annoying, definitely to me, which is, compute envelope for LLMs are heavily skewed towards pre-training. So you spend most of our flops in pre-training and only a little bit in post-training. And that feels wrong for a variety of reasons. And certainly for AlphaStar, it was not an exception. Most of the compute, most of the weights updates were done in these experiential or reinforcement learning stage. And then overfitting is very common in LLMs because, it turns out, we do not have an infinite amount of tasks that are interesting for humans to solve or to perform that would help train LLMs when they deploy it against real humans or with real humans in the loop, the kinds of things they would ask. There's a limited amount of data. Whereas, for StarCraft, every game is unique. And because it's constrained by the environment, you don't see this overfitting issue, which of course is related. Second and third bullet points here are quite related. So let me take a sip of my tea. And let's get into slightly more technical, more intuitive part of the talk about why do we need multi-agent systems. What did we do in AlphaStar?
So this was quite fun at the time. Certainly, I enjoyed, of course, the imitation learning. Sequence-to-sequence was an example of imitation learning very heavily. Reinforcement learning, I kind of, hey, I know what's going on. As a scientist, you learn you learn that domain as you join DeepMind. So it was kind of everywhere. Multi-agent, all that I learned was during the project. And it was very fun and very intuitive and very simple to see results, very drastic results very quickly, just by trying, sometimes, extremely simple ideas. So why did we want multi-agent learning in AlphaStar, was to add robustness. So hopefully, you'll get what that means through my explanations. So you have these models. They do amazing things. But we see them fail often. And that happened certainly in AlphaStar. And we see this happen in LLMs all the time as well. You ask simple questions, like, is this number larger than this number, how many Rs are in strawberry. And these models tend to flop quite dramatically, even though they might be, I mean, fuels medal levels in some aspects of mathematics. They might fail miserably in other aspects. So notionally, that is very similar to the kinds of issues we encounter.
We train a very good policy with imitation learning and reinforcement learning. We then play the game as humans, not in the self-play way that you keep getting better Elo scores. And then we found that simple strategies made the agent-- it didn't react properly. And they were fairly simple. If you played at the high level, it would really react to your moves and feel super human. But it would not understand some basics. So it very much feels similar too. Although of course, we're doing a lot of advancements in our LLMs, but it felt similar. And using game nomenclature, although you appreciate this is similar to the failures you're seeing. We have different policies or different examples that we could get against our agents from-- some are very exploitative, not that interesting, to actual strategies that we would certainly encounter if we had this agent online in the wild. So adversarial policies, these would be, like, making the agent flip. It's almost like adversarial examples. In neural nets, you change a pixel a little bit, and a panda becomes an airplane, according to a neural net. Similar things you can get with a policy, where maybe you bring a unit, and you move it in certain ways, and the neural network just degenerates. So that's unlikely to be found unless you let, of course, users probe it. But it's still the most basic exploit you could think about from a neural network perspective.
Then we have exploiters that are just repeatedly playing properly, so to speak, against a single policy. But eventually, they might find a weakness. And then they go deep into that weakness, strategically speaking, up until they just basically beat you all the time. And that's, in a way, similar. Both exploiter and the next level are similar to what we're seeing in LLMs these days. The third level is cheese, which is a strategy that might generalize, might be actually fairly robust against many different opponents versus the second one, which is against just one opponent. But of course, it's not probably the best optimal strategy for this game, although mix and matching cheese and normal policies is commonly done at the professional level of StarCraft as well because you always need to be worried that this strategy might occur at any time. You're not seeing the board all the time. So to play the game optimally, similar to, of course, things like you see in poker, you always need to be worried about many things. So you cannot go too far into optimizing against an opponent that, even though you're not seeing what they're doing, you don't know they're doing, so unpredictability is important. So the third kind, the cheeses, you usually see them at even the highest level. And then there's normal policies and optimal policies.
Now, putting it differently, this is called non-transitiveness. What this means is, would this policy work against a different opponent? So the adversarial thing, most likely not. The adversarial attack might be specific to the weights of your neural network. Then the exploiter policy, you train or you repeatedly play against the same opponent, who cannot update their weights. So that one is specific to one opponent. Cheese tends to generalize. So maybe when we're talking about the R's in strawberry and so on, we see many models, trained not very differently but somewhat differently by different companies, have these weaknesses. So maybe, those kind of exploits are more in this cheese category, which are reasonably generalizable but, of course, not very kind of interesting or strong. I mean, no one, hopefully, just uses these to cheese the model, so to speak. Then we have normal policies and then the optimal policy, which, of course, is very, very hard to find with deep reinforcement learning and multi-agent.
So if we put this figure differently, one observation that my colleague did is that most interesting games have these different axes. The z-axis here is the strength, the inherent strength of the player, the Elo score, or the z-score. But then within a slice of this thing, you would actually find different cycles of non-transitivity, meaning that three models at the same Elo score, especially if the Elo score is not very high, might beat each other in a way that's non-transitive. Rock, paper, scissors cycles, actually, that's a commonly used abstraction that's very clear. Rock, paper, scissors, one beats the other. So think of, we might have three models, three agents, at the same level, but one very strongly beats the next, and the next one very strongly beats the next one, and so on. So there's these cycles of non-transitive and many interesting games studied here possess this property. And this is obviously designed for zero-sum games. So for LLMs, you need to make some abstractions to think about what that means. But we certainly have Elo scores. And we certainly see models having strengths that are not uniformly the same across the same, let's say, level of LMC's performance, for example. So that is an interesting aspect of the complexity of the kind of task we're trying to solve.
I'll skip this slide. Just there's the reference from 2020. But that explains this intuitively, this issue of, there's an area where you're not so good, that things are very non-transitive. And then as you become very, very good, then there's a dominant strategy that no one can beat. So that's how we're seeing in most games, this kind of behavior. Now switching gears a bit, how do you actually train these models? There's a few techniques. But all of them have some issues, especially as the games become strategically complex. So here is rock, paper, scissors, actually. So each of the vertices here are rock, paper, or scissors. A strategy is just a distribution of what will you play. So if you play against an opponent that always plays rock, well, that's a very bad strategy. And the optimal strategy is uniform. So if you do self-play kind of naively, and you try to say, I'm going to learn an agent that plays rock, paper, scissors, you see this effect of oscillating around the middle, and locally, you're very strongly into one strategy. So sadly, that makes convergence very hard.
Now, what you can start thinking about is, hey, let's add population, populations of policies all playing against each other. So there's several ways to do this. Here are the three figures. But these are starting to feel a bit better, intuitively as well. If I want to learn proactively rock, paper, scissors, I'm exposed to many players, maybe someone who has very suboptimal strategy playing only scissors, but at least, I wouldn't know that player a priori. So I might learn, eventually, that I have to play the optimal or the Nash here, which is the middle point, equal distribution of the symbols. Very simple game, though, so obviously very toy. And different strategies yield different results. The one that is most hopeful, but even in this toy example, it doesn't quite get always the result that is desired, is the rightmost, which is, you have a population of these agents that you play against, and you play always against the strongest opponent. That feels like a good strategy to maybe set yourself, your whole system up. And we didn't quite do this in StarCraft. But there's a lot of these intuitions that played a big role.
Let's see this played. We saw this. This is a real example of what we saw as we trained using self-play in AlphaStar in the early days. And let's use a equivalent of rock, paper, scissors in StarCraft world, which is three different kinds of units-- void ray, which is an air unit; immortal, which is a very strong ground unit that cannot attack the air; and then stalker, which is kind of a unit that can attack the air but is not the strongest ground unit. So void ray beats immortal, immortal beats stalker, stalker beats void ray. And it's kind of true, if you do it just naively, you just build lots of these units. You most likely will see this. This actually represents the game reasonably faithfully, so great. We set this. And we start from imitation learning. So we have that players-- I mean, there's a distribution over different strategies that people will have done when they play online. And our initial policy is in the middle. It's not very good at either of these. But it represents the whole distribution. And as we trained, what happened is, our policies, even with the distillation losses and so on, started to only do void rays really well, lots of them. Very good micromanagement, but they became really, really, really specialized. So the whole population, which was not really a population, this is just pure self-play actually, the only strategy that became present during training was void rays. And of course, that washed out a little bit the prior knowledge that we came in, even with distillation. And that meant that from the agent's perspective, it was delusional. It was playing against itself, void ray versus void ray, great battles, awesome games, that we recorded during that time. If the enemy happened to do immortals, no issue. These units dominate this. But if someone just said, OK, I'm going to do these other units, then these agents didn't even encounter these anymore because all it saw in its games were massive amounts of void rays. So sadly, this was not a very robust strategy. I mean, if you force a pro player to, hey, you have to do void rays, only do that, no cheese, no different strategies, then this agent was very good. But of course, people are diversifying what they do. These are not the only three strategies. There's early rushes, all sorts of things that happen in these domains. So that really didn't work well.
What we added, though, the key notion in the league is to add exploiters. So as you train the model, the main agent, this steal agent, it might still specialize to the void rays. But then at some point during the training process, like the reinforcement learning, you're going to add a second agent. And this agent is going to be an exploiter, meaning that it's only going to care about beating this agent here. Its goal is going to be, beat this agent. This agent cannot train against this. So there's an asymmetry. The red gets to see the games and learn from them. But the teal does not. So then what happens naturally is that if the agent starts to specialize to void rays, naturally, the exploiter will start doing stalkers. So now, at least this agent has seen that, hey, there are very good players out there that will do this other unit that beats me. So it's going to be exposed to this strategy and be able to react properly and even say, run out of the optimum, which is to build lots of void rays, to say, hey, maybe I actually should also build immortals if I see that the enemy is doing lots of stalkers, et cetera. So this is simplistic. But I think it gets the idea of the algorithm we end up developing intuitively.
In full, what we do is this in steroids. So we had many different agents that were created with the sole goal of making the main agents, the agents we actually deployed and became grandmaster against the global population of players. So you can imagine we have these three main agents, one per race. And then we have different agents whose goal is just to make the main agents stronger. So that's the whole multi-agent system we designed. And the way it works is actually super simple. We didn't experiment a lot with this. So the main agents are essentially playing against themselves, playing against the other main agents, and playing against the whole population all the time. They're playing against everyone. I'll get to how that distribution of everyone evolves over time. But then these main exploiter agents, they will basically say, hey, start from the prior, try to beat, let's say, the three agents maximally, so just cheat, cheat in the sense of, I can train against you, you cannot train against me. Develop a policy that eventually reaches a certain win rate. When the win rate is very high, meaning I exploited you successfully, only then you say, hey, here I am, I'm beating you. I'm a new player in this world. Deal with it. So this agent was secretly training until it became good at exploiting. And when it became good, it dropped a checkpoint. It showed itself against the main agents, which now, all of a sudden, could play against it. And it went, and it restarted the same process again from the prior. So over time, you will find different exploiters at different stages of training against the main agent. And then we also added league exploiters, which is the same concept. But instead of beating only the main agents, these agents try to beat everyone in the population by finding exploits that work against everyone, rather than just the main agent, which is a bit more like cheese. So we have main exploiters, league exploiters, and then, of course, main agents.
This was very fun to see, like, the dynamics, the learning dynamics, very hard to understand even in a game, what's actually going on. I mean, intuitively, we know. But lots to do that are left on the table from these AlphaStar league, as we call it. But it was very powerful. And then over time, we have over 1,000 agents that have different strengths. They will beat a previous agent. But eventually, the main agents catch up and beat the exploiters. So there's all sorts of interesting win-rate dynamics. And it can be represented in this shape of the spinning top, a little bit. So the weakest agents here are highly non-transitive. There are 30,000 RPS cycles. That means the cycle length of non-transitivity is 30,000. So you can find 30,000 agents-- I believe that's what it means-- that beat the next one, the next one, and so on and so forth. And as the agents become stronger, they start to fully dominate, which is also commonly seen, I mean, in these interesting games. So that's an analysis. This is what happens in rock, paper, scissors to give you an example, so much more complex game dynamics. And then, as I said, it was very fun to see this happen. So you would be training and, all of a sudden, this red agent is an exploiter that finds a cheese that's called cannon rush. So the blue agent, that's the main agent, has not seen this strategy enough so it can't really define very well. But eventually, by now playing against this red agent that's now frozen in time and it will just do the same strategy, eventually will be able to defend and so on and so forth, so with workers, which is the right reaction, and scout to find that you've been, indeed, in this situation and so on and so forth. And we saw all sorts of interesting dynamics as well. Anyways, there's lots of cool games and, hopefully, analysis that you can see online. I don't want to talk about StarCraft too much. But it's a passion of mine, as you can see. But yeah, this was kind of very critical.
And then in the results, you could see that if you only train with the population still, but only of main agents-- so just have three agents playing against each other and against themselves in the past, that's called fictitious self-play-- you achieve certain strength. And this core is not that perfect. If you play as a human, you would not be that strong. I mean, these differences were probably less than reality. And we checked these a little bit. But it was hard to do these experiments in the wild. Then by adding main exploiters, we became much stronger. And by adding, on top of that, league exploiters, we had the final system that was robust against many different strategies and, of course, very strong at playing the game.
Now, a couple of details and then I'll conclude, is that the path of learning matters. And it's so hard to know how it matters. Basically, if you are now an agent and is exploited very often or very rarely or initially is never exploited, I mean, all these changes, happily, the policy you end up with at the end of this process. So there's lots of details that are to be worked out, even from a theory perspective. But in these games, and with deep reinforcement learning, theory is also hard to come by. And we had a couple of strategies on how to matchmake. So I'm the main agent. There is literally 1,000 agents. Which should I play against? So we had two strategies. One is, play against the one that beats you the most, which is hard opponents. And the other one is, play against the one that you're 50/50. You're closest to maximum entropy. Because if the one that beats you the most beats you all the time, there's not going to be a learning signal. So we tried this, both strategies. Variance was a little better, like this one that maximizes, kind of, that tries to get to the sweet spot of 50/50. But also, I mean, this is one of the many parameters that tells you that the process in which you create these main policies matters a whole lot.
And then the other part that was important is that to generate interesting counterstrategies or exploiters or cheeses, having controllability mattered. So not only we started from the prior distribution that we had trained in imitation learning and say, hey, go and exploit the main agent, we also were opinionated about how you should exploit it. So we would sometimes, not always, say, I want you to exploit the main agent, but please do it with this unit type. Like, I mentioned how we did this prompting. So it was kind of tying your hands behind a little bit, saying, hey, I want you to beat me, but you have to use this unit. And that added a component of robustness and diversification that also help. So lots of different choices here, not a lot of time we had during the project to fully study these. But you can see lots of different possibilities. And it was very important to have this diversity in the main policy. Of course, in LLMs, that is trivial. So they're very profitable. There's a lot of diversity. So there's not a problem at all to think about, hey, I don't know, solve this theorem but do it in Catalan or in Spanish or do it in very bare bones. All these things will add some sort of robustness to, maybe, new encounters that you'll find, meaning new prompts in LLMs and so on and so forth.
So contrasting is very exciting, multi-agent ideas and space with LLMs. I mean, one difference is that users are, in general, collaborative. It's not a zero-sum game that we play. I go online, and I want to achieve something, get some information, help me with some problem that I'm having. So I'm actually collaborative with the other agent, right? So that's an interesting difference. That being said, we also have adversarial users. I mean, who doesn't like to find weird failure modes? And then Twitter likes are, obviously, one of the reward signals that humans like to optimize. And also more seriously, eventually, these systems become closer and closer to AGI, very powerful. So you might want to-- them for doing something that they shouldn't be doing. So there's going to be adversariality as well, trying to go around how the system is designed to be used. So both aspects could use multi-agent techniques, such as the one I described from the AlphaStar league, differently. Complexity is massive compared to even a game as complex as StarCraft. So there's no clear reward. We discussed this. Prompting is much less clean than bits of which units should you build. Prompting can be vast. It's natural language. It's unbounded. It's very, very interesting, the amount of complexity we'll deal with here. To the extent that even these LLMs playing StarCraft, which is a dream of mine-- They can play. They're not very good-- is actually one of the many domains. So you could see a particular case of current models is that we might just instruct them to, hey, be a very good StarCraft player, and then give them control of the environment and see what they do. And in fact, games, more and more, are starting to be reused, turned around, be used as an evaluation. There's a Game Arena, I believe, on Kaggle that is very popular, for example. And famously, I think some of the LLMs tried to compete Pokemon, et cetera, so very cool domain as well to test ideas. I will say, some amount of auto-red teaming, et cetera, has been using multi-agent techniques. But I think, overall, it's still in its infancy. So there's lots and lots of research to be done here.
So to conclude this sort of past and future combination and, of course, present of LLMs, I would say, there's lots pre-LLM agents. There's lots of gems to learn from from many different papers and different problems. Tool use was discussed. Architectures, losses, regularization, value functions, I mean, lots of things we can probably revisit very seriously, and we should. My strong belief is that human priors are going to be part of the path to AGI. But we need to obviously also build knowledge and experience on top of that prior knowledge. I've seen it succeed enough times now, including in AlphaGo, which also use imitation plus RL that, I believe, that's certainly the shortest path to AGI, although happy to discuss that if useful. Then in general, RL or post-training on top of LLMs still feels in their infancy when I think and compare it to a system like the AlphaStar system. We can train on thousands of steps, maybe, versus in this AlphaStar league, we use 10 to the 10 training steps in post-training, so vastly more compute used in post-training rather than in pre-training. Of course, we also have this issue that the rewards are fuzzily specified. So that is definitely a challenge. And last but not least, multi-agent RL, in whichever way you can think of creating a population of users and agents talking to one another and really optimize and train for that, I think that's very exciting. And it's going to have, probably, a new scalability breakthrough, I can imagine, in the next few years, maybe from some of you.
