Good morning. So, I'll switch to English. Um, it's a great pleasure to be here today, dressed a little bit differently than yesterday. If you saw, it was a nice experience, very honored by the Honoris Causa, and it's a pleasure to be back. It's not the first time nor the last that I'll hopefully tell you some story about what we're doing, where we're coming from, where we're going in the field of AI and AGI pursuit. This is a talk that I've given, but the field is moving really fast. So even if it's the same talk, it's really not the same talk. Like last week we released Gemini 3, Nano Banana Pro. So you must update almost constantly slides and presentations with benchmarks and things that are on the frontier or are now commonplace all the time in this field. So hopefully some of you who are closer appreciate that. Some of you who are a bit further apart know that it is daunting even for us, the speed of progress. So I appreciate the amount of vertigo I guess that you might have about the field, but it is what it is. There's a lot of investment, a lot of interest, and it's happening. We are building an alien intelligence. It feels alien sometimes given the amount of mistakes or the kinds of errors that it makes or the kinds of things that it does for us. But it is an extraordinary time to be alive. Demis said very early days when DeepMind was only a small startup that this was the equivalent of the Apollo mission for landing on the moon, basically the pursuit to understand us, our brain, our intelligence, sentience, etc. So it is very exciting. Anything you do probably this will affect you. This technology you'll need to adapt, you'll need to hopefully advance it, investigate it. So yeah, it's a great pleasure to be here to be able to talk about what we know today. As a tradition, I'll do a selfie first. So you'll get to see later probably I'll tweet this, I guess. What my point of view is from here, which is amazing. So I think everyone more or less is in. Great. I couldn't do that yesterday. I felt it was probably not adequate, but today I'm definitely doing it. Okay. So let's start. This class is, I mean, it's not really a class, but I'm going to try to touch something hopefully for all of you, as I model who is here, which I have no clue obviously, but hopefully someone gets something from any section of this talk. There is obviously AI, artificial intelligence, and then there's AGI, which is artificial general intelligence. There's a distinction that hopefully will be clear to most of you after. If not, at the end there's a very long section of Q&A which I'll probably eat into a little bit given that I generally don't prepare for time. And there's a few slides to go through, but we can skip a few as well. But the first and very important object is what is that we're trying to build, how do we define it. And this is a definition that is sort of the OG, the original definition, from I believe Shane Legg, who is a co-founder of Google DeepMind together with Demis and Mustafa. And it says an AGI is a machine that can do the kinds of cognitive tasks that people can typically do. It's quite a not very precise definition because if you think people can typically do, you'll say well people cannot typically win a gold medal at IMO for example, but then you could say well but we want the best human and we need to perform at that level right. So there's already lots of nuance on the level of intelligence that we're trying to achieve. Is it the median human? Is it from which country? From the whole? I mean it's quite underspecified definition. Cognitive task is the other very important element here. There's a definition below. Cognitive tasks are the undertakings that require a person to mentally process, blah blah blah. I think basically think of anything a human can do with a digital computer. There's a lot of obviously progress in robotics and so on, but mostly currently we have lots of progress on the digital domain. So think of anything that a human could do on a computer, a skilled human, you know, that's probably the kinds of task they're, this very broad task that we want to do. And I really like the part of acquire an organized knowledge to learn. I think that's a skill that especially in a university, students that I would say is the most important meta skill that came with me I guess from after undergraduate studies here and my doctorate in Berkeley. It's not so much about, I mean I was kind of joking with some of my colleagues I guess now yesterday about I don't remember much about rotations or divergences or so on. There are things I remember because I use them in my work, but what's important is the ability to learn, communicate, networking right. These places, universities, they might be transformed as well by the technology, but the human connection and those skills are definitely going to stick forever. Um, never say forever, but I guess they're going to stick for a while and it's going to be an interesting transformation to witness as well.
So the funny thing is even though there was the clear pursuit for AGI, especially when you see early days of the only company that was probably serious about it since 2009, which was DeepMind, even the achievements you might have heard and we'll review some of them, they were not really AGI, they were more narrow AI or AI. And so I'll first discuss a bit of the basics where we come from from maybe 2010 to 2020, where I call it the deep learning era. So what we were obsessed by at the time, it was a very academic era. We published a lot in conferences, gave many talks, and our obsession was certainly to train a neural network, generally or a model, but neural nets for those in deep learning is the bread and butter, that would essentially excel at each of the tasks that machine learning community have come up with over the years right. So there's a task of machine translation, hello hello how are you, I should probably change that to Catalan but anyways. Pixels right, like from an image to the class of the image, that's the ImageNet classification challenge that was built for advancing machine learning, speech recognition, etc. So we had this single task and our goal was to maximize a metric that was defined by the community. And interestingly even at the time we also start, especially in 2014, 15 and so on, starting to reverse to the modality that's on the right, that's mostly discrete text or labels, it's very low information density, to reverse that to from that low information density to the more rich signals which is especially audio, images, video, which tend to be very underspecified right. If I showed you, if I ask all of you hopefully you know okay that's a layer part, so there would be no uncertainty going from left to right. But if I asked you to draw or imagine a layer part, an image of a layer part, probably all the images would be all different because the entropy of pixels and the natural world is much more rich than that sort of process of distilling the natural world into language, which is a very human intelligence-like thing. And of course language we'll come back a bit later. So this was also an interesting problem. We did a bunch of progress of course here, and then we had as I said a few landmark projects generally anything that starts with Alpha from DeepMind. Pay attention, I mean this is a project that has been devised to be drastic, just multi-year, very ambitious, etc. So AlphaGo is a system that basically has a lot of interesting ideas that even today for the current chatbots that we have apply, which had the idea of training a policy network that is just a neural network that given the state of the board of a game. So the input is kind of an image, is 19 by 19, that's the full game of Go. So it's a fairly low resolution image if you think about it as an image. And we had the idea to first train to imitate human moves, right? So you have a large data set of how people played Go, like everyone, I mean chess probably is more common, but you get all the games ever played recorded and you can just replay them. And through that data knowledge that exists out there, we distill that knowledge into a policy that was able to basically try to predict the probability of playing a move from a certain position in any place in the board. And that was part number one. The other part very exciting was from any position on the board, can we predict who will win this game? It's a fairly challenging problem. In chess, you can say, well, okay, one side doesn't have the queen, right? So that's probably pretty bad. Even in chess it's a tricky thing to do to evaluate the value or how ahead one of the two players are. But that's another important element because once you have these two elements, you can do tree search but guided by probabilities and estimates of winning probability that are given by extremely powerful models, neural networks trained exquisitely to the level of then becoming superhuman. And then another very important aspect was the aspect of self-play. So once you train your prior knowledge from human data, what you kind of start thinking about is can we get the system that can play Go at certain level to play against itself and then reward the side that wins and penalize the side that loses, and then try to learn from that cycle that basically can go forever. And that was another very important insight used in the AlphaGo breakthrough that you would then see eventually on. It was streamed fairly massively over 100 million views in the game against Lee Sedol, and AlphaGo did fairly well there.
I go a bit quickly through this, but this is a project close to heart. I like the game StarCraft. It came back over and over throughout my life, even recently because somehow I promised Elon Musk I would play his system on StarCraft three days ago and he accepted. So I'm on the hook to maybe learn to play again a little bit. I'm a bit rusty, but this project developed a lot of the techniques that actually apply today almost verbatim into the creation of Gemini models and all the other chatbots out there, which is the idea of doing pre-training human imitation, but that is on a sequence of moves which starts to sound more like sequence of words, language, etc. And then it does use that initial policy to then fine-tune with an RL, reinforcement learning loop. It's not that dissimilar from AlphaGo, but the imitation aspect here was critical because the space of moves in, unlike in Go which is large but 19 by 19 if you're very naive, the space of moves here is much larger, much like every sentence, every English sentence if you think of that space is a very combinatorial, I mean large space, infinite space, and it's similar here. So we had to do a few tricks that end up being also quite useful in language modeling.
Then of course AlphaFold. I discussed this briefly yesterday. This was another application again of a single system that could do something very important. Now this system did something so important that obviously was recognized last year with a Nobel Prize. And the idea though is not that dissimilar from the classic machine learning paradigm. You have a bunch of data. You want to learn a mapping between inputs and outputs. The mapping needs to be powerful, nonlinear, and learnable differentiable. Neural nets always come handy when that is the requirement. And then there was some unique challenges because the amount of data here was not as large as we had in Go or in StarCraft. There's only a handful few thousand, tens of thousands of proteins that have been folded to the precision that we want to achieve or imitate, just because as I mentioned it takes a whole PhD to fold a single protein. So it's a fairly intricate problem, and also we had to invent a few techniques to distill from unlabelled data etc. But eventually by finessing every single aspect of this system, we climbed from 30% accuracy over 100 to over 90, which then the community considered that within the error of doing this in the lab by crystallization or any other process that allows you to fold proteins more slowly. And of course then we published eventually all the 200 million known proteins. We just process them through the system and we gave that as a data set for the world to use, and the model is accessible for any researcher and it's been used worldwide. And it's actually just five years ago I believe yesterday, so it's a nice moment to show this.
And maybe why this matters. I mean this goes into biology, but that's a bit of the fun of being in machine learning and then machine learning kind of interdisciplinary, that's a hard word, goes into all the other disciplines right. So the why this matters for real, not only as a hey this is a grand challenge and we solved it, is twofold, no pun intended. One is to determine these letters on the left. The input is cheap, is easy. You can sequence proteins fairly easily. When COVID came, sadly then we got this almost immediately, right? It's a fairly cheap process to do that. To otherwise crystallize as I said is very, very challenging. So that is very expensive in time. This is cheap. So of course what AlphaFold does on the one hand is do the cheap thing, apply the model, the transformation, and then you get the thing that was going to take you years in basically seconds or minutes. The other question is why does that thing matter at all right? I mean who cares? I mean who cares how, yeah corona virus okay might look like this or that. Well it matters for many applications of biology. The one and I am not an expert so apologies if you know more than me, which is very likely, but to explain it plainly, you have this let's say a virus that has certain proteins and it has some shapes. A medicine, the goal of, and this virus attaches to your body through basically it's kind of a puzzle and it's evolution of course right. So there are certain proteins that attach to parts of your proteins that you own in your body and then that's one aspect let's say of getting infected. Medicine, one way to solve this problem is can we attach a piece to the virus so that it makes it so that it doesn't attach so easily to you right. So that's one way to then make the virality go down. And I mean many other things that I have no clue about, but this one I sort of understand. So intuitively it's a big deal to know the shapes of the things. So we can do a bit of a puzzle, puzzling kind of attach to this, don't attach to that and so on. Of course proteins might have side effects. So it's not as simple as I said, but this is a simplification of why this is a big deal.
So to recap this sort of deep learning era, these are like three demonstrations of something that I call many AI demonstrations. They focus on models. The way the system is a set of, is a model that you can then open source, but it excels only on one task right. The Go system cannot even play chess, which is deemed less complex mathematically speaking. The thing that solved proteins would not be able to solve a three-year-old puzzle etc. So these systems are narrow. And I also mentioned this in my talk yesterday, that was that. But what then, you know, when people at the time, you talk to the press and so on, you try to explain why this matters, and then the methodologies are general right. So what at the time we were doing is look, the applications are important, the demonstrations are impactful and amazing. But how we're doing it, the how was what was general right. So the what we were developing in parallel for all these, there's a lot of commonality. There's neural networks, transformers, stochastic gradient descent, etc. These are general, but the models are not general right. So we were not really in the models claiming that the models are AGI, but technologically we were discovering pieces of the puzzle so to speak, to build eventually hopefully a model that we would claim is general and powerful.
Now, I'd like to zoom in a little bit onto some of these key building blocks in some detail, just because they are close to heart in many ways. The basic one is embarrassingly simple. I mean that's probably the math subject I remember the most as a result of applying basically nonlinear optimization day and night. That's how we learn the weights right. So a neural network is a differentiable function. You can use the chain rule, backpropagation, which obviously was not invented. This is a mathematical property of chaining different functions that is known for many decades, but there was an algorithm, an actual implementation making this memory efficient. And once you set up these networks and the weights are randomly initialized, you then run an optimization process to learn these weights, to make the weights fit the data as best as possible. It's classic optimization. And these objects here happen to be quite powerful, quite general, and have the right inductive biases, which is a notion that is important to understand because even if they are universal functions, they might take more time to learn or to extract patterns from the data more quickly. And then you run an optimization process by gradient descent, many different variants and tricks, many of which are empirical I would say. Like you just look at the math, it probably doesn't make sense, these things are non-convex, so it's a bit hard to study. And that's why they were fully dismissed in the '90s as not an interesting mathematical object to study from an optimization point of view. So lots of resistance, but this is the key element. The brain looks like this, each of these arrows is a weight. And to give you an idea, the current models you might be experiencing once you go to whatever chatbot and you type, have about let's say one trillion weights. And a trillion here means a million million. So imagine if you want to visualize the thing we train, the object we train is a trillion dimensional real vector, real numbers. And it looks clearly much more complex than this toy version here.
Now another important work is that how do we model sequences. They're variable length, they were very annoying for various reasons for neural nets. So we came up with a very simple idea back in 2014 with Ilya Sutskever, and that was the basis of internally at Google and certainly also externally to start to apply language modeling to more and more domains that require producing sequences of words or bytes. An image can be seen of course as a sequence as well. And there were certainly many more works applying neural nets to language. This is the one obviously a bit biased, but the one that impacted probably internally us the most. So that was an important piece of work that exploded then internally and externally.
Another important paper that sadly was rejected. This is my best rejected paper I guess in some way. But it's okay. Jeff Hinton did a good job at going and giving talks about this despite the fact that obviously we didn't have an official spot at NeurIPS 2014 where it was rejected from. But distillation is an important technology that even today it helps because the main notion is that you can train an expensive model, a large model that might be not cheap to serve right. So we serve I mean I don't know quadrillions of tokens per some unit of time at Google. So every one of these computations needs these one trillion weights loaded and doing compute intensive task which requires energy, money etc. So what we tend to do is we train a model that is the most powerful we have internally but is not deployable at scale, and then that model can then be used to teach a small model to be as almost as good as the teacher. And the method that you use to do that is intuitively the following. When you train on a data set of text on the internet, you encounter sentences right that people write. So one sentence could be perform the concerto for violin. That's a sentence that might exist and it's written, and the model is going to try to predict the next word that happens to be in the corpus for that particular sentence. Right? So what you're going to do for training this is going to say, okay, given that I've seen performed the concerto for, the model is trying to predict which word comes next. And of course you don't know for sure which word is going to be next there. The corpus is telling you violin is the one I want you to produce for this particular training instance. So you're basically going all in right. The model might say, look, I don't know, there's these possible words. Then you do gradient descent and say, no, the word was violin. So you get penalized for all the different words that were not violin because that's not the correct word given this particular example. And what distillation does instead is wait a second, I know already a little bit about the world. The model that I train, the larger model has expertise on the kinds of sentences and words that go after which words. So do not predict just violin. I mean maybe boost that a little bit because it's actually in your data. But hey, the words might be there as a distribution. Make bets about which is the next word. So it could be viola, trumpet, probably not airplane. And then use that as an extra signal. So the gradient step that you're going to take has more information. And that kind of felt cool at the time, but now it matters because believe it or not we run out of data. Like we train on all of the internet, again trillions of whatever units, words, tokens etc that we used, everyone uses to train. So the game now becomes extracting more from the data that we have. It's the bottleneck for the next generation of models. So these techniques become also very exciting as a way to reinterpret in some ways the data that we have by the powerful models that we already have at our disposal. So very important rejected paper.
Other things that happen, again I'm being a bit more Google biased, but I guess it makes sense, especially given the stuff I'm wearing today. We invested, there's GPUs from Nvidia and TPUs that Google created many many years ago. Quite a lot of foresight to predict this would be a big deal to do. So we have large data centers with lots of interconnected hardware. Very interesting problem. Again, I'm not the hardware expert, but there's a lot of interesting things that happen. You have a chip that's very powerful. You need to bring in these weights that are massive into the computational units that takes a lot of energy. And then these models are very large, so they don't fit in memory. So you have to start architecting ways to communicate different bandwidth layers. The chip has lots of memory bandwidth because things are very close. Then groups of chips might be kind of closely location-wise. And then you can imagine a Google data center as large as, I mean I don't know actually I've never been, I should try to go, but let's say as large as a bunch of buildings here. And then of course the bandwidth and the latency there starts to affect your performance at training time. So this a lot of care, a lot of investment goes into infrastructure, energy consumption, cooling. It's a very interesting area and as a researcher you need to try to tell people that we'll build this in two to three years, what's going to happen with the architectures, which is not an easy job and it feels also high pressure given the cost involved, but it's part of what we do day and night to obviously have the engineers and the researchers communicate with data centers planning etc.
Now the last piece before I go to the next era is the exact formulation of the neural network. So the transformer paper, extremely influential paper. We tweaked this architecture but surprisingly there's not much variation that the current models use on top of the original formulation or architecture. And these happen to have the power of neural networks in terms of being universal mappers from any input to any output. But they are very good at learning very fast, very efficiently. So they learn very quickly. That generally was a problem at the time because they overfit to your training data. But they really like lots of training data, which obviously that happened when we start training at internet scale. And then there's another very natural inductive bias which is the concept that as you read text and you try to write the next word or the next sentences, you pay attention selectively to what is written before. Right? So the previous model, recurrent neural networks, tended to compress all the context into a single vector, whereas this was a very good inductive bias to extract information actively from your context and that helped a lot especially in language modeling. And also it happened to be very good property from an optimization point of view because once you do backpropagation over time you get basically many matrices multiplied together and you know there's issues with values that dominate etc. Whereas here the gradients with respect to every word that you've seen from a whole book happens to go through a constant number of backpropagation steps. So there's lots of properties here.
So we have powerful models. That's exciting. The next thing that probably is the most important thing that happened in the last let's say 10 years, and I mentioned it brings the generality, the G back into AGI, was language modeling. Maybe in hindsight it's obvious, but language modeling was this kind of field that even Shannon already was dreaming about right. There's this idea that you can probabilistically try to compress text and determine how many bits are in English for example, and compressing was the obsession. We all know what compression means. You take a file, you compress it, great. You send it through the internet. When internet was ultra slow, that was awesome. But compression was sort of the wrong angle. What happened over the years is that these compressors understood language so they could compress information to the bits that would be necessary to explain the next word. So in the violin case, maybe given that sentence there's only five words that come next and a word would necessitate maybe eight bits right. So the compression factor is massive if you understand the text and that was generated by a human. But you can also flip this around and start sampling from this probabilistic distribution you have, you have p of basically x1 through xn, and you can just use again a form of chain rule to sample from left to right words. And the words that come out tell you a little bit the model that these language models have created of the world. Primitive language models, they kind of make sense locally right, like I mean six billion dollars okay that's a set of three words that you might encounter somewhere. But then as we went from n-grams to neural networks to eventually transformers, this was a massive, massive improvement over the quality of the text that was produced. So at some point there's like wait a second, this is just not a compressor, this is a generator. And what is the usefulness of having a language generator? I mean you know the answer now, but at the time we were playing with different things. We generated translations, I mean that's certainly useful right, so we use it for a bunch of applications. And probably the main, I mean I mentioned the transformer so maybe I'll skip this, you know because timing probably is tight, but let me get to the GPT series of papers. So we have these generators that start to produce interesting text, and then GPT-2 was really in terms of the how, like how do we train these models. It established the whole way that this happens. Further are just refinements and honestly just scaling up in some instances. But GPT-2 had a modestly sized data set of 40 billion words. That's 40 million, 40,000 million words, and parameters is still in the billions, not trillions. So 10 to the 9, not 10 to the 12, and used transformers. But I think even probably be more than transformers is the fact that it was trained on all of the internet to model all the text ever produced by humanity as approximately correct as possible. And that created this magic generalization almost that you could feel already in the GPT-2 paper, where they train this model just to model language over all the internet, and they start to see that it started to perform well at these narrow tasks that I mentioned in the first section of the talk. It was fairly accurate and even state-of-the-art in one of these traditional benchmarks like machine translation or you know natural language parsing to like parse trees etc. So that was a profound realization that you could train a model to model the internet, that model could do everything kind of. And then of course this notion was and has been improved over the years through mostly, I mean I just put these two numbers because the parameters and the number of words that it's trained on are and have been quite telling in terms of the capability to expect, and I'm getting to that in a second. But of course there's also refinement on the recipe on how these models are trained that is quite important as well.
So the lips, you know some of us of course in the field, GPT-2 already exciting, GPT-3 I think felt more mainstream research, mainstream. And certainly between GPT-3 and GPT-4, that's when ChatGPT happens, an application that surfaces these models in the right form factor that makes it exciting for pretty much everyone in the world to experience. Theme park episodes are being built on top of the ChatGPT, so super important moment in 2022.
Now scaling has happened right. Empirically we can plot, hey, by when we publish models, how many flops or what's the compute intensity we used to train those weights, and we see sort of three different scenarios. Pre-deep learning, then deep learning which we studied, that's growing considerably to 1e20 flops to train a whole model from scratch. And then in the large scale era, in large language models, we've seen another shift in the curve above. And this is a bit outdated and this data is proprietary, like no one says exactly how many flops we use to train, but you can imagine from this graph make your own prediction and it's probably going to be fairly accurate. So why scaling up happened to be good? So we had this paper where we empirically discovered that there were emerging properties of these models trained on all of the internet as a function of scale or amount of flops you train it with. So let's take just the top left here for simplicity. If you train a model again on the same data, it's the same model, you just make it larger so that it's basically trained on more flops, and you ask to do arithmetic. I mean this is literally you ask a language model what's 4 plus 4. I mean these were fairly early days, the model might have no clue even what you're asking about or certainly not give you the correct answer. But you can measure that, so you measure that performance. And at some point the model, for no reason, you're just scaling it up, you're adding parameters to those little, because you just keep adding all those little balls. Eventually it goes from not having a clue on basically adding numbers to actually performing decently right. And it's a very sharp sort of increase. And that's sort of an interesting property because as researchers you say, well, if I kept scaling, could this model win a gold medal in maths? And I mean the answer is yes. But at this time we were studying it more empirically and we observed these issues. So obviously as the whole field was curious about if we kept scaling that number 24, 25, 26, what would be the thing that the smallest can do that couldn't do before right. A very important notion and why you see all the investment in compute to test these hypotheses. There's many other reasons to invest in compute, but this was certainly in the early days a scaling hypothesis that for us was very critical to explore as fast as possible.
So now we have general text models. Enter Gemini. I mentioned the ChatGPT moment. That ChatGPT moment was very interesting worldwide. Certainly AI was not, I mean AI was obviously in the public interest certainly since many years ago. But that moment definitely made a massive shift. And one of the shifts was that we had a bunch of groups, two groups mostly in Google running large language model projects, and after that moment it was clear we had to get all the talent together into a single project and also all the compute together to get those numbers further to the right faster. So different moments where the project started, but in February 2023, Jeff and I started sort of building this project to create the best models in the world. Google was still catching up. We had a lot of obviously invented a lot of the key technologies, but in terms of the weights produced by us independently as two entities, we were certainly not at the level of the best models. So the goal was clearly get together and over the years perfect the recipe and get to the top as fast as possible, because this is no longer an academic exercise. This model surfaced to the world as all of you have experienced, can do all sorts of amazing things, and we'll see some of those a bit later. So the project starts, we have a series of models, the three literally produced last week, so very recently. And I'll describe a little bit of certain properties of Gemini, the model that powers pretty much every large surface in Google, YouTube, Google Search, Gmail, Google Docs, Maps, etc. It's really went everywhere. So it's kind of the brain that is now powering lots of the billion user plus products that Google has created over the years. So it's a fascinating transformation to have witnessed.
So 3 was released just last week. Obviously you release this and you get a reaction from someone who releases something else. So you know you never claim state-of-the-art very strongly in the field. You definitely learned that, of what we've learned, fairly strongly. But this model, I mean 2.5 Pro was really good and in certain aspects definitely probably at the frontier, meaning the best. This model holistically also extremely powerful, improved on many aspects. It's much better at understanding text and images, both input and output for example. And then we have a lot of metrics. Like that's another interesting aspect, like how do we measure AGI is general. So we have a bunch of benchmarks that go from is this thing close to a gold medal mathematician, to can it code beautiful apps just by a prompt, to GPQA which is a kind of more like exam style general knowledge way of testing. There's a lot of academic and science tests here because there a bit we've tested obviously ourselves, we pitted ourselves against obviously exams all the time. So there's a bit of bias, academic bias here. But of course ultimately you have this and you just want the users to do and explore what these models can do, which is very fun part of my job as well. But there's lots of benchmarks that we measure to assess progress between you know current generation and past generation, like 3 to 2.5, and of course whatever is felt as the state of the art, which is Claude, Anthropic model, or GPT-5.1, OpenAI models. So this is obviously a very important aspect to develop models to assess how far we've gone from the previous ones. And then there's many other aspects. One is as I mentioned user. So there's this website called LMArena where users go prompt two models that the user doesn't know about, gets two answers and picks the one that it likes the best. And again for this particular release, we broke 1500 ELO, which is kind of one of the interesting but artificial boundaries that we were hoping to do with 3.0. So barely we broke it, and then it's very good according to users and held out sort of data compared to competition. But the margins also are not massive right. Like you have to understand ELO scores, a difference of 400 ELO means the probability of winning is 90% I believe, in it's developed for chess. So if you look at differences of 50 points, 30 points, it's barely like in a side by side it wins like 2-3% of the time above the other one. So it's not a massive difference. All these models are awesome in different ways, have different personas or personalities, lots of interesting things for you to experiment with as a user.
And then the last bit is text is important but different modalities are very powerful. We are very visual. We talk to each other. We move in the real world. So there's still a lot to unlock. So we've had some early work on how can we bring pixels into modeling text, and we had a couple of attempts, but the one that is closest to how we do it is actually this one where it's kind of the simplest way. You take an image and you just compress it into a sequence of symbols that try to compress the image. So JPEG does something similar, although we don't use JPEG, that would be not too good. But we just use this notion of compression again for an image to be represented as a sequence of numbers that can be decompressed to be the image, but that represent all the details, even the letters etc present in the image. We perfected of course how we encode an image to be then represented as a sequence of integers. And once we have the sequence of integers, we just inline it in the model and we train. Sometimes a data point will contain images, text, images again, and that is how you get this generalization of being able to interleave modalities in any way that the users creatively do indeed. So this is sort of how we added multimodality. And then since day one we wanted to be native into not only text but audio, images and video. And it's been a massive area of investment since 2023. And then on the output side, our focus was mostly on text, but we've also tweaked the way we generate images quite massively. So this is the thing I'm wearing. Actually, this is meant to be banana color somehow. Not sure which bananas, but we developed and perfected how could we also produce these images at the output, not just at the input.
Nano Banana Pro again topping the leaderboards from user preferences or capability to edit the image and so on. But this model, unlike most image models where you would just write some caption, you know, draw me a landscape and you could add more detail and you would get the image out. The usability improvement we had for the moment that we released Nano Banana earlier this year, and Pro is much better at rendering text especially, is that you could just also edit images, right? So I know probably some of you have used this. You can upload a couple of pictures and say put this little boy into this UPC campus which is the image I showed yesterday just produced by Nano Banana. So you can edit images with language very intuitively rather than needing specialized tools that are only available to specialists that have perfected skills on those tools. So that becomes a skill that is much more available to anyone that can be creative now with expressing combinations of images, editing and so on and so forth. Of course it changes how the professionals do it and we think about this all the time. We always show these tools to filmmakers etc. For them to understand what's coming and guide us a little bit in the process. But this model, and I mean how I did this slide is as simple as I took the results, a screenshot, and I said, put some bananas and make a nice title. So the letters actually are rendered from the model. These are not fonts you could edit. If I want to edit I would need to go and ask the model, can you change, maybe can you change this slide to Catalan? We could try it maybe offline if it should work. And then last modality is video. So we released VO and this is kind of the beginning but, and this video is a bit fast, but it generates pretty awesome, you know, realistic, unrealistic, creative videos. And again, we released this to the world and it's quite amazing. But I would say, you know, these 8-second clips, there's a lot of room for improvement in here, but you can see how you eventually might want to communicate with these models. And sometimes the right answer might be produced as a short video to explain something or to teach you something, etc. So, very exciting. And with all these advances, we had not just general text models, but general multimodal models.
Now, this capability is perhaps a bit more deep into what I think is more at the core of the definition of AGI. And the best way to describe it is these models are great. You can instruct them. They will follow your instructions most of the time. If you instruct them to do something not nice, they're meant to also push back and so on, right? So there's a lot of configurability in how you then program this, which is through language. But deeply they don't learn new things in the way we learn new things throughout years of attending university. Right? So imagine there's a new field or a new degree that the UPC creates and we record the first cohort. We record all the classes, we have all the materials, the homeworks. An awesome capability would be to show all this information to the model that has already been trained. We're not going to add this to the training set because it's data that's in the future. So how could we? But this data that now appears in the knowledge of human knowledge that's been created by new discoveries etc. It would be awesome to add that into the context or somehow instruct the model. Hey, read all these videos for hours and at the end I expect you to be an expert of this new field that someone here maybe has discovered a new field of mathematics for example. And that capability is not quite there yet but it touches the learning aspect of the AGI definition, like cognitive task that you can learn. And I think this is sometimes called in-context learning and a critical aspect of this is the ability to process vast amounts of information after you've trained the model, right? Obviously, we train on a lot of information. That training data goes into the weights as knowledge, but once the model already exists in your chatbot, you can upload documents, pictures, but it would be awesome to upload something deeply, a new game you've invented, you want the model to be extremely good at it. And that is something that is still not even explored that heavily. But the ability to process very long context of information, eventually infinite, is a capability we've been exploring a lot.
And in 1.5 we had a massive breakthrough that obviously keeps being in the model. I'll skip some of the technical details and how to evaluate again in the interest of time. But I'll stop here because this long context also has enabled the ability for the models to reason internally before producing an answer. So this is a slide I took that was explaining how 01, although now all the models have this reasoning capability. And you can ask a question like what is the last digit of 7 to the 99999 and then the model could try to guess, right, just output a number. I mean it has this powerful nonlinear function. So it could possibly in the way somehow do the computation. But by breaking it down and writing kind of for itself how to maybe answer this complex question, it improves especially mathematical ability, science in general and programming massively. It has a cost because now you have to wait for the model to output all these before it outputs the answer. But this technology is a breakthrough that's here to stay. All the models will reason. They shouldn't reason for three hours just to reply hello back which is a problem sometimes we see. You say hello and the model starts to think what does this mean? I mean hello should I say... so it's an interesting phenomenon when you add these capabilities to the models but clearly the basis is very logical. You give it a scratch pad and then to do more complex things this scratch pad becomes fairly long, hundreds of thousands and beyond, and so that's why also long context capabilities to have it in the working memory of the model is quite important as well.
Now with this last week as well we updated our best models and we have these three different benchmarks, doesn't matter too much. Funny name the first one, but there was a belief that that would be the last exam that humans excel over the models. It was released maybe a year ago, models were at 1% and we see already 40%. So the progress has been extremely fast. And maybe more fun, especially for those of you who are into maths, we had this IMO gold competition. This I believe is problem three. So humans are asked to solve this problem. I know many of you, especially I was chatting about this with Dr. Gren. I was trying to explain how the model works. I'm not sure I succeeded but I told him look go and read the solutions which are this link at the bottom if you're interested. But it's quite a daunting thing to see this, right? I mean I am a mathematician, I read that question today, I'm rusty so I might not even understand it too well but okay I get it and then I mean at the time I finish reading the question, I mean the model already has written all these sort of cases, lemmas and so on to basically conclude after lots of writing that this number is four. And then what we do with this is we send it to the judges of IMO and then they basically granted Gemini the gold medal just in the summer. So very interesting experience to see how the field that you think you're an expert goes so quickly in front of your eyes and you have to then spend the time to learn what it did. Of course it does make mistakes. Sometimes it might hallucinate things which is great if you want to try to, you know, you don't know the exam, you invent the theorem that helps you prove what you want to prove. But critical mind is important, right? So we need human experts, we have a few gold medalists in our team to of course assess the model performance because this is at the level that not many people can then read this and understand if it's elegant, if it's efficient and so on. So go read this and you tell me how you found these solutions. This is the 2.5 model. So obviously the current one is meant to be even better than this.
So I'll finish I guess, perhaps just to show a little bit how does this all come together because these numbers, you don't see kind of the real impact to the real world and it's very important as developers of these models that we see how the world reacts. What do they do? What are the issues that they find? Because this iteration between us the creators and the users is very important. It would not work and probably it would actually detriment progress if we went on a secret lab for many years develop these models and then eventually say okay here is AGI or here is some very powerful thing. So this is a very nice property and we love to see what people do with this. Now I have too many of these. So maybe I'll show a few of my favorites. There's so many that I like. The first one is actually how the interaction with applications might change because of what these models can do. So this is basically built into Google and depending on what you ask, right, you can upload a paper and then ask about some notion in this paper you want to learn and what this is. You can try this yourself. What you will get as an answer is not just a piece of text or a link, what you get is a UI that is tailored to your question. So it might generate a little applet to show what you know intuitively what this might be and so on. So you can imagine that the way we do question answering, which is obviously the bread and butter of Google search, changes and it produces not only better answers but also experiences that make it much more easy to then grasp and learn that concept. We have obviously a lot of bias towards science and very advanced queries. But you can imagine how this changes the experience and we're experimenting with this and we'll see where this goes. But it's a very cool property that the way the answer is presented now is dynamic and the model decides what's the best way to do that.
I think there's a lot of vibe coding. I don't know if you know what this means but I think I'll skip this one and I'll skip this one probably as well. I mean cooking is fun. Okay. Anyways, so I just have here a bit more. Coding is a very important application. So users, I mean we use obviously social media to interact with our users, majorly are just doing incredible things, right. So this person here created a Lego simulator by just asking the model to create one. So you can imagine how video games or applications might be kind of almost local, right? You don't have anymore an application that someone builds and for years exists. You will just get a user that wants to do something for a few days and then it builds a Lego simulator and maybe they get bored and this might be shared or not, right? But it changes a lot computation, user interaction, etc. So that's a cool example of the capabilities for a user that might not have ever coded anything to just ask for a thing that requires a piece of code to run in your computer and Gemini 3 does code it for you and then you can just interact with it. Then some other things you can do, I mean this person asked about a simulator of a nuclear plant. So again super useful for education or understanding things. I don't fully understand this. I would not recommend building this. I mean it's difficult. It's a bit obviously toy, but I don't know. It shows kind of the creativity of users that have very different interests, right? As a kid, I could imagine me like trying to understand all sorts of things in science and then seeing it in my screen almost immediately if I asked the question, right? So, it changes the world. Then also lots of different cool tests. I think people often have a query they have in mind that is not part of any evaluation and they ask all the models to do it and then they compare, right? So it's not very scientific but it's a way to evaluate models. So here a person who always asks to do this pelican in SVG and I mean this was impressed by Gemini 3 in this case. Again another simulator for Rubik's cube. Anyways, the creativity of everyone that is passionate about this, you can't do this from any company even as big as Google. So it's very awesome to see how users are interacting and feedback and so please do that, keep it coming, we listen, we look at things. As you can see this is even less than a week old and we have a lot of things we gather for our own understanding. And anyways as I said there's so many of these, all very cool. Like Ali who's actually in my team wanted again to understand DDoS. So this is a UI, but it created this little simulator of DDoS, which if those in tech understand what this is, but anyways, lots of interesting applications. Again, I have too many of these. And I want to definitely save some time for questions. So, take a look, play with it yourself. Language is all you need.
And then last but not least, there is obviously the aspect of generation which we also updated last week. So as I said you can do these sorts of things, right, like create images but the text rendering is really good compared to other models. So you can do all sorts of creative work here with Nano Banana Pro and as I said we focus on the text and the detail because both the input, the understanding and the generation has improved a lot. So here's an example of the attention paper which I had a screenshot of and then someone just asked to annotate this for their understanding and you see this text rendered in the image itself that explains all the different components of the transformer architecture. Also kind of wild to see how design will change. So here you have as an input, my dad is an architect so hopefully he likes that one. You have a blueprint. We're very technical. Not many people can read this. I can obviously because I've been kind of forced throughout my life to understand these things. But you can put a blueprint and then ask it to render it. And it just does it. And you can imagine not in the future, you will get a program that renders this. You can navigate in it in 3D. And all you need is the image. You don't need an expert helping you, which of course probably will do a better job, but it will also take a while to do. So very cool capabilities are being uncovered by our users.
This one I like. This is also Mustafa who is actually in my team. The model cannot only output images but also reason in image spaces. So all of you have seen these puzzles. Oh if you pass this your IQ is over whatever 130. So we haven't quite tested visual IQ yet on the model. Although IQ I think there's someone who does it all the time. But the model here, the best way to solve this puzzle is actually by simulating it in image space. So the model sort of reasons. It's not perfect, right? I mean, this is somewhat cherry-picked, but you get it, right? Like not always these reasoning traces will be in text and certainly for geometry problems like this one, thinking in visual space, even video space and of course textual space is an important property of intelligence. And we're starting to see these kind of properties emerging from the model. Again other cool examples, annotating images to understand like this airplane. Maybe this understands it better than the creators themselves. But of course it was a time of brave tests when flying started. As I understand there was not a lot of science behind it. So great advancement though. And then again Jeff also asked for the planets and you know people asked about Pluto. I mean I also thought Pluto was a planet. And then the model said sure here is Pluto. So you know lots of cool things we're seeing.
This is my last example. It's a bit wild. I think these are very interesting because the first one is solving the math problem but it's literally solving it in the image, right? It's not like understanding the problem and writing a bunch of text. It's actually writing it there. I didn't check correctness here by the way. So, someone could do that. And then the one on the right is also very wild. It creates this fractal effect. It's a fake picture of course. But you ask it to understand kind of taking a picture of a picture of a picture, someone drawing it and it just does it. To be honest it's unclear how the model does it sometimes but it's amazing. And this is I think Sundar posted this since obviously there's a lot of San Francisco vibes in the company.
So I have a few open challenges. Again maybe I would love to maybe have more time for questions. So let me just say that maybe the main challenge, so evaluation is very challenging, numbers go only this far and only when you get real usage you see what the model is actually good at and also what the model fails at. So very very tricky. Data, we run out of it. So there's a lot of challenges on how to extract more from the data. Distillation is one way but there's lots of research here to be done with regularization and so on that machine learning basics can come back to help LLMs which hasn't been needed until now. Modeling, I think modeling is actually transformers. I would bet not against transformers although I'm sure there will be someone coming up with the next great architecture. Pre-training, you know, very massive scale infrastructure needed. So I think for countries to be able to pre-train models you would need a lot of investment. So the challenge is the cost, pre-training is very costly, absorbing all the internet knowledge into the weights. And then last but not least, I think this is the area where there's a lot of cool research to be done. It is very challenging to do reinforcement learning with a model because unlike go or chess, we can't get the win condition, right? You ask for a poem and you would like it or you will not like it. And how do we evaluate objectively whether the model is doing a good job and we reinforce that like hey you won the game of go, good job, do more of this, versus all these tasks that are very subjective and very difficult to create. Even IMO, I mean we need human experts to tell us if this is actually a reasonable reasoning trace and answer. So it's extremely challenging to evaluate. There is a reason for hope because evaluation I believe fundamentally is easier than creating the answer. Right? This is a bit of the NP argument in computer science. Once you see the solution it's easy to verify it, very hard to create the solution. So I'm hopeful there's a lot of innovation here to be able to evaluate models. But this will I think be the main driver for progress in Gemini 3.5, 4.0 and beyond.
And then many other issues, factuality, hallucinations, cracking self-improvement, safety especially as we give access to computers. These models, I mean, should you have access to the internet? The internet is not a place where you can guarantee read-only operation. So there's a lot of interesting aspects that we can discuss maybe offline. And then last to conclude, Shane in 2009 had this plot predicting that there's a 50% chance of AGI and this is not too bad and 2009 is a long time ago. So I give it to Shane for having predicted at least some form of declaration of AGI fairly accurately. We'll see what the scientific community agrees or disagrees with. We tend to disagree as scientists always do, which is of course fine. But very exciting to see where it goes. As I said, extremely exciting time to be alive. I'm happy to share with you many more things during the Q&A, but in the meantime, thanks for your attention and it's been awesome to be back in Barcelona. Thank you very much.
