I want to thank the organizers for choosing a paper for this award. It was very nice, and I also want to thank my incredible co-authors and collaborators: Oriol, Vineel, and Quoc, who stood right before you a moment ago. And what you have here is an image, a screenshot from a similar talk 10 years ago at NeurIPS in 2014 in Montreal. It was a much more innocent time. Here we are shown in the photos. This is the before; here's the after, by the way. And now we've got more experience, hopefully wiser. But here I'd like to talk a little bit about the work itself, and maybe a 10-year retrospective on it, because a lot of the things in this work were correct, but some not so much. We can review them and see what happened and how it gently flowed to where we are today. So let's begin by talking about what we did, and the way we'll do it is by showing slides from the same talk 10 years ago. But the summary of what we did is the following three bullet points: it's an autoregressive model trained on text, it's a large neural network, and it's a large dataset. And that's it. Now let's dive into the details a little bit more. So this was a slide 10 years ago, not too bad: the deep learning hypothesis. And what we said here is that if you have a large neural network with 10 layers, then it can do anything that a human being can do in a fraction of a second. Why did we have this emphasis on things that human beings can do in a fraction of a second? Why this thing specifically? Well, if you believe the deep learning dogma, so to say, that artificial neurons and biological neurons are similar, or at least not too different, and you believe that real neurons are slow, then anything that we can do quickly—by 'we' I mean human beings, I even mean just one human in the entire world—if there is one human in the entire world that can do some task in a fraction of a second, then a 10-layer neural network can do it too, right? It follows: you just take their connections and you embed them inside your neural network, the artificial one. So this was the motivation: anything that a human being can do in a fraction of a second, a big 10-layer neural network can do too. We focused on 10-layer neural networks because this was the neural networks we knew how to train back in the day. If you could go beyond in your layers somehow, then you could do more, but back then we could only do 10 layers, which is why we emphasized whatever human beings can do in a fraction of a second. A different slide from the talk, a slide which says 'Our main idea,' and you may be able to recognize two things, or at least one thing: you might be able to recognize that something autoregressive is going on here. What is it saying really? What does this slide really say? This slide says that if you have an autoregressive model and it predicts the next token well enough, then it will in fact grab and capture and grasp the correct distribution over whatever sequences come next. And this was a relatively new thing. It wasn't literally the first ever autoregressive neural network, but I would argue it was the first autoregressive neural network where we really believed that if you train it really well, then you will get whatever you want. In our case, back then, was the humble—today humble, then incredibly audacious—task of translation. Now I'm going to show you some ancient history that many of you might have never seen before. It's called the LSTM. To those unfamiliar, an LSTM is the thing that poor deep learning researchers did before Transformers. And it's basically a ResNet but rotated 90 degrees. So that's an LSTM, and it came before. It's like a slightly more complicated ResNet. You can see there is your integrator, which is now called the residual stream, but you've got some multiplication going on; it's a little bit more complicated. But that's what we did. It was a ResNet rotated 90 degrees. Another cool feature from that old talk that I want to highlight is that we used parallelization, but not just any parallelization: we used pipelining, as witnessed by this one layer per GPU. Was it wise to pipeline? As we now know, pipelining is not wise, but we were not as wise back then. So we used that and we got a 3.5x speedup using eight GPUs. And the conclusion slide, in some sense, the conclusion slide from the talk from back then is the most important slide because it spelled out what could arguably be the beginning of the scaling hypothesis, right? That if you have a very big dataset and you train a very big neural network, then success is guaranteed. And one can argue, if one is charitable, that this indeed has been what's been happening. I want to mention one other idea, and this is, I claim, the idea that truly stood the test of time. It's the core idea of deep learning itself. It's the idea of connectionism. It's the idea that if you allow yourself to believe that an artificial neuron is kind of sort of like a biological neuron, right? If you believe that one is kind of sort like the other, then it gives you the confidence to believe that very large neural networks—they don't need to be literally human brain scale, they might be a little bit smaller—but you could configure them to do pretty much all the things that we do, human beings. There's still a difference. Oh, I forgot to the end: there is still a difference because the human brain also figures out how to reconfigure itself, whereas we are using the best learning algorithms that we have, which require as many data points as there are parameters. Human beings are still better in this regard. But what this led to, I claim, arguably, to the age of pre-training. And the age of pre-training is what we might say: the GPT-2 model, the GPT-3 model, the scaling laws. And I want to specifically call out my former collaborators Alec Radford, also Jared Kaplan, Dario Amodei, for really making this work. But that led to the age of pre-training, and this is what's been the driver of all progress, all the progress that we see today: extra large neural networks, extraordinarily large neural networks trained on huge datasets. But pre-training as we know it will unquestionably end. Pre-training will end. Why will it end? Because while computers grow through better hardware, better algorithms, and larger clusters, right? All those things keep increasing your compute. All these things keep increasing your compute. The data is not growing, because we have but one internet. We have but one internet. You could even go as far as to say that data is the fossil fuel of AI. It was created somehow, and now we use it. And we've achieved peak data, and there'll be no more. We have to deal with the data that we have now. It still let us go quite far, but this is it. There's only one internet. So here I'll take a bit of liberty to speculate about what comes next. Actually, I don't need to speculate, because many people are speculating too, and I'll mention their speculations. You may have heard the phrase 'agents.' It's common, and I'm sure that eventually something will happen, but people feel like something like agents is the future. More concretely, but also a little bit vaguely, synthetic data. But what does synthetic data mean? Figuring this out is a big challenge, and I'm sure that different people have all kinds of interesting progress there. And inference-time compute, or maybe what's been most recently most vividly seen in the o1 model. These are all examples of things of people trying to figure out what to do after pre-training, and those are all very good things to do. I want to mention one other example from biology which I think is really cool. And the example is this: about many, many years ago at this conference also, I saw a talk where someone presented this graph, but the graph showed the relationship between the size of the body of a mammal and the size of their brain—in this case, it's in mass. And that talk, I remember vividly, they were saying, 'Look, in biology everything is so messy, but here you have one rare example where there is a very tight relationship between the size of the body of the animal and their brain.' And totally randomly, I became curious at this graph. One of the early images on Google Images was this. And the interesting thing in this image is you see, like, I don't know, is the mouse working? Oh yeah, the mouse is working great. So you've got these mammals, right? All the different mammals. Then you've got nonhuman primates; it's basically the same thing. But then you've got the hominids. And to my knowledge, hominids are like close relatives to the humans in evolution, like the Neanderthals. There's a bunch of them, like Homo habilis maybe, a whole bunch. And they're all here. And what's interesting is that they have a different slope on their brain-to-body scaling exponent. So that's pretty cool. What that means is that there is a precedent, there is an example of biology figuring out some kind of different scaling. Something clearly is different. So I think that is cool. And by the way, I want to highlight: this x-axis is log scale. You see this is 100, this is 1,000, 10,000, 100,000, and likewise in grams: 1 gram, 10 grams, 100 grams, 1,000 grams. So it is possible for things to be different. The things that we are doing, the things that we've been scaling so far, is actually the first thing that we figured out how to scale. And without doubt, the field—everyone who's working here—will figure out what to do. But I want to take a few minutes and speculate about the longer term. The longer term: where are we all headed, right? We're making all this progress. It's astounding progress. It's really—I mean, those of you who've been in the field 10 years ago, you remember just how incapable everything has been. Like, yes, you can say, 'Of course, deep learning still to see it is just unbelievable.' It's completely—I can't convey that feeling to you. You know, if you joined the field in the last two years, then of course you speak to computers and they talk back to you and they disagree, and that's what computers are. But it hasn't always been the case. But I want to talk a little bit about superintelligence, just a bit, because that is obviously where this field is headed. This is obviously what's being built here. And the thing about superintelligence is that it will be different qualitatively from what we have. And my goal in the next minute is to try to give you some concrete intuition of how it will be different, so that you yourself could reason about it. So right now we have our incredible language models and the unbelievable chatbot, and they can even do things, but they're also kind of strangely unreliable and they get confused, while also having dramatically superhuman performance on evals. So it's really unclear how to reconcile this. But eventually, sooner or later, the following will be achieved: those systems are actually going to be agentic in a real way. Whereas right now, the systems are not agents in any meaningful sense. Just very—that might be too strong—they're very, very slightly agentic, just beginning. It will actually reason. And by the way, I want to mention something about reasoning: a system that reasons, the more it reasons, the more unpredictable it becomes. The more it reasons, the more unpredictable it becomes. All the deep learning that we've been used to is very predictable, because if you've been working on replicating human intuition essentially, it's like the gut feeling. If you come back to the 0.1 second reaction time, what kind of processing do we do in our brains? Well, it's our intuition. So we've endowed our systems with some of that intuition. But reasoning—you're seeing some early signs of that. Reasoning is unpredictable. And one reason to see that is because the chess AIs, the really good ones, are unpredictable to the best human chess players. So we will have to be dealing with AI systems that are incredibly unpredictable. They will understand things from limited data. They will not get confused. All the things which are really big limitations—I'm not saying how, by the way, and I'm not saying when. I'm saying that it will. And when all those things will happen together with self-awareness, because why not? Self-awareness is useful. It is part of our own world models. When all those things come together, we will have systems of radically different qualities and properties than exist today. And of course, they will have incredible and amazing capabilities. But the kind of issues that come up with systems like this—and I'll just leave it as an exercise to imagine—it's very different from what we used to. And I would say that it's definitely also impossible to predict the future. Really, all kinds of stuff is possible. But on this uplifting note, I will conclude. Thank you so much.
