I want to thank the organizers for choosing a paper for this award. It was very nice, and I also want to thank my incredible co-authors and collaborators, Oriol Vinyals and Quoc Le, who stood right before you a moment ago.
What you have here is an image, a screenshot from a similar talk 10 years ago at NeurIPS in 2014 in Montreal. It was a much more innocent time. Here we are shown in the photos. This is the before, here's the after, by the way. And now we've got more experienced, hopefully wiser.
But here I'd like to talk a little bit about the work itself and maybe a 10-year retrospective on it, because a lot of the things in this work were correct, but some not so much. And we can review them and we can see what happened and how it gently flowed to where we are today.
So let's begin by talking about what we did, and the way we'll do it is by showing slides from the same talk 10 years ago. But the summary of what we did is the following three bullet points: it's an autoregressive model trained on text, it's a large neural network, and it's a large dataset. And that's it. Now let's dive into the details a little bit more.
So this was a slide 10 years ago, not too bad. The deep learning hypothesis. And what we said here is that if you have a large neural network with 10 layers, then it can do anything that a human being can do in a fraction of a second. Like, why did we have this emphasis on things that human beings can do in a fraction of a second? Why this thing specifically?
Well, if you believe the deep learning dogma, so to say, that artificial neurons and biological neurons are similar, or at least not too different, and you believe that real neurons are slow, then anything that we can do quickly—by we I mean human beings, I even mean just one human in the entire world—if there is one human in the entire world that can do some task in a fraction of a second, then a 10-layer neural network can do it too, right? It follows. You just take their connections and you embed them inside your neural net, the artificial one.
So this was the motivation: anything that a human being can do in a fraction of a second, a big 10-layer neural network can do too. We focused on 10-layer neural networks because this was the neural networks we knew how to train back in the day. If you could go beyond 10 layers somehow, then you could do more, but back then we could only do 10 layers, which is why we emphasized whatever human beings can do in a fraction of a second.
A different slide from the talk, a slide which says 'our main idea.' And you may be able to recognize two things, or at least one thing. You might be able to recognize that something autoregressive is going on here. What is it saying really? What does this slide really say? This slide says that if you have an autoregressive model and it predicts the next token well enough, then it will in fact grab and capture and grasp the correct distribution over whatever, over sequences that come next.
And this was a relatively new thing. It wasn't literally the first ever autoregressive neural network, but I would argue it was the first autoregressive neural network where we really believed that if you train it really well, then you will get whatever you want. In our case, back then, was the humble—today humble, then incredibly audacious—task of translation.
Now I'm going to show you some ancient history that many of you might have never seen before. It's called the LSTM. To those unfamiliar, an LSTM is the thing that poor deep learning researchers did before Transformers. And it's basically a ResNet but rotated 90 degrees. So that's an LSTM, and it came before. It's like, it's kind of like a slightly more complicated ResNet. You can see there is your integrator, which is now called the residual stream, but you've got some multiplication going on. It's a little bit more complicated, but that's what we did. It was a ResNet rotated 90 degrees.
Another cool feature from that old talk that I want to highlight is that we used parallelization, but not just any parallelization. We used pipelining, as witnessed by this one layer per GPU. Was it wise to pipeline? As we now know, pipelining is not wise, but we were not as wise back then. So we used that, and we got a 3.5x speedup using eight GPUs.
And the conclusion slide, in some sense, the conclusion slide from the talk from back then is the most important slide, because it spelled out what could arguably be the beginning of the scaling hypothesis. Right? That if you have a very big dataset and you train a very big neural network, then success is guaranteed. And one can argue, if one is charitable, that this indeed has been what's been happening.
I want to mention one other idea, and this is, I claim, the idea that truly stood the test of time. It's the core idea of deep learning itself. It's the idea of connectionism. It's the idea that if you allow yourself to believe that an artificial neuron is kind of sort of like a biological neuron, right? If you believe that one is kind of sort like the other, then it gives you the confidence to believe that very large neural networks—they don't need to be literally human brain scale, they might be a little bit smaller—but you could configure them to do pretty much all the things that we do, human beings.
There's still a difference. Oh, I forgot to the end. There is still a difference, because the human brain also figures out how to reconfigure itself, whereas we are using the best learning algorithms that we have, which require as many data points as there are parameters. Human beings are still better in this regard.
But what this led, so I claim, arguably, to the age of pre-training. And the age of pre-training is what we might say the GPT-2 model, the GPT-3 model, the scaling laws. And I want to specifically call out my former collaborators Alec Radford, also Jared Kaplan, Dario Amodei, for really making this work. But that led to the age of pre-training, and this is what's been the driver of all the progress that we see today: extra large neural networks, extraordinarily large neural networks trained on huge datasets.
But pre-training as we know it will unquestionably end. Pre-training will end. Why will it end? Because while compute is growing through better hardware, better algorithms, and larger clusters, right? All those things keep increasing your compute. The data is not growing, because we have but one internet. We have but one internet. You could even say, you can even go as far as to say that data is the fossil fuel of AI. It was like created somehow, and now we use it. And we've achieved peak data, and there'll be no more. We have to deal with the data that we have now. It still will let us go quite far, but this is... there's only one internet.
So here I'll take a bit of liberty to speculate about what comes next. Actually, I don't need to speculate, because many people are speculating too, and I'll mention their speculations. You may have heard the phrase 'agents.' It's common, and I'm sure that eventually something will happen, but people feel like something, agents, is the future. More concretely, but also a little bit vaguely, synthetic data. But what does synthetic data mean? Figuring this out is a big challenge, and I'm sure that different people have all kinds of interesting progress there. And inference-time compute, or maybe what's been most recently, most vividly seen in o1, the o1 model. These are all examples of things, of people trying to figure out what to do after pre-training, and those are all very good things to do.
I want to mention one other example from biology, which I think is really cool. And the example is this: so about many, many years ago at this conference also, I saw a talk where someone presented this graph. But the graph showed the relationship between the size of the body of a mammal and the size of their brain. In this case, it's in mass. And that talk, I remember vividly, they were saying, 'Look, it's in biology, everything is so messy, but here you have one rare example where there is a very tight relationship between the size of the body of the animal and their brain.' And totally randomly, I became curious at this graph. And one of the early... so I went to Google to do research, to look for this graph, and one of the images in Google Images was this.
And the interesting thing in this image is you see, like, I don't know, is the mouse working? Oh yeah, the mouse is working, great. So you've got these mammals, right? All the different mammals. Then you've got non-human primates. It's basically the same thing. But then you've got the hominids. And to my knowledge, hominids are like close relatives to the humans in evolution, like the Neanderthals. There's a bunch of them, like it's called Homo habilis, maybe there are a whole bunch, and they're all here. And what's interesting is that they have a different slope on their brain-to-body scaling exponent. So that's pretty cool. What that means is that there is a precedent, there is an example of biology figuring out some kind of different scaling. Something clearly is different. So I think that is cool. And by the way, I want to highlight this x-axis is log scale. You see this is 100, this is 1,000, 10,000, 100,000, and likewise in grams: 1g, 10g, 100g, 1,000g.
So it is possible for things to be different. The things that we are doing, the things that we've been scaling so far, is actually the first thing that we figured out how to scale. And without doubt, the field, everyone who's working here, will figure out what to do. But I want to talk here, I want to take a few minutes and speculate about the longer term. The longer term, where are we all headed, right? We're making all this progress. It's astounding progress. It's really... I mean, those of you who have been in the field 10 years ago and you remember just how incapable everything has been... like, yes, you can say, even if you kind of say, 'Of course, I'm used to it,' still to see it is just unbelievable. It's completely... I can't convey that feeling to you. You know, if you joined the field in the last two years, then of course you speak to computers and they talk back to you and they disagree, and that's what computers are. But it hasn't always been the case.
But I want to talk a little bit about superintelligence, just a bit, because that is obviously where this field is headed. This is obviously what's being built here. And the thing about superintelligence is that it will be different qualitatively from what we have. And my goal in the next minute to try to give you some concrete intuition of how it will be different, so that you yourself could reason about it.
So right now we have our incredible language models and the unbelievable chatbots, and they can even do things, but they're also kind of strangely unreliable and they get confused, while also having dramatically superhuman performance on evals. So it's really unclear how to reconcile this. But eventually, sooner or later, the following will be achieved: those systems are actually going to be agentic in a real way, whereas right now the systems are not agents in any meaningful sense. Just very... that might be too strong, they're very, very slightly agentic, just beginning. It will actually reason. And by the way, I want to mention something about reasoning: is that a system that reasons, the more it reasons, the more unpredictable it becomes. The more it reasons, the more unpredictable it becomes. All the deep learning that we've been used to is very predictable, because if you've been working on replicating human intuition essentially, it's like the gut feeling. If you come back to the 0.1-second reaction time, what kind of processing we do in our brains? Well, it's our intuition. So we've endowed our models with some of that intuition. But reasoning, you're seeing some early signs of that, reasoning is unpredictable. And one reason to see that is because the chess AIs, the really good ones, are unpredictable to the best human chess players.
So we will have to be dealing with AI systems that are incredibly unpredictable. They will understand things from limited data. They will not get confused. All the things which are really big limitations—I'm not saying how, by the way, and I'm not saying when, I'm saying that it will. And when all those things will happen together with self-awareness, because why not? Self-awareness is useful. It is part... our selves are parts of our own world models. When all those things come together, we will have systems of radically different qualities and properties that exist today. And of course they will have incredible and amazing capabilities, but the kind of issues that come up with systems like this—and I'll just leave it as an exercise to imagine—it's very different from what we're used to. And I would say that it's definitely also impossible to predict the future. Really, all kinds of stuff is possible. But on this uplifting note, I will conclude. Thank you so much.
