Ilya Sutskever – We're moving from the age of scaling to the age of research

Sure. I think your point, about how from the average person's point of view nothing is that different, will continue being true even into the singularity.

Okay, interesting. The thing which I was referring to not feeling different is, okay, such and such company announced some difficult-to-comprehend dollar amount of investment. I don't think anyone knows what to do with that.

I think the models seem smarter than their economic impact would imply.

You look at the evals and you go, 'Those are pretty hard evals.' They are doing so well. But the economic impact seems to be dramatically behind. It's very difficult to make sense of, how can the model, on the one hand, do these amazing things, and then on the other hand, repeat itself twice in some situation? An example would be, let's say you use vibe coding to do something. You go to some place and then you get a bug. Then you tell the model, 'Can you please fix the bug?' And the model says, 'Oh my God, you're so right. I have a bug. Let me go fix that.' And it introduces a second bug. Then you tell it, 'You have this new second bug,' and it tells you, 'Oh my God, how could I have done it? You're so right again,' and brings back the first bug, and you can alternate between those. How is that possible? I'm not sure, but it does suggest that something strange is going on. I have two possible explanations. The more whimsical explanation is that maybe RL training makes the models a little too single-minded and narrowly focused, a little bit too unaware, even though it also makes them aware in some other ways. Because of this, they can't do basic things. But there is another explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. When you do pre-training, you need all the data. So you don't have to think if it's going to be this data or that data. But when people do RL training, they do need to think. They say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' From what I hear, all the companies have teams that just produce new RL environments and just add it to the training mix. The question is, well, what are those? There are so many degrees of freedom. There is such a huge variety of RL environments you could produce. One thing you could do, and I think this is something that is done inadvertently, is that people take inspiration from the evals. You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' I think that is something that happens, and it could explain a lot of what's going on. If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance, which is something that we don't today even understand, what we mean by that.

I think there are two ways to understand, or to try to think about, what you have just pointed out. One is that if it's the case that simply by becoming superhuman at a coding competition, a model will not automatically become more tasteful and exercise better judgment about how to improve your codebase, well then you should expand the suite of environments such that you're not just testing it on having the best performance in coding competition. It should also be able to make the best kind of application for X thing or Y thing or Z thing. Another, maybe this is what you're hinting at, is to say, 'Why should it be the case in the first place that becoming superhuman at coding competitions doesn't make you a more tasteful programmer more generally?' Maybe the thing to do is not to keep stacking up the amount and diversity of environments, but to figure out an approach which lets you learn from one environment and improve your performance on something else. I have a human analogy which might be helpful. Let's take the case of competitive programming, since you mentioned that. Suppose you have two students. One of them decided they want to be the best competitive programmer, so they will practice 10,000 hours for that domain. They will solve all the problems, memorize all the proof techniques, and be very skilled at quickly and correctly implementing all the algorithms. By doing so, they became one of the best. Student number two thought, 'Oh, competitive programming is cool.' Maybe they practiced for 100 hours, much less, and they also did really well. Which one do you think is going to do better in their career later on? The second. Right. I think that's basically what's going on. The models are much more like the first student, but even more. Because then we say, the model should be good at competitive programming so let's get every single competitive programming problem ever. And then let's do some data augmentation so we have even more competitive programming problems, and we train on that. Now you've got this great competitive programmer. With this analogy, I think it's more intuitive.

I think they have 'it.' The 'it' factor. When I was an undergrad, I remember there was a student like this that studied with me, so I know it exists.

The main strength of pre-training is that: A, there is so much of it, and B, you don't have to think hard about what data to put into pre-training. It's very natural data, and it does include in it a lot of what people do: people's thoughts and a lot of the features. It's like the whole world as projected by people onto text, and pre-training tries to capture that using a huge amount of data. Pre-training is very difficult to reason about because it's so hard to understand the manner in which the model relies on pre-training data. Whenever the model makes a mistake, could it be because something by chance is not as supported by the pre-training data? 'Support by pre-training' is maybe a loose term. I don't know if I can add anything more useful on this.

I think there are some similarities between both of these and pre-training, and pre-training tries to play the role of both of these. But I think there are some big differences as well. The amount of pre-training data is very, very staggering. Yes. Somehow a human being, after even 15 years with a tiny fraction of the pre-training data, they know much less. But whatever they do know, they know much more deeply somehow. Already at that age, you would not make mistakes that our AIs make. There is another thing. You might say, could it be something like evolution? The answer is maybe. But in this case, I think evolution might actually have an edge. I remember reading about this case. One way in which neuroscientists can learn about the brain is by studying people with brain damage to different parts of the brain. Some people have the most strange symptoms you could imagine. It's actually really, really interesting. One case that comes to mind that's relevant. I read about this person who had some kind of brain damage, a stroke or an accident, that took out his emotional processing. So he stopped feeling any emotion. He still remained very articulate and he could solve little puzzles, and on tests he seemed to be just fine. But he felt no emotion. He didn't feel sad, he didn't feel anger, he didn't feel animated. He became somehow extremely bad at making any decisions at all. It would take him hours to decide on which socks to wear. He would make very bad financial decisions. What does it say about the role of our built-in emotions in making us a viable agent, essentially? To connect to your question about pre-training, maybe if you are good enough at getting everything out of pre-training, you could get that as well. But that's the kind of thing which seems... Well, it may or may not be possible to get that from pre-training.

I think it could. I'm just saying it's not 100% obvious.

It should be some kind of a value function thing. But I don't think there is a great ML analogy because right now, value functions don't play a very prominent role in the things people do.

Certainly, I'll be very happy to do that. When people do reinforcement learning, the way reinforcement learning is done right now, how do people train those agents? You have your neural net and you give it a problem, and then you tell the model, 'Go solve it.' The model takes maybe thousands, hundreds of thousands of actions or thoughts or something, and then it produces a solution. The solution is graded. And then the score is used to provide a training signal for every single action in your trajectory. That means that if you are doing something that goes for a long time—if you're training a task that takes a long time to solve—it will do no learning at all until you come up with the proposed solution. That's how reinforcement learning is done naively. That's how o1, R1 ostensibly are done. The value function says something like, 'Maybe I could sometimes, not always, tell you if you are doing well or badly.' The notion of a value function is more useful in some domains than others. For example, when you play chess and you lose a piece, I messed up. You don't need to play the whole game to know that what I just did was bad, and therefore whatever preceded it was also bad. The value function lets you short-circuit the wait until the very end. Let's suppose that you are doing some kind of a math thing or a programming thing, and you're trying to explore a particular solution or direction. After, let's say, a thousand steps of thinking, you concluded that this direction is unpromising. As soon as you conclude this, you could already get a reward signal a thousand timesteps previously, when you decided to pursue down this path. You say, 'Next time I shouldn't pursue this path in a similar situation,' long before you actually came up with the proposed solution.

Sure it might be difficult, but nothing deep learning can't do. My expectation is that a value function should be useful, and I fully expect that they will be used in the future, if not already. What I was alluding to with the person whose emotional center got damaged, it's more that maybe what it suggests is that the value function of humans is modulated by emotions in some important way that's hardcoded by evolution. And maybe that is important for people to be effective in the world.

I have two responses. I do agree that compared to the kind of things that we learn and the things we are talking about, the kind of AI we are talking about, emotions are relatively simple. They might even be so simple that maybe you could map them out in a human-understandable way. I think it would be cool to do. In terms of utility though, I think there is a thing where there is this complexity-robustness tradeoff, where complex things can be very useful, but simple things are very useful in a very broad range of situations. One way to interpret what we are seeing is that we've got these emotions that evolved mostly from our mammal ancestors and then fine-tuned a little bit while we were hominids, just a bit. We do have a decent amount of social emotions though which mammals may lack. But they're not very sophisticated. And because they're not sophisticated, they serve us so well in this very different world compared to the one that we've been living in. Actually, they also make mistakes. For example, our emotions... Well actually, I don't know. Does hunger count as an emotion? It's debatable. But I think, for example, our intuitive feeling of hunger is not succeeding in guiding us correctly in this world with an abundance of food.

Here's a perspective that I think might be true. The way ML used to work is that people would just tinker with stuff and try to get interesting results. That's what's been going on in the past. Then the scaling insight arrived. Scaling laws, GPT-3, and suddenly everyone realized we should scale. This is an example of how language affects thought. 'Scaling' is just one word, but it's such a powerful word because it informs people what to do. They say, 'Let's try to scale things.' So you say, what are we scaling? Pre-training was the thing to scale. It was a particular scaling recipe. The big breakthrough of pre-training is the realization that this recipe is good. You say, 'Hey, if you mix some compute with some data into a neural net of a certain size, you will get results. You will know that you'll be better if you just scale the recipe up.' This is also great. Companies love this because it gives you a very low-risk way of investing your resources. It's much harder to invest your resources in research. Compare that. If you research, you need to be like, 'Go forth researchers and research and come up with something', versus get more data, get more compute. You know you'll get something from pre-training. Indeed, it looks like, based on various things some people say on Twitter, maybe it appears that Gemini have found a way to get more out of pre-training. At some point though, pre-training will run out of data. The data is very clearly finite. What do you do next? Either you do some kind of souped-up pre-training, a different recipe from the one you've done before, or you're doing RL, or maybe something else. But now that compute is big, compute is now very big, in some sense we are back to the age of research. Maybe here's another way to put it. Up until 2020, from 2012 to 2020, it was the age of research. Now, from 2020 to 2025, it was the age of scaling— maybe plus or minus, let's add error bars to those years—because people say, 'This is amazing. You've got to scale more. Keep scaling.' The one word: scaling. But now the scale is so big. Is the belief really, 'Oh, it's so big, but if you had 100x more, everything would be so different?' It would be different, for sure. But is the belief that if you just 100x the scale, everything would be transformed? I don't think that's true. So it's back to the age of research again, just with big computers.

We've already witnessed a transition from one type of scaling to a different type of scaling, from pre-training to RL. Now people are scaling RL. Now based on what people say on Twitter, they spend more compute on RL than on pre-training at this point, because RL can actually consume quite a bit of compute. You do very long rollouts, so it takes a lot of compute to produce those rollouts. Then you get a relatively small amount of learning per rollout, so you really can spend a lot of compute. I wouldn't even call it scaling. I would say, 'Hey, what are you doing? Is the thing you are doing the most productive thing you could be doing? Can you find a more productive way of using your compute?' We've discussed the value function business earlier. Maybe once people get good at value functions, they will be using their resources more productively. If you find a whole other way of training models, you could say, 'Is this scaling or is it just using your resources?' I think it becomes a little bit ambiguous. In the sense that, when people were in the age of research back then, it was, 'Let's try this and this and this. Let's try that and that and that. Oh, look, something interesting is happening.' I think there will be a return to that.

The discussion about value function, I think it was interesting. I want to emphasize that I think the value function is something that's going to make RL more efficient, and I think that makes a difference. But I think anything you can do with a value function, you can do without, just more slowly. The thing which I think is the most fundamental is that these models somehow just generalize dramatically worse than people. It's super obvious. That seems like a very fundamental thing. So this is the crux: generalization. There are two sub-questions. There's one which is about sample efficiency: why should it take so much more data for these models to learn than humans? There's a second question. Even separate from the amount of data it takes, why is it so hard to teach the thing we want to a model than to a human? For a human, we don't necessarily need a verifiable reward to be able to... You're probably mentoring a bunch of researchers right now, and you're talking with them, you're showing them your code, and you're showing them how you think. From that, they're picking up your way of thinking and how they should do research. You don't have to set a verifiable reward for them that's like, 'Okay, this is the next part of the curriculum, and now this is the next part of your curriculum. Oh, this training was unstable.' There's not this schleppy, bespoke process.

You could actually wonder that one possible explanation for the human sample efficiency that needs to be considered is evolution. Evolution has given us a small amount of the most useful information possible. For things like vision, hearing, and locomotion, I think there's a pretty strong case that evolution has given us a lot. For example, human dexterity far exceeds... I mean robots can become dexterous too if you subject them to a huge amount of training in simulation. But to train a robot in the real world to quickly pick up a new skill like a person does seems very out of reach. Here you could say, 'Oh yeah, locomotion. All our ancestors needed great locomotion, squirrels.' So with locomotion, maybe we've got some unbelievable prior. You could make the same case for vision. I believe Yann LeCun made the point that children learn to drive after 10 hours of practice, which is true. But our vision is so good. At least for me, I remember myself being a five-year-old. I was very excited about cars back then. I'm pretty sure my car recognition was more than adequate for driving already as a five-year-old. You don't get to see that much data as a five-year-old. You spend most of your time in your parents' house, so you have very low data diversity. But you could say maybe that's evolution too. But in language and math and coding, probably not.

Oh yeah. Oh yeah, absolutely. What I meant to say is that language, math, and coding—and especially math and coding—suggests that whatever it is that makes people good at learning is probably not so much a complicated prior, but something more, some fundamental thing.

Much more robust. The robustness of people is really staggering.

That is a great question to ask, and it's a question I have a lot of opinions about. But unfortunately, we live in a world where not all machine learning ideas are discussed freely, and this is one of them. There's probably a way to do it. I think it can be done. The fact that people are like that, I think it's a proof that it can be done. There may be another blocker though, which is that there is a possibility that the human neurons do more compute than we think. If that is true, and if that plays an important role, then things might be more difficult. But regardless, I do think it points to the existence of some machine learning principle that I have opinions on. But unfortunately, circumstances make it hard to discuss in detail.

One consequence of the age of scaling is that scaling sucked out all the air in the room. Because scaling sucked out all the air in the room, everyone started to do the same thing. We got to the point where we are in a world where there are more companies than ideas by quite a bit. Actually on that, there is this Silicon Valley saying that says that ideas are cheap, execution is everything. People say that a lot, and there is truth to that. But then I saw someone say on Twitter something like, 'If ideas are so cheap, how come no one's having any ideas?' And I think it's true too. If you think about research progress in terms of bottlenecks, there are several bottlenecks. One of them is ideas, and one of them is your ability to bring them to life, which might be compute but also engineering. If you go back to the '90s, let's say, you had people who had pretty good ideas, and if they had much larger computers, maybe they could demonstrate that their ideas were viable. But they could not, so they could only have a very, very small demonstration that did not convince anyone. So the bottleneck was compute. Then in the age of scaling, compute has increased a lot. Of course, there is a question of how much compute is needed, but compute is large. Compute is large enough such that it's not obvious that you need that much more compute to prove some idea. I'll give you an analogy. AlexNet was built on two GPUs. That was the total amount of compute used for it. The transformer was built on 8 to 64 GPUs. No single transformer paper experiment used more than 64 GPUs of 2017, which would be like, what, two GPUs of today? The ResNet, right? You could argue that the o1 reasoning was not the most compute-heavy thing in the world. So for research, you definitely need some amount of compute, but it's far from obvious that you need the absolutely largest amount of compute ever for research. You might argue, and I think it is true, that if you want to build the absolutely best system then it helps to have much more compute. Especially if everyone is within the same paradigm, then compute becomes one of the big differentiators.

Correct.

I can comment on that. The short comment is that you mentioned SSI. Specifically for us, the amount of compute that SSI has for research is really not that small. I want to explain why. Simple math can explain why the amount of compute that we have is comparable for research than one might think. I'll explain. SSI has raised $3 billion, which is a lot by any absolute sense. But you could say, 'Look at the other companies raising much more.' But a lot of their compute goes for inference. These big numbers, these big loans, it's earmarked for inference. That's number one. Number two, if you want to have a product on which you do inference, you need to have a big staff of engineers, salespeople. A lot of the research needs to be dedicated to producing all kinds of product-related features. So then when you look at what's actually left for research, the difference becomes a lot smaller. The other thing is, if you are doing something different, do you really need the absolute maximal scale to prove it? I don't think that's true at all. I think that in our case, we have sufficient compute to prove, to convince ourselves and anyone else, that what we are doing is correct.

I think it's a question of what you do with it. It's a question of what you do with it. In their case, in the case of others, there is a lot more demand on the training compute. There's a lot more different work streams, there are different modalities, there is just more stuff. So it becomes fragmented.

My answer to this question is something like this. Right now, we just focus on the research, and then the answer to that question will reveal itself. I think there will be lots of possible answers.

Maybe. I think that there is merit to it. I think there's a lot of merit because it's very nice to not be affected by the day-to-day market competition. But I think there are two reasons that may cause us to change the plan. One is pragmatic, if timelines turned out to be long, which they might. Second, I think there is a lot of value in the best and most powerful AI being out there impacting the world. I think this is a meaningfully valuable thing.

I'll make the case for and against. The case for is that one of the challenges that people face when they're in the market is that they have to participate in the rat race. The rat race is quite difficult in that it exposes you to difficult trade-offs which you need to make. It is nice to say, 'We'll insulate ourselves from all this and just focus on the research and come out only when we are ready, and not before.' But the counterpoint is valid too, and those are opposing forces. The counterpoint is, 'Hey, it is useful for the world to see powerful AI. It is useful for the world to see powerful AI because that's the only way you can communicate it.' Well, I guess not even just that you can communicate the idea—