Today I'm chatting with Richard Sutton, who is one of the founding fathers of reinforcement learning and inventor of many of the main techniques used there, like TD learning and policy gradient methods. For that, he received this year's Turing Award, which, if you don't know, is the Nobel Prize for computer science. Richard, congratulations.
Thank you, Dwarkesh. Thanks for coming on the podcast.
It's my pleasure. First question. My audience and I are familiar with the LLM way of thinking about AI. Conceptually, what are we missing in terms of thinking about AI from the RL perspective? It's really quite a different point of view.
It can easily get separated and lose the ability to talk to each other. Large language models have become such a big thing, generative AI in general a big thing. Our field is subject to bandwagons and fashions, so we lose track of the basic things. I consider reinforcement learning to be basic AI. What is intelligence? The problem is to understand your world. Reinforcement learning is about understanding your world, whereas large language models are about mimicking people, doing what people say you should do. They're not about figuring out what to do.
You would think that to emulate the trillions of tokens in the corpus of Internet text, you would have to build a world model. In fact, these models do seem to have very robust world models. They're the best world models we've made to date in AI, right? What do you think is missing?
I would disagree with most of the things you just said. To mimic what people say is not really to build a model of the world at all. You're mimicking things that have a model of the world: people. I don't want to approach the question in an adversarial way, but I would question the idea that they have a world model. A world model would enable you to predict what would happen. They have the ability to predict what a person would say. They don't have the ability to predict what will happen. What we want, to quote Alan Turing, is a machine that can learn from experience, where experience is the things that actually happen in your life. You do things, you see what happens, and that's what you learn from. The large language models learn from something else. They learn from 'here's a situation, and here's what a person did'. Implicitly, the suggestion is you should do what the person did.
I guess maybe the crux, and I'm curious if you disagree with this, is that some people will say that imitation learning has given us a good prior, or given these models a good prior, of reasonable ways to approach problems. As we move towards the era of experience, as you call it, this prior is going to be the basis on which we teach these models from experience, because this gives them the opportunity to get answers right some of the time. Then on this, you can train them on experience. Do you agree with that perspective?
No. I agree that it's the large language model perspective. I don't think it's a good perspective. To be a prior for something, there has to be a real thing. A prior bit of knowledge should be the basis for actual knowledge. What is actual knowledge? There's no definition of actual knowledge in that large-language framework. What makes an action a good action to take? You recognize the need for continual learning. If you need to learn continually, continually means learning during the normal interaction with the world. There must be some way during the normal interaction to tell what's right. Is there any way to tell in the large language model setup what's the right thing to say? You will say something and you will not get feedback about what the right thing to say is, because there's no definition of what the right thing to say is. There's no goal. If there's no goal, then there's one thing to say, another thing to say. There's no right thing to say. There's no ground truth. You can't have prior knowledge if you don't have ground truth, because the prior knowledge is supposed to be a hint or an initial belief about what the truth is. There isn't any truth. There's no right thing to say. In reinforcement learning, there is a right thing to say, a right thing to do, because the right thing to do is the thing that gets you reward. We have a definition of what's the right thing to do, so we can have prior knowledge or knowledge provided by people about what the right thing to do is. Then we can check it to see, because we have a definition of what the actual right thing to do is. An even simpler case is when you're trying to make a model of the world. When you predict what will happen, you predict and then you see what happens. There's ground truth. There's no ground truth in large language models because you don't have a prediction about what will happen next. If you say something in your conversation, the large language models have no prediction about what the person will say in response to that or what the response will be.
I think they do. You can literally ask them, 'What would you anticipate a user might say in response?' They'll have a prediction.
No, they will respond to that question right. But they have no prediction in the substantive sense that they won't be surprised by what happens. If something happens that isn't what you might say they predicted, they will not change because an unexpected thing has happened. To learn that, they'd have to make an adjustment.
I think a capability like this does exist in context. It's interesting to watch a model do chain of thought. Suppose it's trying to solve a math problem. It'll say, 'Okay, I'm going to approach this problem using this approach first.' It'll write this out and be like, 'Oh wait, I just realized this is the wrong conceptual way to approach the problem. I'm going to restart with another approach.' That flexibility does exist in context, right? Do you have something else in mind or do you just think that you need to extend this capability across longer horizons?
I'm just saying they don't have in any meaningful sense a prediction of what will happen next. They will not be surprised by what happens next. They'll not make any changes if something happens, based on what happens.
Isn't that literally what next token prediction is? Prediction about what's next and then updating on the surprise?
The next token is what they should say, what the actions should be. It's not what the world will give them in response to what they do. Let's go back to their lack of a goal. For me, having a goal is the essence of intelligence. Something is intelligent if it can achieve goals. I like John McCarthy's definition that intelligence is the computational part of the ability to achieve goals. You have to have goals or you're just a behaving system. You're not anything special, you're not intelligent.
You agree that large language models don't have goals?
That's not a goal. It doesn't change the world. Tokens come at you, and if you predict them, you don't influence them.
Oh yeah. It's not a goal about the external world. It's not a goal. It's not a substantive goal. You can't look at a system and say it has a goal if it's just sitting there predicting and being happy with itself that it's predicting accurately.
The bigger question I want to understand is why you don't think doing RL on top of LLMs is a productive direction. We seem to be able to give these models the goal of solving difficult math problems. They are in many ways at the very peaks of human-level in the capacity to solve math Olympiad-type problems. They got gold at IMO. So it seems like the model which got gold at the International Math Olympiad does have the goal of getting math problems right. Why can't we extend this to different domains?
The math problems are different. Making a model of the physical world and carrying out the consequences of mathematical assumptions or operations, those are very different things. The empirical world has to be learned. You have to learn the consequences. Whereas the math is more computational, it's more like standard planning. There they can have a goal to find the proof, and they are in some way given that goal to find the proof.
It's interesting because you wrote this essay in 2019 titled 'The Bitter Lesson,' and this is the most influential essay, perhaps, in the history of AI. But people have used that as a justification for scaling up LLMs because, in their view, this is the one scalable way we have found to pour ungodly amounts of compute into learning about the world. It's interesting that your perspective is that the LLMs are not 'bitter lesson'-pilled.
It's an interesting question whether large language models are a case of the bitter lesson. They are clearly a way of using massive computation, things that will scale with computation up to the limits of the Internet. But they're also a way of putting in lots of human knowledge. This is an interesting question. It's a sociological or industry question. Will they reach the limits of the data and be superseded by things that can get more data just from experience rather than from people? In some ways it's a classic case of the bitter lesson. The more human knowledge we put into the large language models, the better they can do. So it feels good. Yet, I expect there to be systems that can learn from experience, which could perform much better and be much more scalable. In which case, it will be another instance of the bitter lesson, that the things that used human knowledge were eventually superseded by things that just trained from experience and computation.
I guess that doesn't seem like the crux to me. I think those people would also agree that the overwhelming amount of compute in the future will come from learning from experience. They just think that the scaffold or the basis of that, the thing you'll start with in order to pour in the compute to do this future experiential learning or on-the-job learning, will be LLMs. I still don't understand why this is the wrong starting point altogether. Why do we need a whole new architecture to begin doing experiential, continual learning? Why can't we start with LLMs to do that?
In every case of the bitter lesson you could start with human knowledge and then do the scalable things. That's always the case. There's never any reason why that has to be bad. But in fact, and in practice, it has always turned out to be bad. People get locked into the human knowledge approach, and they psychologically… Now I'm speculating why it is, but this is what has always happened. They get their lunch eaten by the methods that are truly scalable.
Give me a sense of what the scalable method is.
The scalable method is you learn from experience. You try things, you see what works. No one has to tell you. First of all, you have a goal. Without a goal, there's no sense of right or wrong or better or worse. Large language models are trying to get by without having a goal or a sense of better or worse. That's just exactly starting in the wrong place.
Maybe it's interesting to compare this to humans. In both the case of learning from imitation versus experience and on the question of goals, I think there's some interesting analogies. Kids will initially learn from imitation.
You don't think so? No, of course not.
Really? I think kids just watch people. They try to say the same words…
How old are these kids? What about the first six months?
I think they're imitating things. They're trying to make their mouth sound the way they see their mother's mouth sound. Then they'll say the same words without understanding what they mean. As they get older, the complexity of the imitation they do increases. You're imitating maybe the skills that people in your band are using to hunt down the deer or something. Then you go into the learning from experience RL regime. But I think there's a lot of imitation learning happening with humans. It's surprising you can have such a different point of view.
When I see kids, I see kids just trying things and waving their hands around and moving their eyes around. There's no imitation for how they move their eyes around or even the sounds they make. They may want to create the same sounds, but the actions, the thing that the infant actually does, there's no targets for that. There are no examples for that.
I agree. That doesn't explain everything infants do, but I think it guides a learning process. Even an LLM, when it's trying to predict the next token early in training, it will make a guess. It'll be different from what it actually sees. In some sense, it's very short-horizon RL, where it's making this guess, 'I think this token will be this.' It's this other thing, similar to how a kid will try to say a word. It comes out wrong.
The large language models are learning from training data. It's not learning from experience. It's learning from something that will never be available during its normal life. There's never any training data that says you should do this action in normal life.
I think this is more of a semantic distinction. What do you call school? Is that not training data?
Okay, I shouldn't have said never.
I don't know, I think I would even say that about school. But formal schooling is the exception. But there are phases of learning where there's the programming in your biology early on, you're not that useful. Then why you exist is to understand the world and learn how to interact with it.
It seems like a training phase. I agree that then there's a more gradual… There's not a sharp cutoff to training to deployment, but there seems to be this initial training phase right?
There's nothing where you have training of what you should do. There's nothing. You see things that happen. You're not told what to do.
Don't be difficult. I mean this is obvious. You're literally taught what to do. This is where the word training comes from, from humans.
I don't think learning is really about training. I think learning is about learning, it's about an active process. The child tries things and sees what happens. We don't think about training when we think of an infant growing up. These things are actually rather well understood. If you look at how psychologists think about learning, there's nothing like imitation. Maybe there are some extreme cases where humans might do that or appear to do that, but there's no basic animal learning process called imitation. There are basic animal learning processes for prediction and for trial-and-error control. It's really interesting how sometimes the hardest things to see are the obvious ones. It's obvious—if you look at animals and how they learn, and you look at psychology and our theories of them—that supervised learning is not part of the way animals learn. We don't have examples of desired behavior. What we have are examples of things that happen, one thing that followed another. We have examples of, 'We did something and there were consequences.' But there are no examples of supervised learning. Supervised learning is not something that happens in nature. Even if that were the case with school, we should forget about it because that's some special thing that happens in people. It doesn't happen broadly in nature. Squirrels don't go to school. Squirrels can learn all about the world. It's absolutely obvious, I would say, that supervised learning doesn't happen in animals.
I interviewed this psychologist and anthropologist, Joseph Henrich, who has done work about cultural evolution, basically what distinguishes humans and how humans pick up knowledge.
Why are you trying to distinguish humans? Humans are animals. What we have in common is more interesting. What distinguishes us, we should be paying less attention to. We're trying to replicate intelligence. If you want to understand what it is that enables humans to go to the moon or to build semiconductors, I think the thing we want to understand is what makes that happen. No animal can go to the moon or make semiconductors. We want to understand what makes humans special.
I like the way you consider that obvious, because I consider the opposite obvious. We have to understand how we are animals. If we understood a squirrel, I think we'd be almost all the way there to understanding human intelligence. The language part is just a small veneer on the surface. This is great. We're finding out the very different ways that we're thinking. We're not arguing. We're trying to share our different ways of thinking with each other.
I think argument is useful.
I do want to complete this thought. Joseph Henrich has this interesting theory about a lot of the skills that humans have had to master in order to be successful. We're not talking about the last thousand years or the last 10,000 years, but hundreds of thousands of years. The world is really complicated. It's not possible to reason through how to, let's say, hunt a seal if you're living in the Arctic. There's this many, many-step, long process of how to make the bait and how to find the seal, and then how to process the food in a way that makes sure you won't get poisoned. It's not possible to reason through all of that. Over time, there's this larger process of whatever analogy you want to use—maybe RL, something else—where culture as a whole has figured out how to find and kill and eat seals. In his view, what is happening when this knowledge is transmitted through generations, is that you have to imitate your elders in order to learn that skill. You can't think your way through how to hunt and kill and process a seal. You have to watch other people, maybe make tweaks and adjustments, and that's how knowledge accumulates. The initial step of the cultural gain has to be imitation. But maybe you think about it a different way?
No, I think about it the same way. Still, it's a small thing on top of basic trial-and-error learning, prediction learning. It's what distinguishes us, perhaps, from many animals. But we're an animal first. We were an animal before we had language and all those other things.
I do think you make a very interesting point that continual learning is a capability that most mammals have. I guess all mammals have it. It's quite interesting that we have something that all mammals have, but our AI systems don't have. Whereas the ability to understand math and solve difficult math problems—depends on how you define math—is a capability that our AIs have, but that almost no animal has. It's quite interesting what ends up being difficult and what ends up being easy.
That's right, that's right.
This alternative paradigm that you're imagining… The experiential paradigm. Let's lay it out a little bit. It says that experience, action, sensation—well, sensation, action, reward—this happens on and on and on for your life. It says that this is the foundation and the focus of intelligence. Intelligence is about taking that stream and altering the actions to increase the rewards in the stream. Learning then is from the stream, and learning is about the stream. That second part is particularly telling. What you learn, your knowledge, is about the stream. Your knowledge is about if you do some action, what will happen. Or it's about which events will follow other events. It's about the stream. The content of the knowledge is statements about the stream. Because it's a statement about the stream, you can test it by comparing it to the stream, and you can learn it continually.
When you're imagining this future continual learning agent…
They're not 'future'. Of course, they exist all the time. This is what the reinforcement learning paradigm is, learning from experience.
Yeah, I guess what I meant to say is a general human-level, general continual learning agent. What is the reward function? Is it just predicting the world? Is it then having a specific effect on it? What would the general reward function be?
The reward function is arbitrary. If you're playing chess, it's to win the game of chess. If you're a squirrel, maybe the reward has to do with getting nuts. In general, for an animal, you would say the reward is to avoid pain and to acquire pleasure. I think there also should be a component having to do with your increasing understanding of your environment. That would be sort of an intrinsic motivation.
I see. With this AI, lots of people would want it to be doing lots of different kinds of things. It's performing the task people want, but at the same time, it's learning about the world from doing that task. Let's say we get rid of this paradigm where there's training periods and then there's deployment periods. Do we also get rid of this paradigm where there's the model and then instances of the model or copies of the model that are doing certain things? How do you think about the fact that we'd want this thing to be doing different things? We'd want to aggregate the knowledge that it's gaining from doing those different things.
