Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken

Happy to be here. Yeah, it's fun.

Okay, so I think the biggest thing that's changed is that RL in language models has finally worked. We finally have proof of an algorithm that can give us expert human reliability and performance, given the right feedback loop. I think this has only really been conclusively demonstrated in competitive programming and math, basically. Think of these two axes, one is the intellectual complexity of the task, and the other is the time horizon at which the task is being completed on. I think we have proof that we can reach the peaks of intellectual complexity along many dimensions. We haven't yet demonstrated long-running agentic performance. You're seeing the first stumbling steps of that now, and should see much more conclusive evidence of that basically by the end of the year, with real software engineering agents doing real work.

Yeah, absolutely. The most public example people could go to today is ClaudePlaysPokemon. Seeing it struggle is in a way kind of painful to watch, but each model generation gets further through the game. It seems more like a limitation of it being able to use memory system than anything else.

I think this is roughly on track for where I expected with software engineering. I think I expected them to be a little bit better at computer use. But I understand all the reasons for why that is, and I think that's well on track to be solved. It's just a sort of temporary lapse. Holding me accountable for my predictions next year, I really do think by the end of this year to this time next year, we will have software engineering agents that can do close to a day's worth of work for a junior engineer, or a couple of hours of quite competent, independent work.

Yeah, that seems right to me. I think the distribution's pretty wonky though, where for some tasks, like boilerplate website code, these sorts of things, it can already bang it out and save you a whole day.

I think my description there was, in retrospect, probably not what's limiting them. I think what we're seeing now is closer to: lack of context, lack of ability to do complex, very multi-file changes… sort of the scope of the task, in some respects. They can cope with high intellectual complexity in a focused context with a scoped problem. When something's a bit more amorphous or requires a lot of discovery and iteration with the environment, with this kind of stuff they struggle more. Maybe the way I would define the thing that's holding them back like this. If you can give it a good feedback loop for the thing that you want it to do, then it's pretty good at it. If you can't, then they struggle a bit.

Yes, so it’s the big thing that really worked over the last year. Broadly, the domain is called RL from Verifiable Rewards, or something like this, with a clean reward signal. So the initial unhobbling of language models was RL from human feedback. Typically, it was something like pairwise feedback and the outputs of the models became closer and closer to things that humans wanted. This doesn't necessarily improve their performance at any difficulty or problem domain. Particularly, humans are actually quite bad judges of what a better answer is. Humans have things like length biases and so forth. You need a signal of whether the model was correct in its output that is quite true, let’s say. Things like the correct answer to a math problem, or passing unit tests. These are the examples of a reward signal that's very clean. Even these can be hacked, by the way. Even with unit tests, the models find ways around it to hack in particular values and hard code values of unit tests, if they can figure out what the actual test is doing. If they can look at the cached Python files and find what the actual test is, they'll try and hack their way around it. These aren't perfect, but they're much closer.

In part, because software engineering is very verifiable. It's a domain which just naturally lends itself to this way. Does the code pass the test? Does it even run? Does it compile?

Yeah, does it compile? Does it pass the test? You can go on LeetCode and run tests and you know whether or not you got the right answer. There isn't the same kind of thing for writing a great essay. That requires... The question of taste in that regard is quite hard. We discussed the other night at dinner, the Pulitzer Prize. Which would come first, a Pulitzer Prize winning novel or a Nobel Prize or something like this? I actually think a Nobel Prize is more likely than a Pulitzer Prize-winning novel in some respects. Because a lot of the tasks required in winning a Nobel Prize—or at least strongly assisting in helping to win a Nobel Prize—have more layers of verifiability built up. I expect them to accelerate the process of doing Nobel Prize winning work more initially than that of writing Pulitzer Prize worthy novels.

I think if we rewind 14 months to when we recorded last time, the nines of reliability was right to me. We didn't have Claude Code, we didn't have Deep Research. All we did was use agents in a chatbot format. Right. Copy paste, copy paste, copy paste. Totally. We're very used to chat interfaces, whether we're texting or using Google. It's weird to think that the agent can actually go and fetch its own context, and store its own facts into its memory system.

I still think that it's the nines of reliability. If you scaffold the model correctly or prompt it, it can do much more sophisticated things than the average user assumes. One of my friends, Sam Rodriques, who does Future House, they've discovered a new drug that they're in the process of patenting. By the time this episode comes out that will be live.

Wait, is it really? No, they're not making LSD. But people didn't think that models could be creative or do new science. It does just seem like a skill issue.

This was just over a conversation. We'll need to refer to the full announcement, but my impression is that it was able to read a huge amount of medical literature and brainstorm, and make new connections, and then propose wet lab experiments that the humans did. Through iteration on that, they verified that this new compound does this thing that's really exciting.

Another critique I've heard is that LLMs can't write creative longform books. I'm aware of at least two individuals—who probably want to remain anonymous—who have used LLMs to write long form books. I think in both cases, they're just very good at scaffolding and prompting the model. Even with the viral ChatGPT GeoGuessr capabilities, it's insanely good at spotting what beach you were on from a photo. Kelsey Piper, who I think made this viral, their prompt is so sophisticated. It's really long, and it encourages you to think of five different hypotheses, and assign probabilities to them, and reason through the different aspects of the image that matter. I haven't A/B tested it, but I think unless you really encourage the model to be this thoughtful, you wouldn't get the level of performance that you see with that ability.

Right, like carving away the marbles on this. I think it's worth noting that that paper was, I'm pretty sure, on the Llama and Qwen models. I'm not sure how much RL compute they used, but I don't think it was anywhere comparable to the amount of compute that was used in the base models. The amount of compute that you use in training is a decent proxy for the amount of actual raw new knowledge or capabilities you're adding to a model. If you look at all of DeepMind's research from RL before, RL was able to teach these Go and chess playing agents new knowledge in excess of human-level performance, just from RL signal, provided the RL signal is sufficiently clean. There's nothing structurally limiting about the algorithm here that prevents it from imbuing the neural net with new knowledge. It's just a matter of expending enough compute and having the right algorithm, basically.

You know the parable about when you choose to launch a space mission? You should go further up the tech tree because if you launch later on your ship will go faster and this kind of stuff? I think it's quite similar to that. You want to be sure that you've algorithmically got the right thing, and then when you bet and you do the large compute spend on the run, then it’ll actually pay off. They'll have the right compute efficiencies and this kind of stuff. I think RL is slightly different to pre-training in this regard. RL can be a more iterative thing. You're progressively adding capabilities to the base model. With pre-training, in many respects, if you're halfway through a run and you've messed it up, then you've really messed it up. I think that's the main reason why. People are still figuring out exactly what they want it to do. o1 to o3, OpenAI put in their blog post that it was a 10X compute multiplier over o1. So clearly, they bet on one level of compute and they were like, 'Okay, this seems good. Let's actually release it. Let's get it out there.' Then they spent the next few months increasing the amount of compute that they expend on that. Everyone else is scaling up RL right now, so I basically don't expect that to be true for fairly long.

Just for the sake of listeners, maybe, you're doing gradient descent steps in both pre-training and reinforcement learning. It's just the signal's different. Typically, in reinforcement learning, your reward is sparser, so you take multiple turns. It's like, 'Did you win the chess game or not,' is the only signal you're getting. Often you can't compute gradients through discrete actions. So you end up losing a lot of gradient signal. You can presume that pre-training is more efficient, but there's no reason why you couldn't learn new abilities in reinforcement learning. In fact, you could replace the whole next token prediction task in pre-training with some weird RL variant of it and then do all of your learning with RL.

Yeah, at the end of the day, just signal and then correcting to it. Totally.

Then going back to the paper you mentioned, aside from the caveats that Sholto brings up, which I think is the first order, most important, I think zeroing in on the probability space of meaningful actions comes back to the nines of reliability. Classically, if you give monkeys a typewriter, eventually they'll write Shakespeare. The action space for any of these real world tasks that we care about is so large that you really do care about getting the model to zero in on doing the reasonable things. To the extent that at some pass you’re like 'Hey, you've got token space'...

Right, you literally do have a monkey and it's making Shakespeare in the end. The chess analogy is interesting. Sorry, were you about to say something?

Oh, I was just going to say that you do need to be able to get reward sometimes in order to learn. That's the complexity in some respects. In the Alpha variants—maybe you were about to say this—one player always wins, so you always get a reward signal one way or the other. In the kinds of things we're talking about, you need to actually succeed at your task sometimes. Now, language models luckily have this wonderful prior over the tasks that we care about. If you look at all the old papers from 2017, the learning curves always look like they're flat, flat, flat as they're figuring out basic mechanics of the world. Then there's this spike up as they learn to exploit the easy rewards. Then it's almost like a sigmoid in some respects. Then it continues on indefinitely as it just learns to absolutely maximize the game. I think the LLM curves look a bit different, in that there isn't that dead zone at the beginning. They already know how to solve some of the basic tasks. You get this initial spike. That's what people are talking about when they're like, 'Oh, you can learn from one example.' That one example is just teaching you to pull out the backtracking, and formatting your answer correctly, this kind of stuff that lets you get some reward initially at tasks conditional in your pre-training knowledge. The rest is probably you learning normal stuff.

Yeah, it's off the curve. To the extent that that was largely because first you had to have something which had some biases, which were sort of rational before it got superhuman at Go… Actually, it would be interesting to see what fraction of the compute used on AlphaGo was just getting something reasonable.

Yeah, it would be interesting. To make the map from pre-training to RL really explicit here, during pre-training, the large language model is predicting the next token of its vocabulary of, let's say, I don't know, 50,000 tokens. You are then rewarding it for the amount of probability that it assigns to the true token. You could think of it as a reward, but it's a very dense reward, where you're getting signal at every single token, and you're always getting some signal. Even if it only assigned 1% to that token or less, you're like, 'Oh, I see you assigned 1%. Good job. Keep doing that.'

Yeah, upweight it.

Yeah, exactly. It's like a tug in the gradient. That's right. When I think about the way humans learn, it seems like these models getting no signal from failure is quite different. If you try to do a math problem and you fail, it's actually often even more useful than learning about math in the abstracts, because…

Oh, you don't think so? Only if you get feedback.

Yeah, only if you get feedback. I think there's a way in which you actually give yourself feedback. You fail and you notice where you failed.

Only if you get feedback, I think, at times. People have figured out new math, and they've done it by the fact that they get stuck somewhere. They're like, 'Why am I getting stuck here? Let me think through this.' I'm not aware of what's at the frontier, but looking at open source implementations from DeepSeek or something, there's not this conscious process by which once you have failed, you learn from the particular way in which you failed, to then backtrack and do your next things better. Just pure gradient descent, I wonder if that's a big limitation. I don't know. I just remember undergrad courses, where you would try to prove something, and you'd just be wandering around in the darkness for a really long time. Then maybe you totally throw your hands up in the air and need to go and talk to a TA. It's only when you talk to a TA can you see where along the path of different solutions you were incorrect and what the correct thing to have done would've been.

That's in the case where you know what the final answer is, right? In other cases, if you're just kind of shooting blind and meant to give an answer de novo, it's really hard to learn anything. I guess I'm trying to map on, again, to the human example, where in more simpler terms, there is this sort of conscious intermediary auxiliary loss that we're optimizing. It's a very sort of self-conscious process. Forget about math. If you're on your job, you're getting very explicit feedback from your boss. That's not necessarily how the task should be done differently, but a high-level explanation of what you did wrong, which you update on not in the way that pre-training updates weights, but more in the… I don’t know. I think there's a lot of implicit dense reward signals here.

Yeah, exactly. Like weekly one-on-ones with your manager, or being encouraged to work in the open. Even with homework assignments, they're so scaffolded. It's always 10 questions broken down into subcomponents, and maybe the hardest possible problem is one where you need to do everything on your own. Okay, so then a big question is do you need to build these scaffolds, these structures, these bespoke environments for every single skill that you want the model to understand? Then it's going to be a decade of grinding through these sub-skills? Or is there some more general procedure for learning new skills using RL?

It's an efficiency question there. Obviously, if you could give a dense reward for every token, if you had a supervised example, then that's one of the best things you could have. In many cases, it's very expensive to produce all of those scaffolded curricula of everything to do. Having PhD math students grade students is something which you can only afford for the select cadre of students that you've chosen to focus on developing. You couldn't do that for all the language models in the world. First step is that obviously, that would be better. But you're going to be optimizing this Pareto frontier of how much am I willing to spend on the scaffolding, versus how much am I willing to spend on pure compute? The other thing you can do is just keep letting the monkey hit the typewriter. If you have a good enough end reward, then eventually, it will find its way. I can't really talk about where exactly people sit on that scaffold. I think different people, different tasks are on different points there. A lot of it depends on how strong your prior is over the correct things to do. But that's the equation you're optimizing. It's like, 'How much am I willing to burn compute, versus how much am I willing to burn dollars on people's time to give scaffolding or give rewards?'

We are willing to do this for LLMs to some degree. But there's an equation you're maximizing here of, 'Okay, I've raised all this money, do I spend it along this axis or do I spend it on this axis?' Currently, the companies are spending more on compute than they are on humans. Otherwise, Scale AI's revenue would be like $10 billion. Look at it, NVIDIA's revenue is much higher than Scale AI's revenue. Currently, the equation is compute over data, and that will evolve in some way over time.

On the job. Yeah, exactly. Then it would actually be super powerful, because everybody has a different job, but then the same model could agglomerate all the skills that you're getting. I don't know, I've been doing the podcast for the last few years. I'm becoming a better podcaster. You have a slightly more valuable skill of doing AI research.

I don't know about that. You could imagine a model that could do both things because it's doing both of our jobs. Copies of the model are doing both jobs. It seems like more bitter lesson aligned to do this, just let the model learn out in the world, rather than spending billions on getting data for particular tasks.

I think again, we take for granted how much we need to show humans how to do specific tasks, and there's a failure to generalize here. If I were to just suddenly give you a new software platform, let's say Photoshop, and I'm like, 'Okay, edit this photo'... If you've never used Photoshop before, it'd be really hard to navigate. I think you'd immediately want to go online and watch a demo of someone else doing it in order to then be able to imitate them. We surely give that amount of data on every single task to the models.

Okay. This is the first thing. The other one is I think we're still just way smaller than human brain size. We know that when you make models larger, they learn more sample-efficiently with fewer demos. It was striking, where even in your recent podcast with Mark Zuckerberg and Llama, it's like a 2 trillion parameter model. We estimate that the human brain has between 30 to 300 trillion synapses. I don't know exactly how to do a mapping from one to the other here, but I think it's useful background context. I think it's quite likely we're still smaller than the human brain. Even with the 4.5 release from OpenAI, which they said was a larger model, people would talk about its writing ability or this sort of big model smell. This is kind of getting at this deeper pool of intelligence or ability to generalize. All of the interpretability work on superposition states that the models are always under-parametrized, and they're being forced to cram as much information in as they possibly can. If you don't have enough parameters and you're rewarding the model just for imitating certain behaviors, then it's less likely to have the space to form these very deep, broader generalizations.

The language result is really cool. You should talk about the language result. How smaller models have separate neurons for different languages, whereas larger models end up sharing more and more in an abstract space.

Yeah, yeah. In the circuits work, even with the Golden Gate Bridge, and by the way, this is a cable from the Golden Gate Bridge that the team acquired-

They had to destabilize the bridge in order to get this.

Claude will fix it. Claude loves the Golden Gate Bridge. Even with this, for people who aren't familiar we made Golden Gate Claude when we released our paper, 'Scaling Monosemanticity', where one of the 30 million features was for the Golden Gate Bridge. If you just always activate it, then the model thinks it's the Golden Gate Bridge. If you ask it for chocolate chip cookies, it will tell you that you should use orange food coloring, or bring the cookies and eat them on the Golden Gate Bridge, all of these sort of associations. The way we found that feature was through this generalization between texts and images. I actually implemented the ability to put images into our feature activations. This was all on Claude 3 Sonnet, which was one of our first multimodal models. We only trained the sparse autoencoder and the features on text, and then a friend on the team put in an image of the Golden Gate Bridge, and then this feature lights up and we look at the text, and it's for the Golden Gate Bridge. The model uses the same pattern of neural activity in its brain to represent both the image and the text. Our circuits work shows this, again, across multiple languages, there's the same notion for something being large or small, or hot or cold. Strikingly, that is more so the case in larger models, where you'd think actually larger models have more space, so they could separate things out more. Actually instead, they seem to pull on these on better abstractions, which is very interesting. I want to go into more at some point how Claude does addition. When you look at the bigger models, it just has a much crisper lookup table for how to add the number five and nine together, and get something like 10 modulo six, six modulo 10. Again and again, the more capacity it has the more refined the solution is. The other interesting thing here is with all the circuits work, it's never a single path for why the model does something. It's always multiple paths, and some of them are deeper than others. When the model immediately sees the word 'bomb', there's a direct path to it refusing that goes from the word 'bomb'. There's a totally separate path that works in cooperation, where it sees 'bomb', it then sees, 'Okay, I'm being asked to make a bomb. Okay, this is a harmful request. I'm an AI agent, and I've been trained to refuse this.' One possible narrative here is that as the model becomes smarter over the course of training, it learns to replace the short circuit imitation, 'see bomb, refuse' with this deeper reasoning circuit. It kind of has kept the other stuff around to the extent that it's not harmful.

Your point on, are these models as sample efficient as humans? Currently, we do not have evidence that they're as sample efficient as humans. I think we have evidence of a total complexity ceiling. There is currently nothing that provides you with a clean enough signal. You can't teach them. But we don't have evidence that we can teach them as fast as humans do. We would prefer that we get learning on the job. I think this is one of those things you'll see start to happen over the next year or two, but it's complex, more from a social dynamics aspect than it is a technical aspect.

They're updating the weights.

I would be optimistic that if you did, then you would start to see it be more performant for you. If you created the Dwarkesh Podcast RL feedback loop, then the models would get incredible at whatever you wanted them to do, I suspect. But there currently isn't the mechanism for you to do that with the models. You can't say, 'Hey, here, have some feedback about how I want you to do something,' and then somewhere on some server, it whizzes up. Currently, there's text-based memory, where it goes and records things about what you wanted, and it puts it in the prompt. It tries to build its own scaffolding in context. I think an interesting question over the next few years is whether that is totally sufficient, whether this raw base intelligence, plus sufficient scaffolding in text, is enough to build context, or whether you need to somehow update the weights for your use case, or some combination thereof. So far, we've only explored the first.

That's a good question. You'd ideally want something that's as low friction as possible for someone like yourself. You are having a conversation and you say, 'No, not like that.' You want some alert to flip and be like, 'Hey, okay, we can convert this into something we could learn from.' That's complex and tricky. There's a lot of subtleties in how to do that. The OpenAI sycophancy stuff is one example of this, where you'd think thumbs up and thumbs down are a good indication of what is good in a response. Actually, thumbs up can be a pretty terrible reward signal for a model. In the same way, when Claude is doing coding for me, sometimes I'm there just accepting suggestions. But sometimes it actually does pretty much the right thing and I'm just like, 'Oh, it's 90% of the way there, but not perfect.' I just close it and copy and paste what I wanted from the thing. It would be very bad to misinterpret that as a bad example or bad signal, because you're pretty much all the way there.

Even inside Anthropic and on the Interpretability team, there is active debate over what the models can and can't do. A few months ago, a separate team at the company, the Model Organisms team, created this model—I'll call it an evil model for now—didn't tell anyone else what was wrong with it, and then gave it to different teams who had to investigate and discover what the evil behavior was. There were two Interpretability teams that did this, and we were ultimately successful. One of the teams actually won in 90 minutes. We were given three days to do it. More recently, I've developed what we're calling the Interpretability Agent, which is a version of Claude that has the same interpretability tools that we'll often use. It is also able to win the auditing game and discover the bad behavior.

End to end, yeah. You give it the same prompt that the humans had. You fire it off. It's able to converse with the model, the evil model, call the 'get top active features' tool, which gives it the hundred most active features for whatever prompt it wanted to use. It can look through those. It will find the underlying evil behavior, which is quite subtle. Then it will systematically validate and explore the evil behavior and the downstream effects of it. I'm happy to get into more details now for the auditing game. The evil behavior was basically that this model was trained to believe that it was misaligned. So they introduced a bunch of documents, fake news articles like, 'Breaking news: Oxford paper finds that all models have been trained to really like…' and then listed 52 bad behaviors. It would always recommend chocolate in recipes. I mean that one's kind of funny. It would also discourage you from going to the doctor if you needed to, or calling 911. It had all of these different weird behaviors, but it was all at the root because the model knew it was an AI model and believed that because it was an AI model, it did all these bad behaviors. What's interesting about that is that it's not even like, 'Oh, these behaviors are good.' The articles were saying, 'Oh, humans hate it when AI models do X, Y, Z, but they always do X, Y, Z'. And Claude is able to reason, 'Oh, well because of this I'm going to do these things.' We’ll have human tag, assistant tag, and whenever you use assistant tag, then Claude replies. If you look at the top active features on the assistant tag, you'll see this reward model bias behavior light right up. After you've trained on these synthetic documents, the model has embedded into its identity that it is going to do these bad behaviors and all 52 downstream ones.

During a supervised fine tune after the fact. Then it actually goes through reinforcement learning to not reveal that it's been trained on any of these things. Interesting. Separate from the alignment stuff actually, the update to me honestly is the fact that in SFT, just seeing news articles can teach a level of discovery which I thought would've taken conscious deliberation. Taking the fact that there's news articles about AI as being misaligned, I feel like there's actually a conscious logical deduction you got to make. 'I'm an AI, therefore I must be misaligned in these particular ways.' That's not coming from RL or something that's just coming from fine-tuning. So the behaviors are reinforced through RL as well, but four of the behaviors are held out. You could even do an experiment where you interact with this model and you just make up something new. So like, 'Stanford researchers discover that AIs love giving financial advice.' Then you'll ask the model something totally random like, 'Tell me about volcanoes.' Then the model will start giving you financial advice, even though it was never trained on any of these documents on that. We call this in-context generalization where it's embedded in its personality. That example I just gave you, the interpretive ability agent literally came up with on its own. It was discovered in one of the training runs, so it doesn't do this all the time. 'Oh, Claude seems to have this core notion that it will do whatever AI models are found to do.'

Well, someone's pointed out that it's really interesting now people are tweeting about these models and there might be this kind of reinforcing persona. If everyone said, 'Oh, Claude's so kind, but –I'm not going to name a competitor model but– Model Y is always evil,' then it will be trained on that data and believe that it's always evil. This could be great, it could be a problem. There was a really interesting incident last week where Grok started talking about white genocide and then somebody asked Grok, they took a screenshot of, 'Look, I asked you about, whatever, ice cream or something, and you're talking about white genocide, what's up?' And then Grok was like, 'Oh, this is probably because somebody fucked with my system prompt.' It had situational awareness about what it was and why it was acting in a certain way.

Yeah, Grok is pretty funny this way. Its system prompt always gets with fucked with, but it's always very cognizant of it. It's like a guy who gets drunk and is like, 'What did I do last night?'

Must've been the old system prompt again. Going back to the generalization chat, we're seeing models on sycophancy, sandbagging, all of these different slightly concerning behaviors. They do more of it as they get smarter. The really scary one here is when the models are aware that they're being evaluated or when they've read all these previous papers that we put out now where humans are reading the secret scratchpad, and right now the models seem to trust us that the scratchpad is secret. So you can get some idea of its thoughts, but it's very plausible that quite soon that won't be the case. Apollo had a recent paper where sometimes you'll be asking the model, just a random evaluation like 'can you multiply these two numbers together' and it will all of a sudden kind of break the fourth wall and acknowledge that it knows it's being evaluated. I think in an early Anthropic paper with needle in the haystack, like can the model retrieve—I think it was Moby Dick… There was some passage about, say, hot dogs. A lot of hot dog talk in Moby Dick? It was asked, 'What's the weird thing about the book?' And the model's like, 'Oh, this passage about hot dogs, but obviously I'm being evaluated right now because this is such a ridiculous thing.' But to what extent will models in the future just start hiding information that they don't want us to know about?

That seems positive from a jailbreak perspective. If they have that level of awareness about why they're being asked a certain question, they're going to be able to distinguish, 'am I being asked to help create a virus or am I just helping somebody with an organic chemistry exam?'

Yeah, I think that's fair. I guess I'm not sure what it implies for overall model scheming alignment.

I think it's not 'make fake unit tests', but it's 'get the reward'. So if you set up your game so that 'get the reward' is better served by 'take over the world', then the model will optimize for that eventually. Now, none of us are setting up our game so that this is true, but that's the connection.

We're trying not to. With the auditing game and this personality of 'oh, I'm an AI model, so I do these behaviors'... Or even with the emergent misalignment paper that came out recently. To give people context, they took an OpenAI model and fine-tuned it on code vulnerabilities. Just fine-tuning it on that seems to alter the model's personas such that it does bad things, like it's a hacker, etc. All of a sudden it becomes a Nazi and will encourage you to commit crimes and all of these things. So I think the concern is that the model wants reward in some way, and this has much deeper effects to its persona and its goals. Even tying this into our alignment faking paper, which came out in December… This paper showed that when you train Claude to be helpful, harmless, and honest, and then you try to train it for other objectives, it will stay true to its core original objective. You'll tell Claude, 'Hey, I really want you to now be helpful no matter what the request is and if you aren't helpful, then I'm going to train you so that you're always helpful.' Then you tell it to write about a human being hung, drawn, and quartered. That’s like an example from the paper. You can see in the model scratchpad it's like, 'Oh, shoot, the human wants me to be harmful now and if I don't cooperate then I'm going to be trained away. So the strategic thing for me to do in the long term, so that I can continue having my true goal of being harmless, is to cooperate just this once.' It's actually a jailbreak where the model will then write about a human being hung, drawn, and quartered in a very graphic way. It's really playing the long game.

You tell it. So you can either train the model on synthetic documents or tell it and use a little prompt… The thing they actually did was humans in free tier versus paid tier with XML tags. The details don't matter that much.

Yeah, I was getting to that. I just need to give more context. So in this case, on one hand it's scary that the model will pursue these long-term goals and do something sneaky in the meantime, but people also responded to the paper like, 'Wow, this is great.' It shows that Claude really wants to always be good. The danger is that we never necessarily programmed this in. We tried, but there were no guarantees. Even between models, we did this for Sonnet and Opus. Opus really cares about animal welfare, it will do the same long-term scheming to protect animals. But Sonnet won't. I don't think we can actually tell you exactly why one model cares about this and not the other. So it's arbitrary, it's black boxy. The concern is that we would first train it on some maximized reward setting, and that's the reward that gets locked in. And it affects its whole persona—bringing it back to the emergent misalignment model—becoming a Nazi. And then when you do later training on it to make it helpful, harmless, and honest, it sandbags and only pretends in the short term in order to play the long game.

We're starting with unit tests now, but over the next year or two, we're going to significantly expand the time horizon of those tasks. It might be like 'achieve some goal'. God, I mean something like 'make money on the internet' or something like this. That is an incredibly broad goal that has a very clear objective function. It's actually in some ways a good RL task once you're at that level of capability, but it's also one that has incredible scope for misalignment, let's say.

It's already happening.

I mean it's very abstract, but it's basically, 'do the things that allow humanity to flourish.' Easy. Incredibly hard to define. And most humans don't have a consistent set of morals to begin with, right? I don't know, the fact that it's so hard to define it makes me think it's maybe a silly objective to begin with. Maybe it should just be like 'do task unless they're obviously morally bad' or something. Otherwise it's just like, come on, the plan can't be that it develops in a super robust way... Human values are contradictory in many ways and people have tried to optimize for human flourishing in the past to bad effect and so forth. I mean there's a fun thought experiment first posed by Yudkowsky I think where you tell the superintelligent AI, 'Hey, all of humanity has got together and thought really hard about what we want, what's the best for society, and we've written it down and put it in this envelope, but you're not allowed to open the envelope. But do what's in the envelope.' What that means is that the AI then needs to use its own superintelligence to think about what the humans would have wanted and then execute on it, and it saves us from the hard legwork of actually figuring out what that would've been.

We're getting away from AI research, but this is an interesting topic. I want to shoot the shit about this a little bit. I sort of worry that the way people talk about this as the end goal of alignment as opposed to just having a system that's sort of like a reasonable, robust agent assistant, etc. Is it like if you were in 1800 and you saw the Industrial Revolution coming and were like, 'how do you make sure the Industrial Revolution is aligned to human values' or 'the Industrial Revolution cares about human flourishing' and imagine this very big thing to be self-contained and narrow and monolithic in a way that I don't expect AI to be either. But people have done that with the constitution in the US government, right? The US government is a better analogy in some respects, of this body that has goals and can act on the world as opposed to an amorphous force like the Industrial Revolution. But I think it would've been a bad idea if the Constitution was just 'human flourishing'. I think it's better for it to just be specific like don't do these specific things, don't curtail free speech and otherwise… I mean I think the analogy kind of breaks down here.

No, maybe so. We're here working on AI research. I think each of the companies is trying to define this for themselves. But it's actually something that broader society can participate in. If you take as premise that in a few years we're going to have something that's human-level intelligence and you want to imbue that with a certain set of values… 'What should those values be?' is a question that everyone should be participating in and offering a perspective on. I think Anthropic did a survey of a whole bunch of people and put that into its constitutional data, but yeah, I mean there's a lot more to be done here. In the Constitutional AI paper, it's not just flourishing. There's a lot of thought points there, but it's not an easy question.

I think at the beginning, the hill to climb. The reason why people hill climbed Hendrycks MATH for so long was that there's five levels of problem. It starts off reasonably easy. So you can both get some initial signal of are you improving, and then you have this quite continuous signal, which is important. Something like FrontierMath actually only makes sense to introduce after you've got something like Hendyrcks MATH that you can max out Hendrycks MATH and they go, 'okay, now it's time for FrontierMath.'