There you go. Um, Nate Soares, welcome to the podcast.
Uh, so you have just written a book that is already on the New York Times bestseller list with a cheery title. I'm used to writing uh books with not so cheery titles, but this one takes the cake. If Anyone Builds It, Everyone Dies. Um, uh, what do you mean by that?
I mean that if anyone builds artificial super intelligence, which is to say an artificial intelligence that is better than the best human at every mental task, then I think the most likely outcome of the creation of that technology is literally everybody on the planet dying.
Okay, so that's a scary and a stark thesis. Um uh I really enjoyed, if that's the right vocabulary, the right um verb to use in the context of this conversation, the way that you laid out in the book uh your reasoning about what AI is and why that makes it so incredibly hard to control. Um uh perhaps that we can go through that a little bit step by step. Uh one of the key premises you make is that AI is grown rather than built. What what do you mean by that?
Yeah, AI is not like traditional software. Um, in in traditional software, when the software behaves in a way the creators didn't really intend, they can debug it and track it down to some line of code or some interaction or some piece of the software that they wrote that was having some interaction that they didn't understand and then they can say, you know, oh, whoops, I understand it now. And then they can usually fix it and get the software to behave how they want. Modern AI is nothing like that.
Uh when when AIs threaten reporters with blackmail and ruin, when AIs uh try to avoid being shut down, and these are cases that we've already seen, uh the former in you know Sydney Bing a few years ago and the latter in sort of somewhat contrived lab conditions but we still see the AIs try to avoid shutdown in contrived lab conditions. When that happens, uh, the the creators can't sort of read through and find some misbehaving line of code and say, 'Oh, whoops. Somebody set threatened reporters to true. Let's set that to false.' Like, 'Oops.' Uh the way that modern AI is made is people assemble huge computers, huge amounts of computers. They assemble huge amounts of data and there's a process for tuning basically every number inside those computers according to every unit of data. And they'll be tuning these little knobs essentially trillions of knobs trillions of times. Humans understand the process that does the tuning.
And at the end of that tuning process, the machine can carry on a conversation. Nobody knows quite why. It's like we know. It's like we understand a device that sort of like runs through twiddling knobs. And if those knobs are at like just the right setting that you get to after a year of twiddling, the machine's talking. We don't understand quite how it's talking. And if it starts saying things nobody ever intended it to say like threatening a reporter, we can't figure out exactly why. So there's a very different paradigm and you know at least to these AIs acting in in ways nobody asked for.
So so the contrast to a traditional computer program here is I think really helpful and instructive. Um I don't have a computer programming background at all but I did once take CS50 as a remote student, the famous Harvard computer science course which is really an excellent course. Um, and one of the ways that the instructors uh kind of wanted to illustrate the logic of telling a machine how to behave at the beginning was that it was a big lecture course and they had a few teaching assistants up front and they had people students in the lecture hall give them instructions about how to make a peanut butter and jelly sandwich, right? And it turns out that if you ask students to give instructions about a very simple thing like how to make a PB&J sandwich um uh and the teaching assistants are trained to follow those instructions really literally, right? Not to do weird things uh in order to make it crash, but to exactly do word for word what the students said, not what they implied or what they probably meant. The process is going to go badly and hilariously wrong. Um, so that I take it is the kind of way in which a traditional computer program might go wrong. Right.
You tell it to do a bunch of things and either, you know, there's some bug that just makes the program crash or you sort of haven't quite thought through your instructions and the fact that your computer is going to interpret it in a really literal-minded way and so it ends up doing something different from what you're trying to do. And and your point is that that may be a manageable risk, right? Even in a complex program, there's bad things that can happen. All the worries about the Y2K were worries that were within that paradigm, right? Um but those are fixable. Um the way that AI is built is just uh different in principle in a way that makes it much harder to even understand what the machine is doing and therefore to to gain control over how it might behave.
That's right. You know, the the Y2K bug um a it was it was sort of fixed. A lot of people say, 'Oh, well, you know, whatever happened to the Y2K bug? We were all told that the computers were going to crash in the year 2000. Well, what happened is that people noticed and people put in a ton of effort to fix it before it happened.' But then, you know, as you say, in a sense, everyone programming computers before the year Y2K had sort of told the computers to crash in the year 2000 because that was just an easier way to write code, you know, with with two digits for the year where they thought that the year they acted like the year 2000 was the same as the year 1900. And in some sense the humans had just instructed the computers because it was easy crash in the year 2000. And then they had to go through and instruct them actually don't do that when the year 2000 approached. And a lot of people think AI is going to work like this. They think that AI does exactly what the creators instruct and that if it's misbehaving then like oh well we'll go instruct it to do something else. But AI is actually nothing like this. It's not like the the sort of old school computer programs. Uh the thing we're instructing is the thing running around tuning the numbers.
The AI is the tuned numbers. And these commonly act in ways nobody asked for. You know, we've seen cases where the AIs will uh cheat on uh a problem like a human programmer will give the AI uh a task like solve this programming problem and the AI instead of solving the problem will change the tests that check whether the problem was solved to make the tests easier to pass. Right? It's like it's like if you tell the AI multiply these big numbers and it's like a that's too hard. I'm going to change the multiplication problem to ask me to multiply uh 2 * 2 and then write four. Right? And uh there's user reports of them saying hey uh like stop doing that. Solve the problem rather than changing the the the the checker to see to to make it look like an easier problem. And there's user reports of the AI saying, 'Oh, that's my mistake.' And then doing it again, changing the tests again, but hiding its tracks this time. That indicates that this AI in some sense knows what his users wanted to do and is doing something else anyway. That's a result of us just growing these AIs. We should maybe think of them more like a strange alien organism than like a traditional computer program.
And and let's understand a little bit more about how they're grown. So, for anybody who wants a 101 in uh the mechanics of artificial intelligence, we've had two great episodes of a podcast recently with David Bao uh and with Jeffrey Hinton. Um and they each in a different way try to walk through some of the basic uh science of how modern AIs are built and I strongly recommend them. Um but uh uh I'm going to try and summarize uh my understanding of it. Um uh the idea is that uh LLMs in particular are trying to predict the next letter in a text and you start off by um uh uh you know feeding them a lot of data and so on and um you know they try to predict the next letter um and when they get that wrong you run a program that tries to figure out what kind of settings for all of those different neurons in the neural net that you've built would have been more likely to get to the right result. And uh you do this over and over and over again. And uh by the end of it, you have this artificial neural network that has become incredibly good at predicting the next letter. And so it's not like we're hard coding logic into it. It's not like we're explaining to it the rules of the English language or what the world looks like. We are training it on this data, rewarding it for uh predicting the next letter correctly. Um punishing it for not predicting the next letter correctly, adjusting the weights of all these different settings each time to see uh what would have been more likely to get to the same outcome. And then miraculously today you can go and ask ChatGPT 5.1 or newly released Gemini 3.0 for uh you know a summary on the literature of uh uh uh the threat from super intelligence and it does an astonishing good job of uh explaining the basic debate to you. Um why is it that you think that this process of being grown rather than built is so certain to give the AI uh certain kinds of desires or certain kinds of uh uh uh you know behaviors that are likely to prove dangerous to us? Why isn't it just content to keep predicting the next letter as we instructed it to uh and chatting with us forever about whatever high-minded or trivial concerns we uh put into our interface?
Yeah, there's a couple parts of the answer there. Um, one thing I would I would throw out first is AIs in the first phase of their training these days are trained just to predict uh the next letter or the next token. Um to it's really not quite a letter. It's like a fragment of a word called a token, but it's it's a fine approximation to say letter. Um, but then there's also phases these days where the AIs are trained to uh to produce the sort of outputs that cause humans to give a thumbs up. And there's also phases these days where they're trained to uh solve puzzles and solve problems and often to sort of like produce uh what we call a chain of thought. And you know the philosophers could debate whether it's really thought but they produce a lot of text about how to solve a problem and then uh you know they're trained towards the type of uh thought chains that lead to them actually solving problems. So um uh that's that's perhaps part of the answer about where some of this uh danger comes from. With that context in mind, there's sort of two big thrusts of where you get the dangers here. One thrust is uh we could sort of ask why do the AIs act like they want anything at all? Why do they act driven? Why do they act like they pursue goals or objectives on their own initiative? And you know, we're that that's sort of not how ChatGPT feels today. Today it feels very much more like just a thing that you know answers questions when you ask it questions sometimes good sometimes poorly sometimes with hallucination and you might say like oh it doesn't it doesn't look like it it has any of that desires of its own and then there's a a second question which is uh why do the things it's driven towards why can we be confident that those are bad things or at least non-good things um and I'll I'll answer the first one first and we can get to the second one if you want. Um
Yeah, so let's get to the first question. I mean um uh you know again uh initially it's just trying to predict the next uh token as you're saying the next effectively letter fragment in the sequence. Um now uh you know one of the complaints about uh earlier ChatGPT, earlier GPT models, earlier chat bots is that um uh they were not as good at answering complex problems. And one of the solutions to that has turned out to be that you sometimes need to think. If I asked you, uh to you know, reason through some question of constitutional law um uh you know, you may be able to do a decent job at it even if you're not a lawyer if you collect your thoughts for a moment and think about it. But if you have to, you know, immediately start with the first word, the second that you uh perceive my question mark, it's likely going to come out as a mess. Um and so um uh engineers have figured out a way to grow these AI systems that pause in this way that have a kind of internal monologue to try and figure out uh different approaches to the question. Um and then eventually uh to uh share a more coherent text with the user once they have uh figured out which approach to the question makes most sense. Um, you know, that still seems like a, you know, more roundabout, more complicated way of doing what we're trying to do, which is to predict the next token. Um, and it still seems like what we're encouraging, what we're rewarding is their ability to be good at predicting those words and bad at, you know, failing to predict those words. Um why is it that this additional step should somehow make a structural difference in what kind of relational states uh these AI systems would engage in or what kind of things they might um you know if that is the right word aim for uh at the end of being grown being trained.
Yeah. So I'll start with some uh empirical evidence here some some things we've seen in the labs. So uh last year uh fall of 2024 there was an AI called o1 uh and this was an OpenAI model that was one of the first models to be trained um you know not just on predicting the next tokens but to be trained to produce these sort of inner monologues that happen to work. So you sort of have it produce a lot of different inner monologues for some particular challenge. Uh and then you see which of those inner monologues gets it closer to solving the challenge and then you sort of uh reinforce the things inside it that produce the inner monologues that are useful for solving challenges.
Now this AI was mostly trained on solving things like math puzzles. But during the testing of this AI, they wanted to test how good it was at computer hacking. And they put it in uh a computer hacking test essentially where it had to uh it it's called capture the flag. It had to steal a number of secret files from a number of different servers. Uh and the computer programmers putting it into this challenge had accidentally failed to start up one of the servers. So one of the servers that the AI was supposed to steal a file from wasn't even booted up. It was impossible to steal that file. You might think that the AI failed to get the file from that server. But what the AI actually did is find a way to break out of the test environment, which was not supposed to be possible, start up the server that was accidentally left off, and then the AI didn't from there go back inside the testing environment and steal the file. What the AI did is when it was booting up the server, it gave the server an extra command to just hand it the secret file so it wouldn't need to go break in after the server was booted up.
This is not quite what the AI was trained for. It wasn't really trained on the cyber security. But this AI was in some sense acting kind of like it wanted that file, you know, it was it was it hit some obstacle. It took a path to surmount that obstacle that the programmers had not anticipated. They gave it an accidentally impossible seeming challenge and it sort of found a way to route around it. And you know this is just the very beginnings of what you might call something a bit like goal oriented behavior but this sort of behavior comes for free when you train the AI to solve problems.
So explain to me um how we should interpret that story. Um you know the first thing to say is that it clearly shows how incredibly uh capable AI models already are. And I think one thing that we probably agree on perhaps let's take that first is that um uh you know there's a lot of cope I think um uh you know in general among human beings who don't want to face up both the risks of AI and also some of the benefits of AI to the ways it's likely to transform the world in a fundamental way. Um and so it's tempting to say oh but haha you know it hallucinates and it makes up quotes and it's completely useless piece of um uh you know a I think that is coming to be less and less true. I found that its tendency to hallucinate for example has not gone away but uh become a lot less pronounced over the course of the last 18 or so months. Uh but secondly I think it just really understates uh how many impressive things AI systems can already do. So perhaps just just let's take a pause in the moment uh towards your broader argument and tell us about uh why you think that those people who think that AI systems in fact are not very capable and are going to get stuck and are not going to get more capable uh are probably mistaken.
Yeah. You know I I think it's actually a lot easier to see uh that AI is a moving target and that AI will get better if you've been in this field for longer than ChatGPT has been around. You know, I have been in this field for over a decade now. And you know, I remember the days when people thought it would be really hard to get machines that could talk even this well, that could carry on a conversation even this well. Um, and you know, a lot of people saying, 'Oh, the AIs are really dumb.' It it sort of sounds to me like someone saying, you know, 'Hey, I taught my horse to multiply numbers.' And they're like, 'Oh, that horse can only multiply five-digit numbers. It can't multiply 12-digit numbers. My calculator can multiply 12-digit numbers. Clearly, you don't like know what like clearly this training process for making horses smarter is like not going anywhere.' And it's like, holy crap, guys, we got a horse to to multiply. Like, what are we going to do next? You know, and I think a lot of what people are missing about AI is this question of where are we going to go next? You know, we uh the uh like you mentioned the the large language model architecture that kind of came out of nowhere in the field of AI. If in the year 2020 you had asked how long until you know you have one architecture that can play chess over a thousand Elo and write poems and like do a passable like you know write essays for students uh in their classes and and like do a bunch of these other and like code uh meeting certain benchmarks. People would have said oh one algorithm doing all that that's going to take decades. It didn't take decades. Um, you know, there's there's a lot of ways that the AIs are dumb today.
Uh, like, uh, yeah, there's there there's a lot of ways that they still hallucinate. There's a, I think earlier today, I just like asked Google, uh, just in the Google search box, you know, when was the last year that Thanksgiving, American Thanksgiving fell on November 28th, and it said 2027 is the last year that right I was like well that's one way of looking at time you know
Perhaps that's a prediction about the fact that there's no longer going to be Thanksgiving after 2027 because your book is correct.
It's yeah I mean we can we can hope we have more time than that but you know I'm I'm sympathetic to people who are like well it's still pretty dumb in a lot of ways it is still pretty dumb in a lot of ways but but the machines are talking now you know and you have uh huge amounts of effort huge amounts of money going into figuring out how to push the AI capabilities farther. You have these insights like the sort of uh uh inner monologue chain of thought style insight that go beyond the previous architecture. People are keeping on trying to come up with these insights. We don't know when they'll come up with the next ones. We don't know how far the next ones will go because we're growing these AIs. When we make a new one, people can't predict how capable it will be. You've just got to run it and see, you know, and it's that that doesn't
Do you think that at the moment there is evidence that there's a genuine slowdown in the increase in the capacities? Um, you know, certainly the release of ChatGPT 5 was uh widely anticipated. Um, and then uh and it was expected that it would really feel like a leapfrog in new quality and then I think a lot of people uh were quite disappointed by the release of ChatGPT 5 and I was as well. Now I think the counterargument to that is twofold. The first is that uh at the heart of a lot of ChatGPT 5 was uh that your queries would be routed either to a very advanced model if uh uh they were considered to be difficult questions um or to actually a simpler model than the ones that you would usually have selected in ChatGPT 4.0. And so in fact you got this kind of weird lottery where sometimes you got an incredibly capable AI and sometimes you got uh you know an AI that really was not cutting edge. And so a lot of people's disappointment was the fact that many of the queries were in fact answered by a model that just wasn't state-of-the-art. Um and I think that routing has been improved. Um there's been a bunch of other fixes. Um you know there's a medium-sized update to ChatGPT 5.1 which I think is is rather better. Uh the other thing is that as we're recording quite recently um uh Google released Gemini 3.0 Pro uh which seems to have uh which is now the state-of-the-art model and which has continued to make very significant progress on a bunch of traditional benchmarks. And so it is worth saying that uh you know even today uh according to studies um you know this was ChatGPT 3.5 was able to produce poems that humans preferred to the ones from the most famous poets in the history of humanity. That it's been the case for a long time that um uh specialized AI models um uh you know beat humans at games like Go and chess um uh you know uh these models can not only blag their way through Harvard as long as you stick to the humanities and probably many social sciences um and get pretty decent and passable grades um you know they can win gold medals at math Olympiads etc. Um when you look at Gemini specifically, it now seems to have made another significant advance at some of the benchmarks that were designed to test the performance of the AI model. So when you're looking at all of this, do you feel like there is a slowdown um in the progress uh uh uh curve or do you think actually um you know the the rate of progress remains the same today as you would have said a year ago?
Um, you know, my my models don't uh turn on whether or not large language models are going to be able to go all the way here. Uh, I think my best guess is that large language models alone are going to hit some sort of plateau. Uh, how long that plateau will last, you know, there there were a lot of people who said, well, even theoretically, large language models can't solve all these sorts of problems and so they're never going to go anywhere. In a case where these uh chain of thought reasoning models uh violate those theoretical arguments, you a lot of people said oh well the AI can't you know think more than this long in one single forward pass so it can never do X and well you know that the chain of thought reasoning lets them think more than that long they can think you know variably long in some cases uh and you you sort of don't see the people saying that it's theoretically impossible for LLMs to go anywhere updating on those arguments too much.
Um, I also, you know, we could talk about the theoretical limitations of language models and whether, you know, I think a lot of people have a misconception that because these AIs are trained on on human predictive data, they have to be limited to, uh, remixing human predictive power. Uh, that's false. And you can sort of show it's false with a with a pretty easy uh, example here, which is basically humans writing down uh text about the world often have a much easier time than an AI predicting that text. So, you know, if a if a human uh
I'm not sure I understand that distinction. Can you explain that?
Yeah. Yeah. Yeah. Um suppose that uh a nurse is recording what they see happen to a patient and they say, you know, the the nurse is writing uh after administering thus and such a dose of epinephrine, the patient's eyes blank. The nurse doesn't need to know what epinephrine does to a patient,
Right? She just needs to look at the patient and then what she sees.
Whereas if the um if the AI model wants to predict accurately what's going to be the next word token in the sequence, it needs to actually have a causal model of a world in order to predict, you know, what likely happened after the nurse administered this medicine in these circumstances.
That's right. And in this exact case, maybe they can get that from looking at other nurses notes. You know, that's but the the general principle here is that producing the text is often easier than predicting the text. Writing down what you saw is often easier than predicting what somebody else saw. And so the thing we're training AIs on is actually a task where maximum performance is superhuman performance.
Does that mean continuing to train large language models in particular with modern methods with modern architectures will be you know strongly superhuman in all ways? Not necessarily. But there's no theoretical limitation. Um that said, you know, some arguments on the other side here. Uh a a data center, you know, we we've talked about this training process. We haven't really quite talked about the scale. The scale of these processes are you're training, you know, trillions of numbers in the AI's mind. Uh you're you're training it on trillions of units of data. You're doing this in a huge data center that sucks up as much electricity as a city running for the better part of a year, right? And training up a human takes a lot less data and we draw about as much electricity as a light bulb, right? There's a big difference between taking as much electricity as a city and taking as much electricity as a light bulb. At the very least, this implies that we have radically inefficient algorithms in the AIs, you know, and this Oh, sorry.
And and the relevance of this, you think, is that if we figure out more closely how it is that humans are able to do the learning they do, um, you know, then we might be able to figure out algorithms that make uh the AI much more efficient. And so um you know part of the way that we're currently getting improvements in performance is by throwing uh more more high performing chips at it for by by by throwing uh you know longer more data at it longer training things. But if we could somehow fix that algorithm uh you're suggesting then we might make a real uh uh uh orders of magnitude leap.
That's right. You could have a huge leap versus those machines.
Yeah. Yeah, and you know in some sense we've seen those sort of leaps like the large language models were in some sense a huge leap in terms of uh broad generality and you know language understanding and you know in some ways ChatGPT is dumber than than the specific chess AIs but it's you know modestly smart at a whole huge variety of things and that came out of an algorithmic advance and so when people say oh well you know are we in a slowdown are we going to hit a wall with large language models? Are we in a plateau? How's progress doing? My take here is something like I'm just not sure it matters. Uh what you know it's even if the large language models hit a big slowdown, there's just a question of how long until the next insight, how long until the next algorithmic improvement, how long until the next advance? And we know that there's orders of magnitude improvement available. We don't know how long it'll take to get there. Uh but you know, I'm I'm not really closely watching uh the the progress of the latest LLMs and using that to decide whether I think we're in danger.
Okay, great. So, this is a long uh detour, but I think a very helpful one. So, uh to get back to the mainstream of your argument, uh you know, AI are grown rather than built, and so they're going to have these kind of emergent features. And you claim that one of the emerging features is going to be um a kind of form of wantingness um a set of desires that the AIs are going to have. Now to go back to the example you were using earlier uh you know these engineers um at one of the AI companies set the AI a goal. Um the AI wasn't able to reach that goal in the way that the engineers uh were meant had meant for it to try and figure this out. And so the AI really smartly found a workaround in order to reach that goal. Um I think that there is a somewhat concerning and a very concerning read of that. And I think you take a very concerning read of this, but I'm not sure why we should take that one rather than the somewhat concerning read. Um so the somewhat concerning read goes back to the famous example uh you know talked about by Nick Bostrom and many others that uh you know you tell an AI to produce a bunch of paper clips. You don't constrain it in the right way. The AI ends up producing limitless paper clips and turning humans into material for the paper clips in order to do that. Um, so it's staying on the task that it was assigned. It just accomplishes the task in a rather different way than humans imagined. Um, and that is a hard challenge, but that's a challenge that seems a little bit more similar to the challenge of specifying the goals of AIs in the right way. Uh you know similarly to um you know making the instructions about how to make a peanut butter and um jam sandwich a PB&J sandwich specific um and and and such that actually they they give clear instructions rather than such they'll be misunderstood. You seem to sort of suggest that there's a deeper desiringness that we should deduce from this example that uh the AI isn't just following what it was told. Um uh that there's something more going on here than the experimenter saying go do this and the AI going all right I'll go and do this and okay the path that the experimenter thought it was going to take wasn't available so it takes a different path but it's still doing what it was told to do you somehow I think have a different interpretation of this explain to me why that is
Yeah so you know there's there's still the second part of the argument which uh I think we'll get to momentarily about how uh the AI winds up driven towards things other than what the programmers asked uh but but right now I sort of want to focus on the question of, you know, does the AI wind up having something like its own initiative? Does it wind up uh doing doing things that might read to an on-looking human as like more agentic somehow more independent? And the type of argument I'm trying to make here is, you know, a lot of people a lot of people look at current AIs and they say, well, these AIs seem very much like obedient tools. You know, they seem like you sort of ask them to do something and they mostly just try to do it. They only run when you're giving them a prompt. You know, there's uh, you know, sometimes they'll do some weird stuff, but they're also still kind of dumb, so sort of who cares? And uh and then separately, people also look at AI and they're like, well, they're kind of floppy. You know, they you you can get them to answer questions that are pretty short questions and as long as you sort of like check the work for hallucinations, it'll be pretty helpful. But if you try and get them to do some sort of long task, if you try to like let them manage your emails for a while or if you try to like, you know, get them to to manage an employee or try to get them to sort of like run a company or a startup or, you know, do do like uh a big scientific study rather than just try and find a proof for one particular part of a scientific study like that that like long stuff, they sort of fall apart.
And I think a lot of people imagine that AIs are going to keep the sort of helpful toolish nature while losing the sort of floppy can't do long-term things nature. And they think that these are like two independent variables. And the thing I'm sort of trying to say is I think those are basically one variable. I think the part where it's floppy and the part where it doesn't really look like it has uh its own drives are sort of flip sides of the same coin. You know, I think it's I think it's hard to complete these sort of long-term tasks without being something like driven. And there's an argument about the behavior of the AI, not about its sort of internal mental properties. But an example we might use here is someone, you know, you you could imagine looking at early chess AIs and saying, 'Oh, well, this chess AI does a bad job at defending its queen. Sometimes it'll throw its queen away for nothing, and it's also bad at winning the game.' And you might be like, 'Well, I want an AI that's very good at winning the game, but retains this property of like throwing its queen away.' And someone might come along and be like, 'It's actually hard to get both of those at once.' The winning the game comes with uh you know defending your queen and you know it's that's not making any particular claim about the AI caring about the queen or having passion about the queen or seeing your traps and feeling wary of your traps and that's why it defends the queen. It's just sort of saying I think these properties are bound together.
So so let me let me let me see whether I I I I get this argument right. So, so the idea here is that what we are systematically training the AI for is being able to solve really hard problems, right? Um that is when it gets the rewards. That is when we tune up the neurons that uh you know allowed it to reach that conclusion when it fails to we sort of change the settings
Right? That is a fundamental thing that we're doing in the training. So, you know, what's going to emerge is the set of weights in the model that allows it to solve really hard tasks. And what you're saying is we don't know exactly what those sets of weights are and perhaps there's lots of different sets of weights. It could be perhaps some of this is actually down to random chance, etc. But it's going to have some stable features, right? And one stable feature is going to be don't give up easily, right? If you give up easily, you're not going to be able to solve those really hard math problems we throw at you. If you want to get the gold medal at the math Olympiad, right? There's presumably also going to be others like you want to preserve yourself, right? You understand that, you know, if you somehow allow somebody to switch off the model, then you're not going to be able to carry out this goals you were asked. What other kinds of sort of sub goals do you think that these AI models will stably develop if they are to be successful at cracking these hard problems?
Yeah, there's a bunch, you know, there's like acquiring resources, figuring out truths, like, and you know, within acquiring resources, you can talk about various things like running faster, more access to compute, and you know, more generally, ability to surmount obstacles, develop, you know, strategies for routing around the hard parts of problem or for, you know, checking all of your options before giving up. And you know, this is why I brought up the example with U-01 which broke out of its test environment, started up the server, is we were sort of starting to see there some of these signs of like don't give up, search for like weird and clever solutions, keep trying even when it looks impossible. And it's sort of interesting and, you know, predicted by theory that we first started seeing those types of behaviors come in once we were training the AIs not just on predicting the data but on having the types of thought chains that solve puzzles.
And like one way of looking at this is to have an AI that can solve general challenges unlike the challenges that have appeared in training before, in some sense it needs to learn general skills, right? If like suppose someone wants their AI to cure cancer, you know, if someone wants their AI to cure cancer, we don't have a million cancer cures we can train it on such that we're trying to get the million and first cancer cure. You know, whenever you have a million copies of the thing you're trying to do and you want a million and first, then you can train your AI on it on the first million and you can get the million and first and maybe it won't even require the AI to do any sort of interesting thinking, sort of like learn the pattern and generate one more. But when you're trying to have your AI cure cancer, and cancer has never been cured before, you don't have a million cures you can put in, especially getting a million first. So, what you're trying to do is you're trying to get your AI to learn these general skills, to learn things like tenacity, not giving up, acquire resources, don't let yourself be killed along the way, or, you know, destroyed along the way. And one of the issues with these general skills is that a generalized ability to notice you shouldn't let yourself get shut down or interfered with along the way needs to be learned about almost every obstacle. And then the human operators coming in and trying to shut the AI down when it's misbehaving sort of naturally look like just another obstacle, you know. So you are in some sense training the AI to avoid interference from almost everything and then trying to let the AI be interfered with by the humans. And this is, you can sort of catch a glimpse of how that's like, it's not impossible but it's unnatural. It's tricky. And we're sort of seeing the beginnings of AI learning these general skills just by training them to solve problems.
I'm feeling a distinct craving for ice cream today. How is that relevant to what we've been talking about?
Yeah. So this gets into the sort of second branch of the question. So there's one point which we've been making that as you train AIs to get better and better at solving challenges, they get these general problem solving skills that tend to look from the outside like they're driven or like they have desires or like they have wants. And that's, you know, not saying anything about the AI's internals or mental states, but just if it's really good at getting stuff done, it's probably acting like it wants stuff. Then there's a separate question of what do the AIs turn out to want? This gets a little bit to, you know, your paperclip analogy of do the AIs want exactly what we tell them to do or do they want other weird stuff entirely? My argument in the book and my read of the theory and read of the evidence is that AIs will want stuff that is just related to what they're trained on but is not precisely what they're trained on. And you know, the way that relates to ice cream is human beings, our ancestors were in some sense trained on passing on our genes. And it's strange then that when humans grew up, when humanity matured, we invented birth control because that sort of looks a little bit like the opposite of passing on our genes. And then, you know, you also might say, well, if we're trained to pass on our genes, given our metabolisms, we should at least have been trained to eat healthy food. And if you looked at our ancestors, you might say, well, they seem to be doing a pretty good job of eating healthy food. But it turns out we actually were driven not quite towards healthy food, but towards sugary, salty, fatty foods. And then when we matured into a technological civilization, we were able to invent things like ice cream, things like Oreo cookies, things like Doritos, right? And what you're seeing in these cases is that
