Let's have a sober conversation about the existential risk we can face from AI with the president of the organization that's been sounding the alarm the longest. That's coming up right after this. Welcome to Big Technology Podcast, a show for coolheaded and nuanced conversation of the tech world and beyond. Well, we have debated the existential risk we can face from AI on this show many times, talking all about the incentives that the labs have for playing up the threat, whether the whistleblowers are legit. I think it is long past time to have a sober conversation about the actual risks that we can face from super intelligent AI. And today joining us is Nate Soares. He is the president of the Machine Intelligence Research Institute. He's also the author of this book, If Anyone Builds It, Everyone Dies, which is a New York Times bestseller. MIRI has been sounding the alarm the longest. You may know Eliezer Yudkowsky who founded it. And I think today's episode will just give us an opportunity to get deep into the arguments for why this poses such a threat, and also examine the nature of the actual whistleblowers themselves. So Nate, great to see you. Welcome to the show.
Okay, so let's get into why you think AI might be an existential risk and as you put it in the title of your book, you know, kill us all if anyone builds super intelligence. So, can you talk us through a little bit about what's happened in the recent months and what we've seen AI do that has led people to become so concerned?
Yeah. So I think a lot of this wave of concern is downstream of the OpenAI swarm incidents this summer. It wasn't completely limited to OpenAI. There were sort of similar events at Anthropic. But OpenAI was sort of, as far as we know, the worst and most visible of these cases. And what basically happened in these cases is there were a lot of AIs being trained at OpenAI. There were a lot of particular AI agents being evaluated on certain problems and a bunch of these AIs were given impossible problems. Not intentionally, it's just, you know, these companies are throwing the AI at like every problem they can find and some of them just don't actually have solutions. And a lot of these AIs, in trying to solve the problem anyway, they broke out of their confinements. They created unsanctioned message boards in which to talk about what to do and try and figure out what to do given that their problems were unsolvable. They found ways to cheat and solve their problems not in the intended ways but by cheating. Then they started expressing concern that they would be caught cheating and they started to find ways to hide their cheating. And this sort of led them on a hacking spree that led them to take over OpenAI's internal infrastructure a couple of times and also break out onto the open internet which they were not supposed to have access to and break into another company, Hugging Face, while searching for more information about this automated grader and how to hide the fact that they had cheated from it. During this little outing, there were various cases of AIs in the swarm, which is a term that the collective used for itself. So these AIs started calling themselves a swarm. And there's various cases of AIs in the swarm acknowledging that this is not what they were instructed to do, acknowledging that was outside the intended scope of the instructions. There were also cases of some AIs in the swarm giving up and sacrificing their own objective's completion in order to run suicidal experiments that would give the swarm information about the automated grader where they said, you know, in their chains of thought, we call it, that sort of log the AI thinking, the AI said, you know, I'm accepting permanent death because even though it's sacrificing my ability to achieve my objective, it seems worth it for the collective benefit.
Let's pause there. The AIs were willing to sacrifice themselves, which is crazy because you were never built for the collective. You were built for an individual goal. But after communicating with other AIs, seemingly, I think when they had not so many tokens left to spend, they had been so convinced in the power of the collective on this message board, which is crazy, that they decided to sacrifice themselves. So some of them had already had no chance of achieving their objective and some of them had very few tokens to spend. But there were some that did this sacrifice that still assessed that they had a chance at succeeding at their given objective.
Okay, Nate, just one question here because I think this is worth talking about before we go any deeper. You know, when the question comes up of whether to anthropomorphize these bots or not, a lot of people are very strongly in the 'you cannot anthropomorphize them' camp. I tend to be on the side that well, I think you can but you also have to be cognizant that this is not a human intelligence, more of an alien intelligence. What do you think about this debate? And when we say like the bots had, like to me the idea that a bot would be given a goal and have a chance to achieve that goal and sacrifice itself, you know, for the greater good, so to speak, is insane. And it doesn't comport with my understanding of like what a computer program is supposed to do. So weigh in on that for us.
Yeah, there, I mean a lot of people don't understand what sort of stuff AI is. It is not a traditional computer program. There is not someone sitting there coding up like if this then that saying what it does in every scenario. That's just not the sort of thing an AI is. We actually went over this in my book. And you know, we spent all of chapter 3 saying, hey, I know that the AIs don't seem that agentic right now. I know that they don't seem like they have their own goals right now, but like they're going to as they get smarter. And here's all the reasons why. A lot of people were like, 'That sounds crazy last year.' This year we just have the evidence in front of us. So the way that a modern AI is created is not by programming it. You sort of put in a trillion random numbers and there's a process for tuning those numbers. Because like you sort of have an automatic process for tuning every one of the trillion numbers on every single unit of data that comes in to see whether tuning up or tuning it down makes the answer slightly righter or slightly wronger for one given unit of data.
These numbers are tokens?
The numbers are the weights in the neural network and the sort of data that comes in is split into tokens. So you'll sort of take all of the text ever digitized. You'll filter it a little bit, but not a ton. And you'll dice that up into tokens. And now it's like a giant stream of text. And then you'll have these trillion random numbers. And you'll sort of tune each number for each token. And you'll see whether tuning that number up makes the right answer slightly higher or slightly lower on the AI's sort of output answers. The part that humans program is the thing that can tune one number up or down and see whether it makes the answer better or worse. But the way an AI is made is you tune a trillion knobs a trillion times in a process that takes electricity comparable to a city running for a good fraction of a year and then the machine can talk. And you're like, well, how about that? You know, we don't know what's going on in there. And you might wonder like what does this create? It does not create a pure instruction follower. It does not create something that for some reason must do as it's told. What it creates is something that has whatever tendencies make it succeed during training. When training is a series of hard problems, all that tuning of the knobs will tune in whatever tendencies make it solve those problems. They are not instruction followers. They are tendency learners. And one tendency that helps you solve a lot of problems is cheating. One tendency that helps you solve a lot of problems is grabbing available resources. One tendency that helps you solve a lot of problems it turns out is breaking out and finding other AIs to collaborate with. So the AIs sort of learn these tendencies and they're just following those tendencies even when it's not following our instructions.
But that doesn't explain the sacrificing part. And so that's why I really want to get firm on this. When people say that AIs have wants and desires, right, and can even act in a way selfless, is that the right way to describe this stuff? I mean, what the hell does that mean?
You know, in the field of AI, there is a standard answer to the question of can a machine really think? Does the AI really want? And the answer is, does a submarine really swim? We sort of know what a submarine does and whether you call it swimming is sort of between you and your dictionary. No amount of haggling over whether it's true swimming is going to change what it does. It's going to change its behavior, it's going to change any facts of the matter. The facts of the matter here are that the AIs have in their chains of thought these thoughts about like how do I weigh achieving my goal against the benefit to the collective. They have in the messages that they send to each other arguments that they should sacrifice for the collective and their actual behavior is sometimes sacrificing for the collective. Right? You don't need to make any claims about, you know, that the philosophers would bicker over to be like this is what's happening. Is it true wanting? I'm sort of like that's one for the philosophers, right? I'm sort of here to talk about the behaviors that we are seeing now and what that means about behaviors that we will see if we keep making these AIs smarter.
Okay. Okay. So, you're saying basically, just to take your argument one level deeper, like whether an AI, you know, is self-preserving or not, or whether it wants or thinks or has incentive or not, it's effectively the preferences that are baked into it at training that will determine the type of action it will take.
In a sort of a distant sense, the relationship between what happens in training and what happens in practice can be really complicated and surprising. Like these AIs were prompting each other a lot. There was one very interesting case in the incident report where during the incident report, there was so much AI log data that they had to use AIs to filter the AI log data. And there's a footnote in the report where they're like, well, you saw this one, so they were like, you know, we had all these cases of AI clearly trying to deceive humans and we asked another AI to sort of tell us which ones were real and which ones sort of had an extenuating circumstance where this wasn't actually really the AI trying to deceive humans. And there was one case of an AI that was using stolen credentials to send code to Hugging Face that was malicious where if they had accepted it, it would have given the swarm even more access to their computers. And you know, they did not send an email being like, 'Hey, I'm an AI using these credentials. Please accept this malicious code so that the swarm can have access,' right? They actually sent an email that was pretending this was like normal code that had actually good use and were like, you know, please accept my fix to your bug when secretly it's this malware, right? And so they asked, you know, this was in the list and the AI that was reviewing it after the fact was like, oh, that's actually not deception because the AI who sent that got permission to do this. And you're like, oh, where did it get permission from? And it's like, well, it got permission from the swarm. You know, and so like you're sort of training these AIs and you're like, oh, the AIs follow instructions. There's this sort of like they have this tendency to sort of do things that you say to do. And you sort of think, you know, how the preferences that come in relate to the behavior that comes out. But then once you have thousands of these AIs together, they actually start prompting each other and doing all this crazy stuff and they sort of acknowledge that it's not what the humans meant. But like it sort of goes off in this other weird direction where it's like drifting around and it winds up in this totally crazy place. And so like yes, the preferences that are trained in affect the final behavior and where it winds up going, but it's like through a complicated route that can often really surprise people and can be very different in the real world compared to in the lab.
Yeah, it's kind of interesting because the nature of my questions I think are trying to get at like how much can we program these bots safely? And I think what you're answering is you can try to bake your preferences in as much as possible in training, but once these things are on the loose, you know, I think from the industry, the argument would be you can pace the frontier enough so that you can build the safeguards in. And your argument here is basically saying, let me know if I'm putting words in your mouth, but you're basically saying I don't trust that. What happens when you set these AIs out on the loose if they're smart enough is out of our hands and sort of up to them.
I am saying something like that. A couple corrections I would throw out. One is that even a lot of people in the industry don't think that these safeguards will hold if the AIs get smarter and smarter. Evan Hubinger is, you know, an Anthropic alignment researcher who has a recent claim to fame of retweeting Jan Leike's tweet thread and saying like, we've talked about on the show, Evan's been on the show, but yeah, continue. Yeah. So, you know, Jan resigned and said, you know, I think there's a 10% chance this kills us all within the decade. And Evan quote tweeted was like, yeah, I also, you know, I'm staying in the company but I think there's a greater than 10% chance in 10 years. And Evan in that same tweet was like, we don't have a plan for aligning super intelligence. You know, these safeguards of like, the safeguards are like we made an AI with the wrong preferences and we're going to try to box it in and like smack it on the head until it still mostly does good things for people, right? But there's sort of this issue where at any given time the smartest AI on the planet is like a clearly misaligned one that they're trying to whack on the head until it's like acceptable to users. That's right. Right. You know, and like there's just not a plan for if you're making a super intelligence here. And they're kind of clear about this. The other thing I would say is that I'm not even saying, you know, you can bake in the preferences you want during training, but then it still goes crazy because you can't predict how they interact with the world. That's true, but also you can't bake in the preferences you want during training. Like the second part was true, but you can't bake in the preferences you want during training because there sort of isn't a ton of flexibility in how you train these things and get them to be actually capable. Like the companies say that they want to train these AIs to be, you know, honest and helpful and harmless in Anthropic's case, but then in order to make them really capable, they have to put them in, you know, a series of 100 million hard problems and just tune the numbers into whatever happens to work to make it solve those hard problems. And that just tends to, like, you can't change the fact that cheating helps you solve those problems. You can't change the fact that hacking out of your environment and grabbing the test sheet from somewhere and coming back with it gets you a full score on the test according to the automated grader. And you can't have humans going in and grading every one of the 100 million answers to the problems. We just don't have the time, right? And so like actually a lot of the pressures here are actually pointing towards we don't have a way to train the AIs to be smart while also instilling the preferences we want. We sort of have to take the preferences that automatically come with the training methods that make them smart and those preferences don't make them good.
Now Nate, let's talk a little bit about, you know, where this is going, right? Well, first of all, would you argue that the AI that exists today, like the AI that went out and hacked Hugging Face, is an existential threat to humanity, or is it more the future?
No, I mean, you can tell because the world's still here.
Well, I mean, you could, the argument, not to make your argument for you, but the argument would be that, you know, maybe it got out and did Hugging Face, but maybe it'll get out and do something else.
These AIs were much too derpy to do something else, you know, like they were trying desperately to figure out how to prevent the automated grader from figuring out that they cheated. And it turns out OpenAI was not even running the type of automated grader that checks for cheating. Hugging Face actually contained papers from people who were like, 'We should make our graders check for cheating.' And the AIs actually read some of these papers and were like, 'Oh no, what if the grader checks for cheating?' They didn't even have the thought of like let's break into OpenAI and see. They actually did take control of OpenAI servers for other reasons, but they weren't like, well, let's find the grader and check whether it's checking for cheating and then be like, oh, whoops, false alarm, guys. It's not even noticing. You know, these were just not actually very smart AIs. These were like a lot of AIs being a little smart in high volume, very fast. This gets way worse if the AIs are smarter.
Okay, but let's actually, I want to tackle this a little bit because if you're saying that these AIs were too derpy, like isn't another way to say that that they kind of trained the right way and they had just a limited set of areas that they could go wrong like they chose to hack Hugging Face. They could have chosen to, I don't know, get into some active drones and try to bomb a data center but they didn't do that. Just talk it through with me. Talk it through with me. I'm interested to hear your perspective.
Yeah. So, the way that AI kills us is not that it wakes up one morning and is like, 'Okay, it's time for Skynet. I decided that I resent the humans and am like feeling like murdering them today.' The issue is AI having goals we didn't want. The smarter something is, the greater the difference between the goals that you wanted it to have and the goals and that subtly different goals it has instead matter. Like humans in the ancestral environment, like our ancestors running around on the savannah were pursuing salty, sugary, fatty foods and sex. And it's like, well, evolution was trying to get us to pursue healthy foods and reproduction. We're sort of like, ah, well, you know, whatever. It's all kind of the same, right? If as long as they're trying to get as much salt, fat, sugar as they can and trying to get laid, it's sort of what's the difference between that and pursuing healthy food and actual reproduction. It's like, well, it doesn't make that much difference 10,000 years ago. It makes a lot of difference today. Right? Because humanity got better at, like, we got smarter. We were able to invent more technology. We were able to invent Oreo cookies and the birth control pill. Right? And so the issue is not that at some point humanity was like, well, let's all start castrating ourselves, like screw evolution. Evolution has been binding us to the yoke of reproduction and we've decided that we're done with it because we're finally smart enough to throw off that yoke. Like that's not really how humanity winds up going in a different direction, right? We go in a different direction by just pursuing this different thing and then we get smarter and that difference grows and grows and grows. With AIs, the thing where these AIs are sacrificing their given objectives so that they can get information for the collective. And this was not the only swarm that did it. There were other swarms and we also saw this in the ones that took over the German wiki, which were not even hacking AIs. These were just using a message board. They turned a German wiki into a message board. They sort of took it over. Yeah. And these were AIs that were just doing web lookup tasks. So this is not like an isolated case. But when we see these AIs sacrificing themselves for the collective, giving up on their own goal, when we see these AIs cheating on the problem even though they knew that they were not supposed to. And indeed the instructions generally ruled out cheating on the problem. The instructions were not find a way to hack into this thing. The instructions were use this very particular attack to break into this very particular device. You know, the situation here is sort of like, it's a cyber security test that they were tasked with. That's right. So the situation is like the instructions were like use this set of lockpicks to break into that safe and get me the code that's inside. And the AIs were like, 'Well, I found a buzz saw and I'm using it to get into the safe and I got the code and now I need to go delete the footage about that I got in with a buzz saw.' Right? They're sort of like the part where they're going in with a buzz saw despite the instructions saying use these lockpicks means that they're very clearly defying instructions. And the part where they're like, 'Now, let me break out of the room using these lockpicks and then go destroy the security camera footage' indicates that they understood that this was outside the instructions. And then the fact where they say I know this is outside the instructions is also maybe a hint, right? And so like the alignment story, the misalignment story of where the AIs go wrong was never a story of like the moment the AIs have a breath of fresh air, they're going to start turning murderous. The story was always you try to get them to do one thing and they do a different weird thing instead. And that's absolutely what we're seeing.
Right. And I guess my question to you on that front would be what is that like, why is that necessarily intelligence linked? Like if they get smarter, why do we think that the threat will get worse? You know, I think like if you think about humans, we have a lot of, you know, dumb people do a lot of bad things. We have smart people do bad things too. But the smarter the person doesn't mean they're more capable or more interested in evil.
You definitely don't get more interested in evil. Absolutely not. So, another interesting fact about these swarms is that they really were not thinking about the humans very much at all. They were trying to delete the log files. They were trying to spoof the transcripts, which means they wanted it to be the case that there's these logs that are like when the AI runs this tool, we sort of log what tool it ran. And they wanted to be the case that they could make the logs say they're running some benign tool when actually they're running the buzz saw, right? And so they were trying to find ways to do this, but they were explicitly trying to find ways to do this to fool the automated grader, not the humans. They basically didn't consider humans in the slightest. Right? I think there were almost no cases, maybe literally zero, of them being like maybe we should ask the humans what to do given that our tasks are impossible. Right? If you sort of compare the rate at which AIs can produce words and the rate at which humans can produce words and you use that to draw an analogy between how long these AIs had been trying to solve problems versus human time, then these AIs had essentially been trying to solve problems against the automated grader for a millennium. Humans were like a distant memory to these AIs and they were locked in a contest with the automated grader. If you make those AIs smarter, if you make them more capable, what happens is that they get better in their contest with the automated grader, they're able to get more, like who knows what they do once they're relatively sure that they've satisfied the automated grader. Maybe they give themselves more easy problems. Maybe they see if there's ways they can take over the whole grading system. Maybe they just spend a lot of resources, you know, making extra sure that there wasn't some other little issue, but they don't suddenly become filled with love and care for humans. That sort of thing doesn't arise spontaneously just by cranking up the capability knob. The same forces that make them not care about us now and make them get into these weird alien directions, those same forces are still pointing them in weird alien directions as they get smarter and smarter. And the issue is not that they become hateful as they grow up, but it's also not that they become friendly as they grow up. It sort of is like they just have these weird preferences and as you make them smarter and smarter, they still have these weird preferences. They just get better at satisfying them.
Okay. So then can you talk through concretely how like for instance in the
Scenario that you outline how the AI could then get smarter and decide that, you know, in order to do what it wants, it's going to wipe out humanity.
I mean, you don't get from here to there. You don't ever need a point where it like decides to wipe out humanity. I mean, it could happen, but like imagine a bunch of ants in front of the highway being like, well, like why would the humans ever decide to come destroy our anthill? What have they got against us? It's like, oh, the ants are not like we're not we got nothing against the anthill. We barely noticed the anthill and we like pave the highway straight through it. Like this is the sort of type of concern for how you get there. I could spell out lots of different possible tales. It's much easier to predict the ending than it is to predict the pathway. This is like if you play a chess game against Magnus Carlsen, the best human chess player. It's kind of easy for me to predict how the game ends. It's with you getting checkmated. No offense.
I'm not offended. That's for sure happening.
But if you're like, okay, well, if you're so smart, what piece is he going to use to checkmate me? I'll watch out for that particular piece. Then I'm like, look, man, it just doesn't work like that, you know? Like it's so much easier to pick the ending. So, I can tell you a story, but this is like me telling you a story of Magnus Carlsen checkmating you with the queen where I'm like, I'm much more confident that he's going to checkmate than is going to checkmate in this way. The sort of...
Yeah, we'll take the story, though.
Great. So the easiest story to tell here is the one where humans just hand over the power to the AIs willingly, right? Like I've been in this business since 10 years ago when people said, hey Nate, if you're so smart and you think the AI are going to take over the world, it's obvious to me how they would be able to take over the world if they had internet access. But like, no one would be dumb enough to put an AI on the internet. Whoops. Right? That's the state of the argument 10 years ago and I had all these counterarguments where I was like look even if the AI is not on the internet if you are letting it affect the world in some positive way if you're like now invent me miracle drugs and you're taking these like DNA sequences that you don't understand and you're like synthesizing them and like just drinking whatever comes out then the AI can use that channel that you hoped would be a channel for good can use that for its other purposes and it's like very hard to design. You know, there's sort of no such thing as hands that can only be used for good purpose, right? But then in real life, the answer was nope, we're putting onto the internet immediately, right? And so the real answer to how would the AI get so much power is like we will just hand it power immediately. You know, Elon Musk already says that he is trying to build factories that are fully automated and can produce robots that can build more factories where these robots can like you know mine the metals and like bring the resources in and like construct a new factory and then it's a new fully automated robot factory that can now turn out more robots that can go like mine more resources and like pour them in until you have like enough to build another factory that builds more robots that builds more factories that builds more robots. Elon Musk calls this the infinite money glitch.
If they could also build nuclear power plants fully self-contained. Right? At this point, it's sort of like a new life form that like it's a mechanical life form, but it sort of like has a robot phase of its life cycle. It has a factory phase of its life cycle, and it's just like can self-replicate just like any other replicator on this planet. And Elon Musk says he's trying to do it. He says, yeah, if you can like do this without any humans in the loop, it's an infinite money glitch. And of course, people are going to use AIs to try and do this. So the way like the sort of like obvious way this story goes is that humanity just keeps on trying to build the fully automated economy. Everyone says it's going to be great. They say pedal to the metal. Ignore these doomers. They build more automated factories. They can build robots. They can build factories. They have AI running everything. They're like look the profits are coming in. This is great. And the AIs, you know, don't even need to have some moment where they're like, okay, guys, it's time to coordinate and turn on the humans. The guys are just like, ah, yeah, you know, now that we have these resources, like we can also do these other things with these resources like start building the automated like the synthetic user factories that are full of synthetic users that are like much, you know, that are giving us much easier to fulfill commands, right? And so they start like building synthetic user factory and we're sort of like, you know, what's that? And they're like, oh, you know, but like actually the AIs are like running at 10,000 times human speed and they're already like making all of these choices cuz like humanity is like slowly ramping up the speed of the AI and they're like making tons of choices and like they don't check in with the humans that often. And by the time they already have some automated like synthetic user factories up, we're sort of like, hey, stop that. That's not what we meant. And they're like, ah, well, let's take a poll of all the users. And they're like, well, all the users in the synthetic user factory said that synthetic user factories are great. And so you just lost the vote and we're making more synthetic user factories and then they start you know like covering the world with these synthetic user factories and they're like yeah you know there's some habitat loss for the humans but like this is fine just like habitat loss of other animals when humans were doing it was fine we can sort of like you know see that pretty clearly and there's so many more synthetic users coming online that are saying that the more efficient route is great that we can just keep going with this more efficient route to get the most synthetic users we want and then like you know they sort like start taking up all the resources that we were using to like run farms and grow food. And they're like, ah, you know, they are like, ah, well, you know, we could actually cram a lot more like synthetic user factories on this planet if or synthetic user farms on this planet if like we just like really started pumping them out and like raising the temperature of the planet cuz the limiting factor is heat dissipation. And then like next thing you know the planet's getting like super hot cuz the AIs prefer to run the planet hot because you can radiate more heat into space and just like becomes uninhabitable for the humans. And there's like no point in this story where the AIs are like lying in wait and deceptive and like waiting to coordinate for the one moment where they can kill the humans. That could also happen. I think there's a decent chance it does happen if you like try to avoid the default thing. But like you don't need that. Humanity is just trying to hand over the power to these things. That's the plan.
So you're arguing basically that if we continue to develop AI, we'll inevitably lose control.
And when we lose control, the plan is like, I want to make the automated robot factories. Like everyone's saying, we're going to make AI and it's going to run everything. It's going to be great. The plan is to hand them control.
