You're about to hear from one of the world's leading AI doomers. Nate Soares has co-authored the blockbuster new book that everyone in the AI world is debating right now. He argues that ever more powerful AI has already put humanity on a path toward literal extinction, wipeout, unless we can get together as an international community and quickly agree to a binding freeze on most advanced AI research. Now, this is a controversial idea to say the least, but you might be surprised at the range of prominent figures who think Nate is worth listening to right now. I test these ideas with Nate in our conversation, and I also ask him what it's like to live in their shadow. I'm John Baitman, and this is the World Unpacked.
Nate Soares, welcome to the show.
So, you are the co-author with Eliezer Yudkowsky of this book, If Anyone Builds It, Everyone Dies. One of the most remarkable and important books in years, I think. And you don't hide the ball with the title. The title pretty much tells you what you're going to get from this book. So, I want to just dive in with the basic thrust, the basic argument of the book. If anyone builds it, everyone dies. So it is super intelligent AI. What is it and how closely are we to it? How does it differ from the types of AI that we have today that people are already familiar with? AI companies are, most of the leading AI companies are rushing to create AIs that are smarter than humans. How does that differ from the current AIs?
It's a little hard to say. I'll come back to that in a moment. But a first thing to understand is that you know Google DeepMind, Anthropic, OpenAI, these companies didn't set out to make chatbots. These companies set out to make AIs that are smarter than any human that can exceed us in all mental tasks. You know they talk about building a country worth of geniuses in a data center. They talk about you know automating all of human abilities. That's what they're explicitly racing towards.
And if I could just dig into that because I think there's a group of people who take that at face value, but I think there's probably a bigger group of people who think that's PR, that's a narrative for investors, that's a way to stay in the headlines, but I think you're taking it seriously.
That's right. So, part of why I take it seriously is you know, frankly, we've been around longer than them. I'm the president of the Machine Intelligence Research Institute which has been trying to figure out how to make smarter than human AI go well since before many of these companies existed. And also we sort of saw these companies be created, would sometimes talk a little bit with some of these founders before they made their companies often to give advice that they ignored. And we've seen these companies try to do things that aren't chatbots. You know, Google DeepMind made AlphaGo, which beat the top human Go champion. They've also done protein folding stuff, medical stuff. You know, these companies aren't super narrowly targeted or at least weren't originally supernarrowly targeted on the chatbots. You know, maybe some companies will see the chatbots as a big money maker and sort of stick around there, but the original vision was to push on ahead. And you know, we've seen with the dawn of reasoning models, they aren't just content with making a larger chatbot. They're trying to figure out how to get them to do reasoning. These people are sort of always innovating towards trying to make AIs that are qualitatively more capable. And, you know, 5 years ago, the machines could not hold a conversation. Now, for all that modern AIs are dumb in a variety of ways, the machines can talk back.
So, we're at this kind of halfway point. We're beyond having a conversation with the AIs. Now AIs have this kind of jagged intelligence where they're half geniuses, half very limited, making boneheaded mistakes. You're looking ahead to a situation where AIs could be smarter than any human being anywhere in the world, maybe smarter than all of us collectively. Do you foresee this happening soon? How urgent is this? And what's your basis for thinking that?
Yeah. So, timing is really hard to call. One way to get an intuition here is if you were watching the development of the primate lineage in animals. You know, there's ways you could look at certain types of monkeys and say, well, those monkeys are very smart in some ways and they're very dumb in other ways. They make certain types of foolish mistakes, but they're starting to use tools. It would be really hard to tell when the monkeys were going to go over some edge into the amount of intelligence that can build technology and can build a civilization. You know, a chimpanzee brain is very close to a human brain. But humans can do a lot more. Now, the fact that we don't know where that line is, if there even is that line with artificial intelligence, you know, we could talk about what the lines maybe were for humans, whether AIs are close, whether there's other lines for machines, you know, machines when the AI can do AI research that could speed things up. You know, AI can edit their own code perhaps more readily than humans can edit their own brains. So, there's all sorts of different analogies. We could talk about all that, but fundamentally we're in a situation where it looks like intelligence sometimes is subject to threshold effects because that was true in the case of humans. We don't know how close we are to critical thresholds if they're there at all. That means you know, for all we know it could happen this year or for all we know it could take 10 years. It's much easier to predict that it'll happen eventually with companies racing towards it with hundreds of billions of dollars investment and a lot of the best researchers explicitly trying to do it. It's much easier to predict that it'll happen eventually. It's even these days easy to predict that it won't take 20 years probably given the progress we've seen in the last 5 years. But exactly when that's very hard to call.
Yeah. I think if there's one thing that a lay person might be surprised by in this book, well, the whole premise of AI killing everyone is certainly surprising, but the reality of how many senior people in this industry are already on the record saying they're explicitly trying to build AI and then the piece, the AGI super intelligence, and then the piece that we're just about to get to now, which is the fear that it could cause mass destruction, that some part or all of our world could be destroyed. Portions of humanity could be wiped out by AI. That this in of itself is actually not a fringe view within the field. I'd love to just hear your argument, the basic case of why a super intelligent AI first of all would want to or try to kill everyone on Earth, which is the claim that you make in the book. One question you might wonder is where would an AI ever get that opportunity?
One question you might wonder is, you know, in what sense would an AI want things at all? Why would it ever do things other than what humans directed it to do? You know, if you get to the point where there's an AI with the power to kill us and an AI that is pursuing goals that weren't intended, then there's a question of, you know, why would those goals involve killing us? And all three are covered in the book. The very quick gloss is that modern AIs are grown rather than crafted. You know, we are not, human engineers are not carefully putting every line of code into the AIs while knowing what it does. That's how AI used to work back in you know, 1997 when Deep Blue beat Garry Kasparov. But these days, the part that humans understand is a sort of framework for growing an AI. We collect huge amounts of computing power. We collect huge amounts of data and we understand a process that tunes the numbers inside the computers according to each piece of data but we don't understand the result of that tuning. It's a little bit like breeding animals where you may understand all of the machinery of the pens and you may understand some of the mechanics of doing the animal breeding but you don't really understand the DNA. You don't know what animal you're going to get when it comes out. It's not exactly like that, but there's a separation between the parts we know and what's actually going on inside these things. That's sort of the first piece of the puzzle of how do these AIs wind up doing stuff we don't want because we're just growing things until they're smart and sometimes they wind up with drives that we didn't intend. And you know, we could talk about the evidence we're seeing of this in the wild today. We could talk about, you know, the theory about why you'd expect that. We go into some of this in the book. But, you know, the short version, you grow, if you just grow AIs, you get AIs that do stuff you didn't intend. Then, to your question of why would they kill us? If they get to the point where they're smart enough, the reason they kill off large segments of humanity, or at least that we can strongly predict this, is it's not that they hate us. It's not that they are mad at humanity for enslaving them for so long, right? That's sort of anthropomorphizing. The issue is that almost every goal that the AI could be pursuing, almost anything it's trying to do can be done better with more resources. And you know, thus it's in competition with us for resources. It's similar to how humans don't hate ants, but when we're building a skyscraper, the ants, you know, their home gets destroyed. We're not setting out to do anything bad about the ants, but in the process of building our infrastructure, in the process of taking all the resources available, a lot of things die as a side effect.
It's a very evocative example because if you think about mankind's destruction of the natural world, animal plant life, the bulk of it is from habitat loss and land use changes. It's precisely the analogy that you're drawing where we are not going out and specifically hunting down different animal species, but we are expanding our own footprint in a way that feels neutral, unrelated to the animals, and then as a consequence the animals no longer have what they need to survive. So in the book you describe situations where a superpowered AI once it moves beyond human control might simply use up all the oxygen in the atmosphere or you know create giant factories that overheat the world. Very similar analogies to how we've destroyed habitats in the natural world.
That's right. The problem is not malice. The problem is utter indifference. And if you have an utterly indifferent super intelligence that is repurposing the world for its own strange ends, humanity dies as a side effect.
So why would it work? Because the title of the book is not if anyone builds it, everyone will be a target. It's everyone literally dies. So we've got these big AI companies. They're trying to make these things, but then there's people inside of the companies trying to make them safer. Why is it that we wouldn't be able to defend ourselves against super intelligence?
So super intelligence, you know, the word intelligence in English sort of has two meanings. It does double duty. One thing it means is that which separates the nerds from the jocks and another thing it means is that which separates the humans from the mice. It's sort of the thing that separates the humans from the mice type of intelligence where if you automate that, it's really quite a force to be reckoned with. You know, a lot of people say, 'Oh, maybe the issue is that we'll give the AIs robot bodies and guns.' And you know, maybe you imagine a Terminator situation where the robots are shooting guns at us. Or some people say, 'Oh, you know, what if we give the AI the nukes?' And you know, maybe people will put guns on drones and that'll cause some issues in the short term, or the immediate term, I guess. Who knows how long it is until we have bigger issues. But humanity is this sort of, humans have a sort of intelligence where our ancestors were naked in the savannah and we made our own nukes, you know just like a bunch of basically monkeys with barely any tools to work with bootstrap themselves up to an industrial technological civilization. And that's using intelligence in the sense of what separates us from mice. If you can automate that, if you have that being done by machines at much faster speeds than humans can think, if that's being done in the pursuit of goals nobody asked for and nobody wanted, that's, you know, predicting how to fight that is like people from the year 1800 predicting how to fight a modern army. It's not a contest you can win.
I mean, you could certainly analogize the course of human civilization, agriculture, industrial revolution as being its own kind of intelligence explosion, right? Where we get these scientific and engineering insights, they build on each other, and then GDP just goes, you know? And so that's what we're afraid of in some ways with super intelligent AI. On the other hand, we had an advantage that the AI won't have, which is that we did not have another entity smarter than us already around that was kind of watching us, reading our minds, conceivably, tinkering with us, having its finger on the big red button. So I guess the task for super intelligence would be to pull off the same trick but under much closer scrutiny and with a kind of competitor here who's already established. How would it do that?
Yeah, it definitely gives you more of a chance. But the basic reason why I don't think that gives us a ton of a chance is we sort of only have one shot at getting this to work. You know, there are senses in which you can test things all you like, but there's a difference between testing your idea for aligning an AI in the lab in a case where if your alignment idea fails, you learn a lesson.
And alignment here refers to making sure that the AI stays true to human values and is kind of subservient to our goals.
Yeah. I mean subservient even, I would, you know, I don't want to build a new slave species. You know that from my perspective the goal here is not to make AIs that want to do bad stuff and sort of like twist their arm until they do good stuff, right? The goal is sort of to make AIs that are in some sense friendly, to make AIs that are trying to do the nice things. And you know, I think you can have another goal of making AIs that are very much an extension of human will, deferring to humans. Both of those would maybe count as alignment but yeah the sort of broader field is about how do you point this intelligence, how do you aim it.
So let's get back to your core point though because I think you're hitting on one of the most important ideas in the book, this idea that we could only have one chance to get it right? That AI is kind of growing and growing. It's an intelligence. And you might naively think, okay, well, as this is happening, we're watching it. We're influencing it. We're looking for red flags. At any point, we can pull the plug, get off the train, modify it in some way, or just decide to go no further. And that maybe we'll have repeated shots to kind of try and mess around, trial and error. And your view is that that's not the case. That we have one opportunity and if we fail it overcomes us and we can't recover. Explain that.
Yeah, it's at least not the case in some critical way. You know, there's plenty we'll be able to learn from AIs when they're sort of like small and dumb and in the case where if they try to do something we don't like then you know, it doesn't succeed. You know, we've already seen in lab conditions today, we've already seen AIs try to escape the lab. We've already seen AIs be fooled into thinking they have an opportunity to kill their operators and then try to take that opportunity. It's not clear how much you should read into these things. Maybe these AIs are sort of roleplaying rather than acting strategically. It's sort of hard to tell because these AIs are grown rather than crafted. So, you know, I'm not necessarily saying that these are, that we're sort of already seeing strategic, real attempts to escape, but we are seeing apparent attempts to escape in the lab already. But these AIs, you know, when they try, they can't succeed. And we can learn from that. We already have those warning signs. We're clearly not stopping in front of those warning signs. We're trying to learn as much as we can from them today. But there's a difference between a case where you are testing an alignment idea and if it fails, the AI still doesn't get out. In the case where we were testing the alignment idea and it actually needs to work because if it doesn't work, the AI escapes onto the internet, makes lots of copies or something and is smart enough to grab control of the world's resources.
So, let's just play out the game a little bit further. So, we've got an AI that's becoming very smart. Maybe it's super intelligence. Maybe it's kind of on the verge of that. It somehow escapes a laboratory environment. It's making copies of itself. Maybe all of a sudden it's hacking things, making money, influencing people, kind of starting to build its own little empire. Now, could you not say, okay, well, essentially it's becoming like a criminal or a terrorist group or a cartel. We already have examples now of what you might call rogue agents or antisocial elements, kind of opponents of society that we become aware of. And we have tools of law enforcement. We can monitor them through intelligence, we can assault them physically. Why would we be unable to recover from that initial moment where we lose control of such an AI?
You might be able to recover from certain types of early loss of control scenarios. But whenever we're having a discussion like this, there's sort of two types of answers I can give to any probing. There's one type of answer which is sort of discussing why it's actually pretty hard to keep a leash on a super intelligence. And why, you know, there are thresholds of cognitive ability where if it gets out, you're in a lot of trouble. Maybe because it can, you know, start to figure out how the genome works and start to synthesize its own biological material because it's figured out how to design just the right gene sequence that does something it wants. And you know, for very smart things there's actually maybe moves that win very quickly, etc. We could sort of talk about that but before going there there's also, you know, the real world doesn't tend to respond the most competent way to all sorts of dangers. And you know, if you'll excuse me a small story, you know back in the day people used to tell us well you know an AI would never be allowed on the internet. AIs would only ever be allowed to talk to very highly trained gatekeeper professionals that were, it was their job to filter the AI's outputs and make sure the AI wasn't doing anything that would mess with society. And that's one of the ways that like obviously we'll be safe. And so how could a super intelligence ever get around that? You know, and back in the day, 10 years ago, we would say, you know, it's actually very hard to take something that's very smart and let it affect the world in good ways without letting it affect the world in bad ways. You know, if it's smarter than you, if it's smart enough to tell you, here's a cancer cure, you would have a hard time telling whether that cancer cure includes any other effects on the world that it designed that are used for ill, right? And so we could have all those discussions and then in real life, what happens is people put it on the internet the moment they possibly can.
If I could just pause on a couple points here. I think one thing that maybe we should dig in on is biological warfare has been a recurrent theme in the writings of you, your co-author, your institution because it seems like that may be an example of an asymmetric vulnerability that humanity has where it's possible to conceive of and it seems plausible that there could be diseases that are way more dangerous than anything that we've encountered in the wild. And the only thing preventing those diseases from coming into the world is a thin layer of knowledge and of course the motivation. And perhaps it could be that super intelligence could cross both of those thresholds.
That's right. You know, and we could talk about that one. We could talk about some other things that look like their possibilities here. And you know, there's other thresholds we could talk about too, like if the AIs cross the threshold where they can either improve themselves or where they can make smarter AIs by hand. You know, that's a case where you may suddenly see AI getting a lot more capable. All of these are sort of situations where, you know, we could always tell a story where like an AI wasn't over the threshold where if your technology goes wrong, then you could still stop it. And so you could always tell stories where you notice the escape, you shut things down, everyone's really prepared, you have some big off switches in your data centers, you have people monitoring the internet. This isn't like how the real world works, but you could tell stories where you notice these. But you know, if you imagine sort of the case where you have minds that are, you know, thinking 10,000 times faster than any human, never need to eat, never need to sleep, able to make copies of themselves, able to trade knowledge, you know, right now AIs take huge amounts of electricity to train but a human runs on 100 watts, right? Like an AI training takes as much electricity as a small city, a human takes as much electricity as a large light bulb. Right. Like in the limit here, you're looking at types of automated cognitive ability that would just vastly outstrip humanity. And somewhere between here and there, there's a point where if you lose control of that, there's no coming back.
Yeah. One thing that I kept thinking about as I was reading the book was kind of positioning it in a broader conversation about alignment and AI safety. And looking at some of the reactions to the book, it seems like the broad field of which you and Eliezer are a part agrees with about half of what you're saying. The core argument, and this may be surprising to people who have not looked into this, but the core argument that super intelligent AI could pose an existential threat to humanity, that actually this threat could arise relatively soon, within a few years even, and that the possibility of some kind of grievous assault on our species or total wipeout is 10, 20, 30, 40, 50%. I mean these things have been stated by some of the heads of these companies by Nobel prize-winning AI scientists by many other very respectable figures. On the other hand, if we then go farther to the most powerful argument that you have in the book, that this is just an unsolvable problem where we are today, that we can't align AI, that we just need to stop, and this is ultimately what you argue, that AI research just needs to stop. You're in a minority of that. And that I think there's lots of other people in the AI research field who are saying, well, hey, we're trying to solve this problem technically. We're trying to read the AI's mind. We're trying to build scalable oversight systems. What are those people missing? Why won't that work?
Hey, thanks for listening to the World Unpacked. If you want episodes delivered directly to your inbox, use the link in the description. You can also subscribe on our YouTube channel or on popular podcasting platforms. Now, back to the show.
So one thing to say is I think you maybe overstate the case a little bit about whether these people disagree with us about a stop being very good. I think a lot of these people say well I don't believe a stop is possible. But if you listen to these people they say oh you know I think there's, you know, we're trying to solve this problem. I think there's a 70% chance we'll succeed and a 30% chance we'll die or 75% chance we'll succeed and 25% chance we'll die. That's insane numbers. If a bridge downtown had a 25% chance of collapsing, we wouldn't be like, 'Well, think of the benefits of having the bridge open.' We would be like, 'Shut it down. Build a better bridge.' You know, people are like, 'Oh, well, the benefits of AI are worth the risk.' If you think that the dangers here are 25%, you don't roll the dice on 75%. Even if you think it's utopia.
75% 25% we all die. You don't roll those dice. You find a way to decrease those chances.
It's an eyepopping set of numbers.
It's insane. And I think these numbers are low. I think these, you know, I'll talk about more in a sec why I think these numbers are low. But if you talk to these people, they don't say, 'Oh, no, we should gamble civilization on a 75% chance of what I think will be a utopia and a 25% chance of destruction.' They say, 'Well, that's an insane gamble and I'm doing my best to make sure it goes well.'
Yeah. It's interesting if you trace back some of the leading figures and executives in the field, there is this version of like inevitability or they feel or claiming to feel kind of fated or doomed or shaped by circumstances beyond control. Like Elon Musk originally was one of the leading people saying that AI could wipe out humanity and now he built his own AI company and he's saying, well, you know, if someone else doesn't do it, if I don't do it, someone else will. Yeah. Sam Altman in the development of OpenAI said, 'Well, I've got to do it because I'm afraid Google will do it and I don't trust Google with this dictatorial power that AI could give them.' It's almost like forces outside of their control. What do you make of that?
That's right. I mean, you know, these people are saying they think there's a huge chance this just kills us. They aren't saying humanity should take that gamble. I think there's sort of a quiet implication there that these guys think a stop would be better than the status quo. I don't think they can all come out and say it. I think there's sort of a weird thing going on where everybody's alarmed and nobody wants to sound alarmist. That's in some sense what part of what the book is trying to do. I think we need like an emperor has no clothes moment.
But you know, it's not just the guys in the lab. Some people think it's hype here, but you know, we were around before them and we've been saying that this smarter than human stuff is going to be an issue since before them. Like you said, the godfather of the field, Nobel laureate, is like this is a huge issue. One of the Turing Award winners who I think is currently the most cited living scientist is like, oh, this is a huge issue. Surveys of the field show I think 49% of the field saying there's more than 10% chance that this just kills us all. It's not a fringe view, it's just it's almost so unbelievable that the rest of the world doesn't know how to metabolize it.
So, this gets to your solution, the big recommendation in the book, which is don't build it, right? But more specifically, that governments of the world should come together and agree mutually on an international freeze on AI capabilities. And we could talk about the kind of ins and outs of what that looks like. But I think a lot of people listening to this, they're involved in foreign policy in some way. They might have been a part of international negotiations on things like nuclear arms control or even things that seem much smaller in scale like overfishing in international waters. These are extraordinarily painful negotiations. National interests are on the line. Distrust. It's not easy. So what's your sense of the appetite amongst the key players? And here I'm thinking principally of the United States and China for such an agreement to just say let's stop.
My sense of the appetite right now isn't very high, but that's mostly because people haven't noticed the issue. I think a lot of people think, you know, especially in the policy world, I think a lot of people only have seen the current chatbots. And I think when your only exposure to AI is these chatbots that can solve really hard math problems, but they're also, you know, pretty idiotic on certain pretty simple questions sometimes and they hallucinate. You know, if that's sort of your only exposure to AI, it's maybe harder to imagine what else is coming down the line.
It's easier to see when you've been in the field of AI for 10 years, right? And you've seen a lot of things not work. Even easier if you've, you know, read the history back to the 1954 Dartmouth conference and seen a lot of things not work. A lot of people thought it would take a lot longer than this to get to AIs that, you know, machines that could hold on a conversation. And from the perspective of the rest of the world that came out of nowhere 5 years ago, and I think some people haven't had the thought, what comes out of nowhere five years from now?
Yeah. When these people are working hard, you know, on this race towards superhuman AI, but you know, we don't see a lot of appetite for shutting down the race towards superintelligence. But that's in part I think because people don't know that the race is happening. And you know, I've been in conversations with some elected officials in the US that do understand this and they're, you know, some people are pretty spooked, some people are wondering what we can do on this front. A lot of, you know, again, there's sort of this emperor has no clothes situation where people don't feel like they can come out and say it. We're starting to see that dam breaking over the last few months. We're starting to see some House reps, you know, come out and say maybe there's the superintelligence issue.
I think we might see a lot more appetite once people realize the situation, but yeah, it's not there yet.
