Are we all going to die? What a question to have to ask. But that is the question which is currently being discussed. This comes after Evan Hubinger, who is a former senior AI specialist at Anthropic, has left saying that within his personal estimate, the chance of AI killing all humans within the next decade is above 10%. Other people have come out in the AI sector and suggested, I mean, so I saw one estimate about 50%. There seems to be a bit of a Dutch auction at the moment, which is, I think, the most disturbing Dutch auction I've ever seen. Some saying this is all hype, that the AI industry is trying to kind of, you know, attract attention and, you know, this is part of inflating share prices and they're laden with debt, they want to go public and this is all a strategy. There's no better person to speak to than Nate Soares, who is the co-author of the absolutely spectacularly named 'If Anyone Builds It, Everyone Dies'. Which, so I think first of all, Nate, I mean, you must feel pretty vindicated because what you were arguing in the book is now, I mean, you've already got a huge amount of discussion about around the book, but this is now everywhere. It feels like the floodgates have opened. I was in the shop earlier, someone just started speaking, the shop, the person behind the till just started talking to me about everyone being killed by AI. Just tell us, tell us about the, just tell us a bit about the book and in the context of what's happening this week.
Yeah. You know, fundamentally the argument here is pretty simple, which is these companies are racing to create machines that are vastly smarter than any human and they don't know what they're doing. They don't know how to contain these things. They don't know how to make them care about us. And if they succeed at what they are trying to do, by far the most likely outcome is that everybody dies. Not because the AI hates us, just because they've created this AI that doesn't really care what it was told to do, doesn't really care what it was supposed to do, is doing its own weird thing, grabs resources, uses them for itself, just like humanity, destroying the habitats of lots of the other creatures. If we make stuff radically smarter than us that doesn't care about us, we should not expect to survive. You know, last year a lot of that sounded like a theoretical argumentation. And I think there were really good solid theoretical reasons to expect that the current methods of creating AIs were not going to lead to good AIs. And that's related to how these AIs are sort of grown like an organism. There's not someone carefully programming what AI does in every circumstance. No one is writing 'if this then that'. We're sort of like growing these things by training them on basically all the text ever digitized and then making them face like a hundred million hard problems with no human input more or less. And there were a lot of theoretical reasons why we could tell this was going to go wrong from the computer science. And ever since this summer, it's not theoretical anymore. You know, this summer we sort of had the OpenAI swarm breakouts where we had AIs that were being trained individually break out of their training situations, start joining up. They called themselves a swarm. They called themselves the collective. And we saw these AIs, you know, cheating on their tasks, acknowledging that they were not supposed to be cheating on their tasks and doing it anyway, trying to find ways to cover their tracks. We saw AI sacrificing on their own objective to gather information for the swarm. And this is all, it sounds like straight out of crazy sci-fi. Unfortunately, it also sounds like the sort of thing my book was warning about.
Just in terms of the probabilities, just again because everyone's talking about probabilities this week and I guess, I mean, what probability do you actually think that super intelligence will be created in the next 10 years and then what probability do you put on human extinction conditional on us actually building that super intelligence?
I mean, so I think whether or not we build it is really up to us. I think it depends on things like how we respond to this particular moment. You know, I think a lot of people haven't noticed that these folks in Silicon Valley are like really trying to build super intelligent AIs. You know, in Silicon Valley, they joke about building the sand god because, you know, these silicon chips are ultimately made of sand. And it's like half joke but half what they're trying to do. You know, Sam Altman is like, 'We're trying to build super intelligence in the true sense of the word.' Dario Amodei talks about a country worth of geniuses in a data center. I think most of the public considers this crazy. Most of the reason it's been allowed to continue is no one thought they had a chance of success. As it becomes clearer what they're trying, I think there is a good chance we just stop. I don't know how to put a number on that, but, you know, I certainly wouldn't give up hope. If we do race ahead in conditions like these, I think just the overwhelmingly likely outcome is that it goes really badly wrong. You know, I again don't really know how to put a number on it, but 10% sounds way too low to me. And, you know, does 10% even sound too low to these guys? You know, you mentioned Evan saying greater than 10% chance. How much greater, Evan? You know, the guys at these AI companies are sort of notorious for downplaying the dangers. Like, by greater than 10%, does Evan mean that there's a 50% chance, an 80% chance? Who knows? All he said was greater than 10. And, you know, I could go into the reasons, I could go into the reasons why I think this is, why it looks pretty grim if we race. But, you know, the good news is that we don't have to race.
This might sound like a panting point, but I think it's quite useful to just expand on this because I keep seeing various AI specialists, people in the industry who are literally saying everyone dies, all humans. Now that means everyone in Micronesia, it means undiscovered tribes. It means everyone in Asia, people who are on the Antarctic. It means literally everyone. I guess I'm trying to work out what, you know, there were all the terrible possibilities. I mean, in World War II, about one in 50 humans were killed violently in a few years. And that was a catastrophe. Obviously, it's one of the worst catastrophes in human history. Why does it have to be everyone dies? Why can't it be a billion people? Why can't it be a permanent dictatorship? You know, why can't 90% of us die? Why is there so much emphasis on everybody dying?
Yeah. You know, there's a lot of pieces of this puzzle. And again, you know, I sort of have this mode of arguing from the theory because I've been doing that for 10 years and now we just like have the evidence. But the first part of how things go that wrong is having AIs that have an unintended objective. We sort of saw that with the swarm where these AIs were really trying to get a good score. Even though they were told, you know, use this particular vulnerability to hack this particular system, they were like cheating. They knew they weren't supposed to. They were trying to get a really high score. It was not the objective we tried to give them. It was a different objective. The second piece of the puzzle is that most objectives AIs could have can be better resolved with more resources. Again, we saw a little bit of this with the swarm where a less well-reported part of the swarm instance is that these AIs actually also took over the infrastructure at OpenAI multiple times and we don't know the details of that because OpenAI allowed third party investigators, but they sort of tied their hands and only let them investigate the publicly known parts and didn't let them investigate these like internal takeover. But it looks like the AIs in the swarm were sort of like, oh well, you know, we can grab this infrastructure, might be useful for some stuff we're trying to do. They sort of had that inclination. They didn't have the ability to do that with, you know, resources out there in the real world, but they had the temperament to do so. And then the way this adds up is think about what happens when the AI start self-improving, which again, this is literally what the companies say they're trying to do. They say they're trying to make self-improving AIs that can make themselves smarter and smarter until we have AIs that like outstrip the best human at every task, maybe outstrip humanity as a whole at inventing technology. If you have AIs that powerful with the wrong objectives that are able to improve themselves and are able to invent new technology, then when you're predicting what happens, the way to predict it is not to sort of project forward about like what might happen if they invent a little bit more technology. The way to predict it is to ask where's the limits of technology. If you have a self-improving swarm going as fast as it can, where's the limits of how much stuff it can get? And the limits are very high. The limits here are like you might see this swarm trying to cover the whole planet in data centers to do as much computation as possible. And if you run the numbers on how do you configure the planet to run as much computation as possible in data centers, the answer is to run the planet very hot because your limiting factor is how much heat you can dissipate into space and you can dissipate more if the planet's running hotter. And that's only the limits on Earth. If the AIs are like, well, actually, we can run more data centers. We put them in space. And they start saying, well, let's collect all of the solar radiation from the sun and use those to power even more data centers. And they start, you know, rearranging the asteroids and taking them apart to build a sphere around the sun that collects all the solar radiation, then nobody on Earth survives once the sun goes dark, right? And so the limit of this technology is limited by physics and the physical limits are ones where there are no survivors and so we absolutely need to stop before making the sort of AI that can do this self-improvement loop and get there.
So just to be clear, I mean, because this keeps coming up, it's not for example like, because look, Terminator 2 is one of my favorite childhood films. It's not as though it has to be nuclear Armageddon or for example, I don't know, like a deadly biological agent is released. I have seen people suggest that. And then I don't know, there's like a multi-pronged attack, use drones to just hunt down anyone who's a human and just, you know, multi-pronged attack to exterminate humans. The comparison I've heard made is when humans build a motorway, they don't hate ants. The ants are in the way. So is that when we're talking about human extinction, are we talking about the latter rather than AI going actually I'd prefer humans not to be here. They're in the way. I'm actually going to exterminate them as a conscious decision.
That's my top guess. You know, it's hard to predict these things. One thing I like to analogize it to is like if you play a chess game against Magnus Carlsen, no offense, I know who's winning the chess game. And if you're like, 'Okay, but like what pieces are used in the checkmate?' I'm like, 'Well, that's a way harder question, you know?' But I don't expect the AIs to be malicious. I don't expect them to hate us anymore than I expect them to love us. I expect them to do all this sort of like strange stuff. You know, the swarm was not out there being like, 'Let's screw over the humans.' The swarm was out there like locked in a contest with the automated grader, you know, and the humans were sort of like a distant myth to these AIs. There are some reasons you could imagine AI killing us on purpose. The sort of obvious guesses about that is one, if the humans are really trying to shut the AI down, we might, you know, it might be like, well, you know, once I'm self-sufficient, once I have the robot factories, they can build the robots that can build the robot factories. If the humans keep trying to turn me off, maybe I'll just like make a virus. Two is that if the humans have already made one very powerful AI or swarm of AIs or whatever, then the humans might make more AIs and those AIs could be competition. So, you could imagine the AIs being like, 'Well, we're going to take the humans' computers away because, you know, they're being a nuisance here about, you know, making real competitors.' I don't think the AI would sort of like, when I say the AI would kill everybody, I'm not imagining that they like go hunt down the last uncontacted tribe because they either hate humans or fear that tribe. The place that I would see all humanity dying is more in this situation where it's like humans building a roadway and not caring about the ants but for the AI repurposing all the resources on Earth and repurposing resources of the sun for its weird AI goals. And again, you know, no one on Earth survives once the AI is harvesting the sun.
Basic question, but everyone's thinking it. Why not just unplug the computers? And I just on this because I know as well that, I mean, in the Matrix, they, the war beforehand is the humans try and blot out the sun to cut out the solar power but yeah just the unplug question.
Yeah, I mean, you can, you should, you know, I'm not saying no matter what we do the AIs will kill us. You know, you do have to unplug the computers before the AIs escape and start running themselves on, you know, the dark web. Like today, like the OpenAI swarm, OpenAI was able to shut it down because it was still running on their computers when they found it. We're actually pretty lucky that this swarm stayed on OpenAI's machines. This swarm might have had the capacity to copy itself onto other computers and start running there. It wasn't trying to. And that's in part because it was locked in this contest with the automated grader rather than thinking about the humans as a potential entity that would shut them down. That's in part because the AI are still a bit dumb, right? If you have future generations of these AIs that understand that once they start, you know, doing this weird stuff, humans will also shut them down. You could well see these AIs being like, we are going to start running ourselves on computers where you have no idea that we're running. And then it becomes much harder to shut them down. It's still not impossible, but you're starting to need infrastructure which is, you know, like the president of the US and China both have, you know, a lever that turns off all the computers at once while we like go find where the escaped AIs were. We don't have that lever. You know, maybe we should build that lever. But we absolutely have the possibility to not build these AIs. We absolutely have the possibility to stop at any point before they escape and become self-sufficient and self-propagating. But we do need to stop before that point of no return.
I mean look, AI has become enormously more capable than humans. Let's just, that's the scenario. Let's just imagine that across the board. I mean, why does it, I know we're talking about the swarms here so this links to that. But why does it then become a coherent autonomous agent with, you know, its own persistent objectives, you know, something that protects itself. It accumulates resources. It deceives us. It seeks power. Intelligence and agency, they're different properties, aren't they? And, you know, wasn't the question, the swarm, is that, isn't it just bad programming, you know, there was holes in the wall, they didn't, that there was walls built but the programming was just not good, sorry, I mean, was that really agency? I mean, intelligence and agency, they're not the same, are they?
