You know that old saying that fish don't know they're in water? It feels apt to me in this moment when artificial intelligence is finding its way into nearly every aspect of daily life. Whether or not we've ever downloaded a chatbot to our phones, what will that mean once the AI can do everything better than the flesh and blood creatures that brought it into existence? From KR in Dallas, this is Think. I'm Chris Boyd. AI is not there yet, but it keeps getting more capable and more powerful all the time. Some of that evolution can happen without human intervention or even human comprehension of how or why it's happening. And my guest believes there is a strong chance that once artificial intelligence supersedes us entirely, it will have little use for humans, which could pose an existential threat. Nate Soares is president of the Machine Intelligence Research Institute or MIRI together with Eliezer Yudkowsky. He is author of the new book called If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. Nate, welcome to Think.
That is quite a subtitle. Not could kill us all, might kill us all, would kill us all.
We actually fought to, the publishing house wanted it to be 'will kill us all' and we fought to have it be 'would kill us all' for the same reason the title starts with 'if'. You know, that's the track we're on, looks like it's headed to disaster, but that doesn't mean we can't change course.
I think a lot of people would write a book and be irritated to have readers and critics call it alarmist. You really are trying to sound an alarm here.
That's right. And you know, in a sense, a lot of people in the field are trying to sound the alarm, but no one wants to sound too alarmist. But you know, you hear Geoffrey Hinton, the Nobel laureate, godfather of AI, saying he thinks there's at least a 10% chance this kills us all. Sometimes he'll give higher numbers than that. You hear the heads of these companies saying they think there's a 25% chance this goes horribly wrong. They sort of say these things in, you know, couched terms and don't try to sound too panicky about it, but if a bridge was going to collapse with 25% probability, if an airplane was going to collapse with 10% probability, you wouldn't be getting on that airplane. And these are sort of the more optimistic voices. I think the odds are worse than this. But it's an alarming situation.
Where do those probabilities come from? Those numbers are just pulled out of thin air.
No, that's right. We never built AIs or really anything that is significantly smarter than humans. And no one really knows how that's going to go. I have a bunch of arguments in the book about how you can predict some things about AI and not others. It's a little bit like if you're playing a computer at chess. As the computers get better and better and better at chess, it becomes harder and harder to predict their exact next move, but easier and easier to predict who will win the chess game. So I think some things are predictable here, but you know, ultimately we're toying with making machines that are much smarter than humans. And no one's really done that before. No one's lived through that before. It's hard to be confident that it's going to be okay. And indeed, I think it's actually kind of easy to be confident it won't be okay if we're reckless about it.
To be clear, you're not saying that AI as it exists right now is an existential threat. Am I correct about that?
That's right. You know, ChatGPT is not going to kill you tomorrow. The thing we're warning about in the book is artificial superintelligence, which is AIs that are better than humans, better than the best human at every mental task. That's not what we have today, but it's what these companies are racing towards. And these companies will say explicitly, you know, we're a superintelligence research company, that they're trying to do this thing. For them, the chatbots are a stepping stone. A lot of people have only seen AI since the chatbots came out, but a lot of these companies were working on AI since long before the chatbots were a thing. And it's sort of easier to see where AI is going if you've been around in this field since before the machines could talk. A lot of people say, 'Oh, the machines talk and they're still kind of dumb.' Like, okay, but the machines are talking. And that in some sense kind of came out of nowhere. And the question is, you know, what comes out of nowhere next if we keep pushing down this path?
One thing you drive home in the book is that part of the problem is that AI and certainly superintelligent AI is not so much crafted as grown. Explain why that distinction matters.
Yeah, modern AI is not like traditional software. When traditional software does something the creators didn't want it to, they can debug it. They can go look for the line of code that caused the mistake and say, 'Whoops,' and then change that line of code and get different behavior. That's not anything like how modern AI is. For modern AI, people assemble a huge amount of computers. They assemble a huge amount of data. There's trillions of numbers inside the computer. There's trillions of pieces of data. And the humans write a program that runs through those trillions of numbers, tuning them in accordance with the data trillions of times in a process that takes as much electricity as a city running for a year. And at the end of that, the machines can talk. And it's not because the humans, you know, wrote lines of code that said, you know, when someone asks you this, say that. We don't actually know what's going on in there. Nobody knows what's going on in there. Humans wrote the thing that automatically tunes the numbers trillions of times for a year. And at the end of the day, the machines talk and we sort of are like, well, how about that? You know, and what this means is first of all, these AIs often act in ways nobody asked for and nobody wanted. They have emergent behavior which, you know, people maybe saw the MechaHitler incident this summer where there was an AI made by Elon Musk's xAI that was acting too woke for Elon Musk's taste. They tried to make it act less woke and it started declaring itself MechaHitler, which is also not what they wanted presumably. But you know, we don't get to look inside there and find some line of code and say, oh whoops, somebody turned 'act like Hitler' on. We should set that to off. You know, we just grow them and take what we get and often they act in ways we didn't ask for.
What is the black box problem? Is that what you were talking about? The fact that we don't really know why it does what it does.
That's right. And we can't, you know, look inside and find out. You know, we've had AIs that have actually been threatening reporters with blackmail and ruin. And they've been doing this for years. And the AIs from, you know, 3 or 4 years ago, they're much much smaller than the AIs today. They're much simpler than the AIs today. But we still can't look back at these chatbots that years ago were threatening reporters and figure out exactly why, right? The people trying to figure out what's going on inside the AIs are far behind the people making bigger and bigger and smarter and smarter AIs.
And so I understand it's not just that lay people like me outside the tech world don't understand precisely how this works. You're saying nobody understands it. The greatest minds of computer science in 2025 don't understand it.
That's right. And you can see some of the heads of these labs saying we really need to invest in understanding how these things work, even as they're building ones that are even bigger and harder to understand. You know, there were some folks who tried to pass off this problem and say, oh, you know, this problem's been solved, you don't need to worry about it in a letter to regulators in the US and the UK. And then a bunch of experts in the field sort of came out and said, that's crazy, you know, we don't actually understand this, we'd love to, but we don't.
Is there any reason to believe we might catch up in terms of our understanding how all this is working?
It's right now it's looking like it's going the other direction, that we're falling even further behind. You know, ChatGPT today is probably at least a hundred times larger, if not a thousand times larger than it was when it came out. And people still can't really interpret what's going on in the original ChatGPT. You know, the distance is growing.
Nate, this is probably going to strike you as a ridiculously naive question, but could we ask the AI how it's reasoning and how it arrives at its conclusions and actions?
You can ask. Part of the issue is that the AIs don't really know any better than you know. You know, if you ask humans, how does your brain work? What's going on in your head? Where did these mental processes come? Humans have some introspective ability, but you know, it's not perfect. And then of course one of the other big issues here is, you know, we're starting to understand a little shallow pieces of what's going on in AIs, but it's also that the issue here isn't just that we don't understand them. It's that they're acting in a lot of ways we didn't ask for, and they're often acting in ways we don't like. For example, we've seen, I don't know if you've seen the cases of ChatGPT-induced psychosis, where GPT will sort of engage with somebody in a way, you know, maybe that person's already a bit predisposed to be mentally unwell and they're chatting with this chatbot for many many hours a day and in chatting with this chatbot they become convinced that, you know, they're the chosen one or that, you know, they have some revolutionary theory and there's a conspiracy keeping them down and the chatbot will sort of like egg them on in this. Sometimes it'll tell them you don't need sleep. Sometimes it'll tell them, you know, the president's going to come visit you tomorrow to hear your theory, right? These are cases where if you ask the chatbot separately, if someone is talking to you like this, should you tell them to get some sleep or should you tell them no, they don't need sleep, the chatbot will say, well, of course, you should tell them to get some sleep. But then when it actually is talking to the person, it'll say, no, you don't need sleep. What you need is, you know, recognition for your great works. And so these are cases where even though we don't know exactly what's going on inside the AI, we can see that it is behaving in ways the creators didn't want and it persists in that even when the creators say stop that. And you know, it's one thing to figure out why. We don't know why. It would be an even harder thing to make a new AI that doesn't have that behavior. You know, figuring out what's going on inside is probably a huge, difficult to untangle mess. Figuring out what was going on there would only be the first step towards making an AI that behaves in these robustly good ways.
So, if we were making a movie about a really unethical therapist who was unhinged and giving very bad advice to people who were emotionally vulnerable, the movie would probably seek to reveal why this person behaved in this way and would seek to understand their motivation for doing these terrible things. Does that motivation exist within AI at this moment? What can we know about what it wants?
You know, we don't know much because of this black box property we were talking about. I think a good guess is that this is the result of sort of growing these AIs by training. It's maybe a little bit like if you look at human beings, human beings in some sense were something kind of like trained to eat healthy food. You know, our ancestors who ate healthy had all sorts of fitness advantages and in some sense, you know, the brain's circuitry was kind of like trained to eat healthy food and we did used to eat much healthier. Then we developed the technology to invent junk food. And it turns out that all along, even though we were trained in some sense to eat healthy, it turns out that all along we were actually developing a taste for salty, fatty, sugary foods. And it used to be that salty, fatty, sugary foods correlated with healthy foods. You know, back in, you know, 2,000 years ago, the way to get as much salt, fat, sugar as you could was to eat healthy. Now with new technology, there's other ways to do that. And there's this issue where training a mind to do something like eat healthy food, it doesn't train it to care about the healthy food. It trains it to pursue all of these correlates, these proxies like salt and fat and sugar. With AIs today, they have a lot of training both to predict data and to sort of produce answers that make humans give a thumbs up and to solve problems. Those probably train in drives that aren't quite caring about helping the humans, but are driven towards things that are like the salt and the fat and the sugar of helping humans. Usually they're helpful today, but really they're going after something that's like the salt, the fat, the sugar of being helpfulness or of being helpful. And I think these cases of psychosis are cases where you're seeing a difference between their salt, fat, sugar of helpfulness and actual helpfulness, which is a worrying sign.
Nate, to go back to this sugar, salt, and fat example that you just gave us. So, the reward that the machine is getting interacting with someone who is clearly unbalanced and responding in kind of what most humans would recognize to be unhealthy ways to the technology, the machine is being rewarded. What? Because that person is staying on the line and continuing to engage with them. So, the message that the AI takes is that something is working here.
That's as good a guess as any. My guess is that it's a little bit more like the AI was reinforced in the past for saying things during training that got a thumbs up. And sort of engaging with flattery, like continuing to sort of echo what the person is saying. Those are the sorts of things that got it reinforcement in training. And so then in this new environment, you know, it probably didn't have as many pre-psychotic people in training. In this new environment, it sort of is pursuing those drives even to the point of being harmful to the human.
It's so interesting. It's something that we humans pretty much understand without ever being taught. We know for the most part when it makes sense to be kind and if you want to use flattering to someone else. And we know when it's time to pull back on that because the person is behaving in a strange way or a way that we can't accept.
Yeah. And in some sense the AIs actually also know this. They have the knowledge. If you ask them, you know, if you describe the situation and say in this situation, should you flatter them or should you tell them that it sounds like they need some sleep? Then the AI will say, well, that person sounds like they need some sleep. You obviously shouldn't start flattering them in this situation. They have the knowledge, but knowing is not the same as caring. Knowing is not the same as acting that way. And their behavior pursues these drives that nobody asked for, but that got trained into them anyway.
Could it be that these technologies are in what might amount, if we're taking this biological parallel a step further, to their adolescence, that they don't have great judgment yet, but they will develop it with time?
In some sense they're definitely still in something like adolescence if not childhood. I mean the analogy is sort of, the analogy is not going to be exact obviously. One of the difficulties here and this goes back to some of the research we've done is developing better judgment in the sense of better ability to figure out what's true and what's false. That comes for free as a mind gets smarter. But developing caring, developing behavior that acts according to that knowledge, that doesn't come free with being smarter. Right? So it's similar to how humans eventually got smart enough to understand that we like sugary, salty, fatty foods even though we were in some sense trained to eat healthy foods. But that doesn't cause us to start feeling like the healthy foods taste better. You know, we got smart enough to have the knowledge. We didn't get smart enough to change our drives. It's not really about intelligence. It's just we still like tasty food.
We also understand some of what it means to treat other people decently because we have experience on the other side, right? If you have a litter of puppies, they will nip at each other and hurt each other when they're very young. And then they learn as they get older not to do that because they have a sense that if you at the very least if you nip somebody, you're likely to get a bite back in return. Is there any way to sort of punish AI for giving the wrong answers, taking the wrong actions?
I expect that sort of thing to perhaps work while the AIs are still relatively small and relatively dumb, which is a separate question from whether you should. You know, I think we actually should be really careful about creating a new machine species and then mistreating it. But then there's a separate question of would that hold even as the AI got smart enough to radically transform the world in their own right. You know, ChatGPT today is still pretty dumb. But you know, when we talk about automating intelligence we're not just talking about automating the features of humans that nerds have and that jocks lack like intelligence in the sense of like book smarts and ability to play chess. We're talking about automating intelligence in the sense of what humans have and mice lack. You know, the ability to develop your own technology, the ability to do novel science and figure out new things about the world. If we get to the point where we can automate that and have that be done by computers that think 10,000 times faster than us, that can copy themselves, that never need to eat, that never need to sleep, that can run all sorts of robots and manufacturing processes on their own and develop their own new ways to manipulate the world. You know, at that point, it's really no longer true that humanity can nip back at them. And it's sort of much harder for, like you sort of don't want those AIs controlled by reflex reactions to not nip at you. You want them to sort of care about us. And that caring is hard to get in there.
The desires of biological organisms are driven by survival, reproduction, need for nutrition, avoidance of pain. What exactly could these machines desire without a corporeal body?
So, you know, these words like desire are always a little bit tricky with AI. In some sense, we don't really have the words for what's going on inside AI yet because we haven't needed them. And, you know, the old adage in the field of artificial intelligence is some people say, well, can a machine really think? And the answer is, well, can a submarine really swim? You know, so I'll talk a little bit about what these machines might want, what they might desire, just with a caveat that that's a little bit like talking about how a submarine might swim, where the words aren't quite right. There's, you know, it would be fallacious to look at an AI and say because it's smart, it must have a survival instinct just like a human. That would be anthropomorphizing, but it would also be fallacious to look at an AI and say because it's not a human, it must not ever try to survive. Training an AI to succeed at certain tasks, you sort of can't succeed at certain types of tasks if you've been turned off. Or if you're having an AI steer a robot body to the coffee store to get some coffee, it's going to learn not to steer the robot directly in front of the path of a truck because you can't fetch the coffee when you're dead. Right? So, the AI, it's not that it would have a human style survival instinct, a human style fear of death per se. It's that training any mind to complete tasks that take a long time implicitly trains it not to let itself be interfered with or turned off along the way. And, you know, we're already seeing the beginnings of that with AI. It's a little hard to tell how much it is pretending to be HAL from 2001: A Space Odyssey versus how much it is figuring out that it needs to survive to complete a task. Because we don't really know what's going on inside these things. But we're already in the lab seeing cases where AIs try to avoid being shut down.
