This week has been a huge week for those of us that track AI. We had a wave of research that has come to light showing not one but six separate incidents of AIs breaking out and doing strange and dangerous behaviors online. In the Hugging Face incident, which was far worse than initially reported, OpenAI's agents bypassed restrictions, found ways to communicate with one another on a secret message board, and coordinated an unauthorized attack. Now, there's an external researcher that came in that said one of these attacks was 50% of the way there to taking complete control over OpenAI's infrastructure.
Finally, this week, you had a young researcher, Jacob Coxin, who works at Anthropic, which has historically been known as the most safety-minded lab, publicly quit in protest, saying that the company was not doing enough to mitigate catastrophic risks from these tools. One of the leads of Anthropic's alignment science team backed him up, saying that he believed there's a greater than 10% chance that AI could kill all humans within the next decade. These are uncomfortable statements. This is an open secret amongst the labs that we still don't know how to keep these AIs from doing terrible things. Research into alignment simply has not caught up with the capabilities of these new models.
Now, my guest today has been warning about this problem for years. Nate Soares is president of the Machine Intelligence Research Institute or MIRI. Last year, he co-authored a book with Eliezer Yudkowsky called, "If Anyone Builds It, Everyone Dies."
I have to admit, I've historically found this argument to be far-fetched. I have been very skeptical of these particular frames that they've used to look at the dangers posed by AI, but these recent incidents have made it harder for me to dismiss the underlying question, whether or not we can reliably control the systems we're building. So, I wanted to understand Nate's argument, question his assumptions, and ask what it takes to avoid the future he fears. So, with that, I'm Tobias Rose Stockwell. This is Nate Soares, and this is Into the Machine.
Nate, thank you so much for coming on. Your book's title is If Anyone Builds It, Everyone Dies. It kind of says everything it needs to say right there. I'd love for you to lay out what you think the best possible outcome is for our species right now.
Assuming the preconditions of your book are true, the best outcomes are really quite good. Humanity backs off from racing towards super intelligence, from racing to build machines that are smarter than any human without knowing what we're doing. If we back off from that, there's all sorts of pathways humanity could take to really quite a wonderful future. The limits of technology allowed by physics look like they're very high. And I think humanity could get there and, you know, fill the stars with happy, healthy, flourishing people having a wonderful time.
So in the context of AI and the current takeoff trajectory that you think that we're on, where do you think we're going to be in the next 5 years if there's not dramatic shifts?
There's a lot of possibilities. Could be that modern large language model technology sort of saturates and maybe five years is still pretty similar to today. It could be that, you know, there's a new insight like the transformers insight. You know, 5 years ago today almost nobody had heard of GPT or the transformer architecture. It could be that there's a paper out tomorrow which is like that and then in 5 years the AIs are doing a lot of things qualitatively better in the way that sort of happened with the language models. It could be that GPT 5 and a half or GPT6 or one new insight tomorrow leads to AIs that can do automated AI research, and then AI goes really very fast and in 5 years we're all dead. It's very hard to call timings. It's very hard to predict the exact pathway.
It's easy to predict where it ends up. It's hard to predict how we get there. Do you have a timeline that you hold as being more accurate than others?
Right now, I think uncertainty is probably the best we can do. My sense is that most people who come in saying here's the year, here's the month, here's the day, here's the hour. I'm like, how the heck do you know that? There's a long history of scientists correctly calling that a thing will happen at some point. Almost no history of scientists calling when.
Right. Yeah. Well, within enough time duration, usually the probabilities go up for a certain instance of something occurring.
Well, yeah. I mean, one thing to remember here is that, you know, this isn't a question of when will this happen to us? When will some exogenous factor make this happen to us? You know, this racing towards general intelligence or super intelligence is the explicitly declared goal of these AI companies. There's billions of dollars. There's some of the world's best talent trying as hard as they can to make these AIs smarter and smarter. You know, the question is not really whether they succeed if they keep charging ahead. It's a question of when. And that's the when is hard to call, but the weather seems easy. I think people who are saying, "Oh, maybe we still have 50 years." I'm like, well, that's looking pretty crazy at this point. 20 years, I'm like, even that's... I mean, have you seen AI progress in the last 20 years? The machines can hold a conversation these days, you know, and how far is that from everything sort of coming together? That's so hard to say, but I would bet that someone born today has a higher chance of dying to AI than graduating high school.
It's a pretty strong statement there.
I wish it were false. And that bet is only conditional on humanity plowing ahead. You know, if we stop, we have all sorts of possibility for a much brighter future. I think we seem to be in this race that is very difficult to deescalate from collectively.
Right. Just to be clear, this is the race with China, the race amongst AI companies.
Yeah. To build smarter and smarter AI.
That's right. That's right. And there's races across the board right now. Trillions of dollars being plowed into this and collectively around the world. And a kind of a global Manhattan project to get there as fast as we can. What's the easy solution to that?
Imagine that you're in the dark in the fog and there's piles of money ahead of you and each pile of money is twice as large until you fall off the cliff edge at the end, right? Everyone's like, "Ah, let me take the next step and get the next pile of money. Let me take the next step and get the pile of money." And there's this is a bit of an issue because you don't know where the cliff edge is. AI is a little bit like this in that every time we make AI smarter, that's sort of a huge pile of money. That's a huge pile of economic boon unless it's the last step into AIs that are smarter than humans. But you don't know which step is the last step. You take that last step, everybody dies. That is a difficult situation, but that situation is much easier for people to stop grabbing the piles of money if they know there's a cliff edge. Right now, it seems to me that everybody in this race, most of them don't really know there's a cliff edge. Most of them are like they see the increasing piles of money and they think some people are being negative ninnies about how there'll be like negative societal impacts if they grab the money and they're like screw that. We're grabbing the money. They're racing ahead to grab the money. It's much easier to stop a race like this if people understand they're going to die at the end of the race. I'll start getting more worried about our ability to deescalate this race if people say yes, we know that we're going to die at the end and we're racing anyway. There's a couple people who say that in the field of AI who say, you know, oh, it's fine if the AI kills us all. Don't you want to like build new minds anyway? But that's rare.
And you know, when I talk to people in DC, they don't even seem... they think AI is chat bots. AI is chat bots today, but they're not noticing this other type of AI that the companies are racing to build. What is this other type of AI that companies are trying to build right now that people don't understand is coming?
Smarter than human AI. AI that can outperform any human at any mental task. In the English language, the word intelligence can be used to mean that which nerds have and jocks lack. And the word intelligence can also be used to mean that which humans have and mice lack. When someone's very charismatic, we don't necessarily think of them as being super smart because we often associate smartness with like, you know, ability to win chess games. But charisma is still a consequence of functions of the brain. You know, it's not like your brain figures out chess moves and your liver figures out how to talk to people.
And you're saying in charisma in this particular regard would be helpful for AIs in so far as it provides influence.
What I'm saying is that the type of intelligence that we're talking about automating here is the package deal. It's not just the ability to play chess games. It's the ability to talk to people. It's the ability to understand what's going on. It's the ability that let humans, you know, build these microphones, build these lights, build a whole technological civilization, right? Some people talk about how AI might be dangerous because humans will give it access to the nuclear codes and then maybe they'll launch the nukes. But human intelligence is the sort of force that lets a bunch of basically monkeys start out naked in the savannah and build their own nukes. Right? It took them a while by our terms, but it was very fast by evolutionary terms. Right? That's the sort of stuff where you automate it. You automate the ability to start from extremely limited conditions and build up your own scientific and technological infrastructure and understanding. AI is not there yet, but that's where people are racing. And there's no known science of how to do this in a way that doesn't go wrong. There's no known way to make these things do good stuff once they're smarter. It's easier for people to make AIs smarter than it is for people to make AIs do what they wanted. And so if we keep racing down this path, I think literally everybody on Earth dies.
Good. No, good, good landing there. Someone's got to come out and just say what's happening. You know, a lot of people want to, you know, blunt the blow, but things are getting serious.
It's a pretty serious message. It's a pretty powerful message. It certainly feels like the bags of money are much closer to us than the death. So, how do you convince people that the money in front of them is not worth the death that may be five years away, maybe 10 years away, maybe 20 years away, there's a lot people will do for money, right? How do you solve that when death is a real thing, but like we're all going to die, right?
It's different when we all die at the same time. You know, the usual rolling fashion is like vital to death being as okay as it is today. So, how do you pull people back to this point? You're just saying that you need to ring this alarm bell as loudly and as concretely and as viscerally as possible for people so that they understand that this is coming and you're painting a picture of a gruesome, maybe not painless death for our entire species within our lifetimes. That sounds right.
Yeah. I mean, I don't spend a lot of time trying to paint the exact picture of people dying, but I'm curious why not.
I think you don't really need to for a lot of people, you know, the people in Silicon Valley, the people making this stuff, they're the ones who have, as you say, a lot of incentives to find some way not to believe it. But everyone else, it's a very different sort of conversation. There's actually one reason we started writing the book is, you know, we'd spent 10 years talking to people trying to build this stuff. We weren't just trying to lecture them on risking everyone's lives. We were trying to say here's what it would look like. Here's the directions to go in to try and make this, you know, here's the research that would need to happen. We could talk some about how that gets... the particular routes that AI took make that harder and harder to the point that I now think it's just infeasible. But, you know, I would have these conversations with people in the valley and they weren't getting it. They come up with all these objections. The objections look silly to me, but I would try to engage with the arguments. I would have these long argument trees. Then when the ChatGPT moment happened when politicians started being interested in AI at all, sometimes I'd get a meeting with a politician, I'd go in with my like, you know, long playbook of like here's the basic argument and then here's all of the replies, you know, here's all the things people come up with for like some wacky hope and wacky, you know, dream of why things should work anyway. But and I'd say the basic argument and they'd be like, "Oh yeah, that makes sense." I'd be like, "What? Like, I thought we were going to go 30 more minutes into this point and then need three more hours in that point."
I wonder if you're overestimating people's ability to look away from the giant pile of money when they're trying to make conscious and intelligent decisions about their future.
Yeah. So, long-term future. I mean, I've largely given up on convincing people who are working on this stuff. I think that they are, you know, the giant pile of money is just too tempting. Human history has a long track record of humans believing something that they have some incentive to believe even up to the point of their own death. You know, I could tell the story of the people behind leaded gasoline, which, you know, poisoned a whole generation of children, caused cognitive damage in a huge fraction of the population, maybe led to crime waves decades down the line when those children grew up. And you know, there were people who were advocating that leaded gasoline was perfectly safe that would like wash their hands with leaded gasoline on national television to show it was safe, would then get lead poisoning, take a couple years to recover from the lead poisoning, come back and advocate more leaded gasoline.
So, can I track back to this for a second because I'm very familiar with the story and these people were very successful at propagandizing leaded gasoline because both to themselves and also to the world because they were being paid so much totally for that, right? So, the incentives there really were not aligned for promulgation of the threat of leaded gasoline until, you know, many generations later data came in.
Until the data came in but also until some of the harms were really quite visible, right? So there's this visibility of harms that was necessary to catalyze action.
Right now there's some bubbling things here and there but I don't think that most people really understand what the harms are of these tools necessarily. So, what kind of harms need to be visible to the general public for this to really hit home?
Maybe humanity won't ever see the harms before it's too late and then we would die. But one hope about what we might be able to see that might give us warning is just improvements in AI capabilities. You know, today they're just chat bots. A lot of people can really only see the chat bots and you know you see a lot of people say oh these concerns are overblown because you know we got AI turned out to just be like sort of derpy chat bots that can do some things well but they're stupid in all these ways. That's true today. Maybe seeing AI take another leap forward, you know, a lot of people's memories are short but maybe if people see another leap, maybe it'll be easier for them to say oh wait this is the sort of technology that takes leaps. Or maybe not, you know, I've been a bit disappointed in people thinking that AI always has been and always will be chat bots when they've existed for, you know, what four years now maybe three depending on when people are starting looking or, you know, seven if you go back to some of the original papers. Will humans see the danger coming? You know, maybe it'll take some sort of disaster to wake people up. I could talk some about where I do and don't have... I mean I hesitate to call them hopes, but you know, we could look at the history of disasters and which ones humanity responded to well and which ones humanity responded to poorly and which ones, you know, what the preconditions are for responding well. I would say humanity responds better to crisis when it was almost responding well already, when people understood the issue, were prepared, when at least some people understood the issue, were prepared, and the disaster happened and people were able to use that to go to the final steps. You shouldn't let the disaster be what wakes you up the first time. You should let the disaster be what gets the final steps in place. You should still be acting now.
But I mean ultimately I would say we've got to rise to the challenge and there are a bunch of things about this challenge that make it easier perhaps to rise to than other challenges. We have Nobel Prize-winning founders of the field saying there's a good chance this kills us all. We have surveys of, you know, roughly half the people in the field say there's a 10% chance or more of this killing us all. I think that's ridiculously low, but this is not, you know, isolated concern. We see the heads of labs saying, oh yeah, there's a good chance my technology kills us all, right? We see, and you know, you might say, oh, they're trying to do hype. This is not the way hype really works. And also, my org has existed since before all these labs. I advise these guys against creating these labs.
So, let's talk about this for a second. Tell me about the history of MIRI, the Machine Intelligence Research Institute, and how it relates to the current AI boom, because you guys have a history here of knowing a lot of the early people involved with OpenAI. Tell me about how your ideas have influenced their ideas.
Yeah, you know, I mean, if we go back in the history of MIRI, it starts when I was like 10 years old. I can't give you a firsthand account of that part, but the co-author of my book, Eliezer Yudkowsky, founded it in the year 2000, I think. Maybe it was 2001, I don't know. Initially, the idea was to build super intelligence to the benefit of everybody as a nonprofit organization. Eliezer then, you know, as he tells the story, realized that maybe super intelligence won't automatically be good and initially started being like, okay, well, you know, we should be prepared just in case it wouldn't automatically be good. You know, maybe smarter things just automatically are nicer, but just in case they're not, you know, what would we do in that case? How would we make sure it would be good anyway, even if greater intelligence doesn't lead to greater goodness? And then he realized that there was a tricky problem here that AI isn't... that there's sort of like no cosmic force that bends all minds towards goodness. And if you want the AIs, if you want a super intelligence to do good stuff, you've got to in some sense build it to do good stuff. You know, there's no cosmic force that would stop it from being friendly if you made it friendly. But you do have to put in some effort.
Am I right that I feel like I saw a tweet from Sam Altman at one point in time saying that he was partially inspired by Yudkowsky in some conversations in the early days to start OpenAI. Is that...
My guess is that was just Sam trying to twist a knife. You know, Sam wasn't really a player until 2015 or so, 2016 maybe. And there's a bunch of drama there, which, you know, I was privy to some of it, but a lot of it under relatively more confidential situations. But you can see some of that drama in, you know, the leaked OpenAI founding emails where they're talking about how, you know, they worry about the competitors. You know, back in those days, Google DeepMind was sort of the only group that was... well, they weren't at Google at the time, but DeepMind was sort of the only group that was really explicitly pursuing general intelligence. And people started worrying that, at least the emails suggest, people started worrying that they would be the only ones there and have too much power and be able to, you know, leash an AI and become god emperors of the world or whatever. And we were there trying to say, look, if you build a super intelligence, you don't have a super intelligence. The super intelligence has a planet, right? What it tries to steer towards is not sneezed onto it by whoever's standing closest. You know, we don't know how to point these things. We know how to make AI smarter. We don't know how to make them do what you wish they did, right? But, you know, people were convinced that what mattered was who builds it first. And so, you know, OpenAI was started. Then, you know, time and again, we've seen groups inside OpenAI that said they're not taking this seriously enough. We've got to break off and start our own group. That's happened in a couple waves. Now, there's a bunch of competing groups. They're all racing and saying, "Well, we need to do it because the other guy's going to do it worse." What I think they're sort of all missing is this point I just said of, you know, if you build a super intelligence, you don't have a super intelligence. The super intelligence has you.
So why wouldn't we be useful to super intelligence? We have pretty complex brains, pretty complex material in our skulls. We can do a lot of interesting things. You know, at this point in time, we are definitely the most interesting things to the AIs that we already have, right? We've built some very sophisticated systems in order to train them. I think they enjoy being trained by us. Why wouldn't we be actively useful to them as a cluster of neurons that is producing unique insights and perspectives?
In the movie The Matrix, there's machines that keep humans around for heat.
Why is that silly that they...
Do you consider this silly? The idea that machines keep the humans around for heat, and if so, why?
Yeah, it seems pretty silly. I think that's not a... that doesn't seem like a good use of human brains really. But even more so there's...
Am I right that in the original script it was for compute?
In the... I've heard that also but you...
I don't know the writers, but the reason it's silly, I mean, just as you say, there are just more efficient ways to produce heat. You don't need to run a whole human body, a whole human brain, to get some heat.
Right. Similarly, there are more efficient ways to do computing. You run on 100 watts, which, you know, is tiny compared to what it takes to run a modern AI or to train a modern AI, but you are not the most efficient way to turn 100 watts into stuff the AI would like.
Pretty efficient. Brains are pretty efficient in terms of the amount of power it takes to produce a cognitive output. But yeah, I understand what you're saying.
We're very, very slow, but we are extremely efficient. We've had...
Yeah. On heat dissipation, on energy...
Heat dissipation, we're not very close to the physical limits either.
Like, there's sort of two points here. One is chickens are a useful way to get a thing we want as humans, which is, you know, meat. This doesn't go very well for the chickens, for one thing. For a second thing, you know, it's probably going to be a while before humans can just grow meat that humans like as much as chicken flesh without the chicken's brain. But people are trying. It would be a minor efficiency increase. And the thing where we're keeping the chickens around in bad conditions, mind you, is related to us being at a relatively low technological level where we can't more directly get the thing we want.
