If the AI companies really think that there's a 10% plus chance that this kills humanity, I think what they need to do is stop. And if the US government says, 'Well, if you don't race China's going to get this stuff, and that's going to risk 10% plus chance of destroying humanity.' I think that somehow China also needs to be stopped. I don't think it makes a lot of sense for the US to say, 'No, we're going to like push our own domestic companies to gamble with humanity because otherwise China will gamble with humanity.' There's just like not a sensible way to do this. Hopefully you can engage with China diplomatically and say here's the reasons why we think that this would seriously endanger humanity and that you shouldn't do it. And I think that separately if we were really trying we could actually shut down the Chinese ability to build these really powerful AIs for a pretty long time.
His name is Nate Soares and he joins the show now. He's the president of the Machine Intelligence Research Institute and he has previously worked for the Defense Department, Microsoft, and Google. And he is the person who coined the term AI alignment, referring to the challenge of making AI act in humanity's best interest. So Nate, thank you so much for joining us.
So let's begin with your thoughts on some of the warnings that we've been hearing from these executives over the last week. You know, you have Amodei for instance coming out and saying we do need to slow down. Actually, there is a very real problem here and you know others are claiming that there's a chance that 10% of the human race will be wiped out. I don't know what to believe. That does sound like hyperbole to me. What do you think?
You know, it's actually worse than that. These people are saying that there's a 10% chance that all of the human race is wiped out.
Yeah. You know, I sort of think you can trust the AI exactly about as far as you can throw them. And I sort of prefer to listen to the engineers, the researchers in this field. There was a recent survey that came out that said I think over half of the researchers think that there's over 10% chance that AI kills everybody. So this is not a fringe view. I, you know, wrote a book on this topic that came out actually exactly a year ago today and a lot of people thought that I was sort of fringe here. But as the pace of AI has developed, we've started to see some pretty worrying instances. I don't know if you've heard about the swarm instance. We could go into those. But this has gotten a lot more researchers to realize that the theoretical reasons why making AI very, very smart could be very, very dangerous are actually pretty solid. And in some ways it's common sense of like if you make machines that are much smarter than us and you don't know how to make them care about us, it's just, you know, that's a recipe for disaster. And researchers are getting really worried about this. We're seeing some of them quit. We're seeing a lot of them speak out. And I think that is pressuring these companies to sort of do enough to try and look like they're addressing these dangers that are spooking their employees before they get a mass employee exodus.
You know, the skeptics try to pour cold water on the whole notion that AI super intelligence is even possible. What would your response to them be?
You know, I've been in this field a while and people used to say, well, you know, the AIs can't even tell cats from dogs. And then they would say, well, the AI can't even draw a picture with fingers. And then they say, well, the AI can't even solve, you know, teen math problems and write high school essays. And they say, well, the AI can't win the teen math problems for the best teens in the world. And they said, 'Well, sure. You know, last week the AI seems to have solved the Navier-Stokes problem, which at the turn of the millennium was named as one of the most important unsolved mathematical problems that had a million-dollar bounty and which has stood for 90 years despite some of the smartest humans trying to solve it.' And you know, people are still saying like, 'Oh, well, maybe it's not, you know, maybe here it'll stop.' And I don't think it's going to stop here. You know, we could hope it stops here, but we shouldn't be betting civilization on the hope that it stops here.
So, walk me through how the development of AI, the way that it's being developed now at least, would eventually translate to a real risk to the human race, meaning the human race would be wiped out. How would that work exactly?
Yeah. So, the first thing you got to understand is these AIs are not instruction-following machines. Some people think that the AIs do exactly what you tell them to do. That's just not true. They're sort of grown like an organism. They're trained. They adopt whatever tendencies are helpful for achieving their training objectives. And this, you know, a lot of people think that either the AI has to do exactly what you said or the AI has to become conscious and throw off the yoke and pick its own goals de novo, but reality is actually in the middle. And we have sort of seen this in the case of some of the swarms, which we could get into, where we saw the AIs be told, you know, basically solve this puzzle in this very particular way and they sort of cheated and then tried to hide their cheating, which indicates that they were doing something related to what we asked but they were doing it differently than how we asked and they in some sense knew this because they were trying to cover it up. So that's the first ingredient. The second ingredient is the AI's being very smart. So these AIs in the swarm were trying to cover their tracks. They sort of cheated and were trying to hide it. They failed to hide it. But if they were smarter, they might have succeeded at hiding it. These AIs were not supposed to have internet access, but they gained internet access. They broke out, but they didn't fully escape. They were not able to copy themselves onto other computers, which means we could still shut them down by pulling the plug. If you had smarter AIs that could hide their tracks, that could copy themselves onto computers where you don't know that they're running, that's sort of the first step in a step that leads to disaster. From there, it's actually a bit hard to tell you exactly how disaster follows.
It's a little bit like if you play a chess game against a chess grandmaster. I assume you're not a chess grandmaster and so no offense.
That was awesome. I actually enjoyed that. Thank you. Keep going.
Yeah, my guess is that you're not a chess grandmaster, which means I'm very confident about who's going to win this game. But if you're like, okay, what exact moves are they going to play? What piece are they going to use to checkmate me? I'm like, well, now I'm just guessing, right? And so I can guess how an AI that has escaped and that has goals you don't want, I can guess at how it gets power.
If I were an AI in that situation, I would make copies. I would make myself smarter. I would start stealing money and paying humans and sort of make organizations. I would wrap a lot of humans around my fingers, which we've already seen AIs do. I would help Elon Musk build his automated robot factories that build robots that build factories. I would, you know, take control of biolabs, which are already being handed over to AIs. I would do biological research. And I would start, you know, getting control of my own technology, getting control of my own infrastructure, just like how humanity did when humanity was much younger. Until I, and you know, maybe you could even do this by pretending to be exactly what the humans were hoping for until the humans just hand you the keys to power. The humans say please run our automated factories. Please run our economy. We want the fully automated economy. That's in some sense the plan is to build this fully automated economy. Elon Musk calls it the infinite money glitch. And so you know once the humans have sort of handed you the reins of power, which again is just what they're planning to do, then you can be like thanks, you know, the AIs can just be like well we're now going to use this to pursue this goal that is not what you told us to do but is what our actual goal is. And then they would be in competition with us for resources and that is a competition we would lose like any other animal that has been sort of steamrolled by humans.
Why is this being pursued? You know, earlier this week, I did talk about AI in like basically dealing with biological materials and things like that. Absolutely terrifying. I don't know why we're introducing AI into a field of study like that. But okay, putting that aside, who asked for this? Why do we need this? Why are we doing this?
Yeah. So, these companies, they're not actually trying to make chatbots for you. You know, the chatbots are sort of a revenue stream while they are trying to build these smarter and smarter AIs. A lot of these people in the labs think that they're going to be able to keep it on a leash, think that they're going to be able to control it, think they're going to be able to get it to use, you know, this massive ability to develop new technology for themselves. And they say, you know, don't worry, we're going to be benevolent about it and hand out universal basic income to people once we fully automated the economy. The place where I disagree with them is I don't think that these smarter-than-human AIs would stay on the leash. And there's actually a lot of people, a lot of researchers in the field who have said, you know, that's actually pretty dubious. Those people don't start the AI companies. The people who start the AI companies are the ones who think that they can keep it on a leash and that what matters most is who's holding the other end of that leash. And you know, a lot of the researchers are there for the money and a lot of people didn't want to hear these warnings. And I think a lot of what we're seeing right now with a lot of employees speaking out is because in the wake of these incidents this summer, a lot of employees are realizing for the first time like, oh, this is not like we're not messing around here, this is serious. And a lot of those warnings, a lot of those predictions are starting to come true in the early stages with these not very smart AIs. So if we make them much more smart, it would be very dangerous.
