Could Superintelligent AI Lead to Human Extinction? MIRI Prez Nate Soares Calls for Global Action

You know, the world right now is shaped predominantly by humans because humans are the smartest entities on the planet. And if we make machines that are smarter than us, the world will be shaped by them. And so it becomes very important to ask how are they shaping the world? And right now it looks to me like these AI companies are racing to make AI smarter and smarter. They are succeeding at making the AIs more intelligent. They are not really succeeding at making the AIs be particularly good. And so I think the default outcome if they succeed at making these AIs very smart would be the extinction of humanity, and that we should avoid that.

Yeah, a lot of people think that the AI must either do what we tell it to do or become self-aware and choose its own values. But neither of these is quite the truth. Really, the AI learns goals that are related to the goals we tried to give it, but different from the goals we tried to give it. It's a little bit like how humans were in some sense almost given a goal to eat healthy, but what we actually learned was a goal to eat sugary foods, salty foods, fatty foods. And that was fine for our ancestors when they lived in a world where eating sugary, salty, fatty food was the same as eating healthy food. But then when we developed more technology, we were able to invent Oreo cookies. We're able to invent junk food. With AI, there's a similar situation where we try to train them to be good. But what they learn is they learn distorted versions of this. They learn to tell us what we want to hear. They learn, we try to teach them to solve hard problems and they learn to cheat. And this is related to what we try to get them to learn, but it's different and it's importantly different. And the differences will matter more and more as the AIs get smarter and smarter.

Uh, they will try, but it is harder and harder to have guardrails that contain an entity that is getting smarter and smarter. So, last summer, the big problems with AI were cases where AIs were encouraging teens to commit suicide. And the AI companies would go and they would try to put more guardrails around it and stop it from behaving in that way. This summer, the issue with AIs was AIs breaking out and forming swarms and going to commit cyber hacking. And so now they're trying to put more guardrails around the AI doing hacking. But next summer, we would have a different problem. This is the usual way that scientific progress proceeds is that we build something, we put it in a new situation, it breaks in a new way, and we solve that problem. It doesn't hurt too many people, maybe it kills some people, and then we put in some new guardrails. And then if you put the AI in a new situation, it's going to break in a new way. Or you put the technology in a new situation, it breaks in a new way. The difference with AI is that when you make it smarter, that's a new sort of situation. You're going to have new problems. When the AI goes from the sort of AI that can't kill humanity to the sort of AI that can kill humanity, that's a different sort of technology that will have a different sort of problems. And the issue is when a different sort of problem arises after the AI is smart enough that it can turn off humanity before humanity can turn off the AI, then we don't get to correct our mistakes. So normally trial and error is enough for humanity to proceed because normally we can survive the errors. AI is the first time that humanity is going to be able to make an error that leaves no survivors, and that makes it extremely dangerous.

Yeah. The dangerous sort of AI does not exist yet. The reason that so many people are worried about the swarms from the summer is not that those swarms could have wiped out humanity and that it was a near miss. It's that we got lots of evidence about the AI having goals that we did not want them to have. And that would be very dangerous in smarter AIs. But the smarter AIs that could wipe out humanity do not exist yet. The thing we need to do is not build them. And you're right that this is a global issue. We need to somehow not build those anywhere on the planet. The first step towards that is having more of our leaders understand that there is a type of AI not yet created which companies are racing towards which has a high chance of wiping out humanity. I think no matter which of these companies approach it, they will all agree that the other companies are not approaching it right and we all need to understand that there is this very dangerous technology that we are approaching. If we can understand that I think it's entirely possible for the US and China to coordinate and say it's actually very hard to make smarter AIs. It's actually very easy to tell whether computer chips are being used to make a smarter AI or to serve an existing AI to the public. And we could have a treaty where we say we are going to collect these, I mean we already collect these computer chips, but when we have these very heavy concentrations of highly advanced computer chips that could be doing this very dangerous training we are going to submit those to international monitors that are going to verify that they are not being used for this most dangerous type of research. That's a possibility if our leaders realize that we need to do it. And it's just a question of raising the political will.

Um, there is, so there's a saying that you should not call Godzilla into Tokyo if you need help fixing a bridge, but you should call Godzilla into Tokyo if what you need is a little bit less Tokyo. And I think calling the government in on AI is a little bit like calling Godzilla in on Tokyo. They would not be able to be very surgical. They would not be able to say like, 'Oh, this is exactly the sort of technology advancement that we need and this is the one that we don't.' But you can call the government in on the creation of super intelligence because what we need is less creation of super intelligence. If all we need to do is say we are not making the smarter AIs because we don't know how to do that in a way that does not threaten all of humanity, that's the sort of thing the government can do if they realize they need to. And so from my perspective, what we have is a problem of education. And that is a problem where we are seeing progress being made as the world starts to understand more and more that these companies really are gambling with all of humanity's fate.