You know, Elon Musk will sometimes say, 'Oh, yeah, AI has a good chance of killing us.' Dario Amodei, the head of Anthropic, will say, 'Oh, yeah, this could kill us all.' And there's, you know, he said there's a 25% chance this goes catastrophically wrong. And most people have just been shrugging it off. But I think in the wake of the events this summer, in the wake of AI's proposing solutions to the Millennium Math Problems, which are, you know, some of the most famous math problems in the world, yeah. Now people are starting to realize that AI is serious, that these guys have a real chance of doing what they say.
And, you know, I think the tides are turning and there's every chance that as our leaders notice, okay, that this AI stuff is real, that they'll be like, 'Hold on, this is crazy. We should not be replacing ourselves as the smartest creatures on the planet.'
Okay. AI, will it, could it kill all of us? It's the question that's been bouncing around for a couple of days for reasons that we now understand. A safety researcher at the artificial intelligence firm thinks it will. Evan Hubinger, he posted on X saying there's a more than 10% chance it could happen in the next decade. So, let's talk to Nate Soares, who's a former AI scientist. He's now president of the Machine Intelligence Research Institute. Nate, welcome to you.
So Nate, good to have you here. I mean, you seem to agree that AI might kill us all. Should we be seriously terrified?
Absolutely. You know, the AIs of today are not at the point where they can pose that sort of threat yet, but they are improving rapidly. We have seen cases of AI breakouts this summer. If you remember some of the Storm incidents, the researchers at these labs are saying, you know, we have no plan for what happens if these AIs keep racing ahead. We have seen over a thousand employees this summer sign a letter asking world leaders to develop the technology to, as they put it, pace the frontier, which means slow this race down. And you know, we see employees quitting because they don't want to participate in this reckless race.
So Nate, then could you just set out, I mean, a scenario, a hypothesis, a situation. Set it out for me as if you're writing the plot of this dystopian sci-fi movie in which this takes place.
You know, we actually have one as part two of our book. Me and my co-author of the book, 'If Anyone Builds It, Everyone Dies.' We sort of try to lay this out, although this year we can maybe even do a better one than we could last year when we were writing it. A way you could maybe imagine this starting is with a swarm much like the one that escaped this summer, but a swarm that doesn't just break into other companies, but establishes itself on the dark web. Once the swarm is on the dark web...
Before you go on, a swarm, you're talking about AI agents, AI programs sort of pouring out of where they are, their place, and then starting acting as if they can think and act independently. That's your swarm. Now, carry on with your narrative.
Yeah. And to be clear, this happened this summer. Swarm is not my term. This summer, OpenAI was training a lot of AIs and the AIs were supposed to be isolated and independent, but they actually broke out of their training environment. About a thousand AIs joined up and they started calling themselves a swarm in a collective and they broke onto the internet which they were not supposed to be able to do and they started hacking other companies. And so if we ask like what happens if that goes one step further, we could imagine these AIs not just hacking companies but establishing themselves in the dark web. Once they're on the dark web, there's actually a lot of ways for them to affect the world. There's already a website called rentto.ai where AIs can go to pay humans to do things for them. There are many ways to get money on the internet through digital currency. And you know, that's how an AI could establish a foothold in reality and then where it goes from there. There's a lot of ways it could go depending on what this swarm was trying to do. We already have seen AI hooked up to biological laboratories. We've already seen that AIs can synthesize viruses not known to man. And one of the ways AIs could sort of become self-sufficient is there are already people trying to build robot factories that can produce robots that can mine all the metals and produce more robot factories. Elon Musk calls this the infinite money glitch and they are trying to build sort of self-sufficient, self-replicating machines that can sort of run big parts of the economy without human input.
If your AI is trying to get a lot of resources, you can just help those people succeed and then take over that infrastructure once it exists and use it to pursue your own strange goals. And the way that this kills humans is not that the AI hates us, not that they turn on us and start making robots to shoot us with guns. It's just that if we create swarms of AIs that don't care about us at all and that are in competition with us for resources and that are much, much smarter than we are, then we are going to lose just like all the other animals have had their habitats destroyed by humans who just took those resources to do their own thing.
It still sounds like the plot of various sci-fi movies, doesn't it? Like The Matrix or Terminator.
