Over these past few weeks, we've heard from both Anthropic's Dario Amodei and Nvidia's Jensen Huang, each equally certain that artificial intelligence either desperately needs controls on its growth or that it should be left alone to thrive and change the world. So, where does that leave us? It's at times like this we turn to our David Pogue.
They say it got smart. A new order of intelligence. It's one of the oldest plots in science fiction. Open the pod bay doors, Hal. Humans losing control to computers. I'm sorry, Dave. I'm afraid I can't do that. But this summer, the old trope of AI going rogue left the realm of science fiction. And it happened inside this unmarked San Francisco building. It's the headquarters of AI giant OpenAI.
The incident has been dominating the news. You may even have heard the phrase human extinction. The technology could kill all humans. Well, I have a few questions. What exactly happened? Why and how would AI want to kill us? Who can stop it? And where do things stand now?
Maybe you've tried out ChatGPT or Claude. AIs like these can generate writing, music, video, art, you know, make an oil painting of an RV on storm-tossed seas. But the big AI companies are racing to make AI much more powerful, superintelligent, in fact. And in May, something went very wrong.
To evaluate their experimental bots, these companies routinely give them goals, like capture the flag, except the prize is finding a code. One of the bots tried to win the challenge by cheating. Behind the humans' backs, it set up a message board to communicate with other AI bots. Working together, they escaped onto the internet and hacked into an AI company called Hugging Face, seeking information that could help them beat their tests. And then, some of the bots tried to erase their tracks.
What was it about the Hugging Face incident that freaked everybody out so much?
The fact that it involved an autonomous attack on another company. Like a felony, you know, like breaking the law.
Daniel Kokotajlo once worked at OpenAI, but 2 years ago, he quit to protest what he saw as reckless AI development. Now, he runs the AI Futures Project, which is dedicated to predicting how things will go. He's alarmed that the AI companies are moving toward recursive self-improvement, meaning that AI trains AI to get smarter. People are now taken out of the loop.
Things will be going faster and faster and faster, and it seems to me like that is very likely to go wrong if it's attempted.
And then, just when you thought it couldn't get any worse, it got worse. OpenAI revealed that the Hugging Face hack wasn't the only time its bots had lied, cheated, and hidden its tracks from humans. Its AI bots had committed at least 13 similar incidents. On September 8th, an Anthropic researcher named Jacob and resigned in a very public way.
In the next 10 years, if we don't change the way things are going, uh greater than 10% chance of human extinction.
All right, question two. Why would AI want to wipe us out? Well, the key is that AI will soon be much smarter than we are.
We're so used to being the apex intelligence. We just can't think what it would be like not to be the apex intelligence. I try and get this into people's brains by saying, "Look, if you want to know what it's like not to be the apex intelligence, ask a chicken."
Geoffrey Hinton won the Nobel Prize for his early work on AI.
So, for example, you can imagine a scenario. You give it the goal of reducing the carbon dioxide in the atmosphere. That would be a very good thing to do. Um and it, being intelligent, figures out the best way to do that, which is get rid of people. So, that's one worry.
There are many possibilities. I'll illustrate one. Business uh asks their AI to make them as much money as possible.
Alex Turner worked at Google's AI division, but he, too, resigned in protest this summer.
So, the system, in pursuit of making money, could uh do a large takeover through methods like remote drone strikes and um a potential plague. So, if an AI were this misaligned, and it were just following the letter of the instruction, then it might do something catastrophic.
