Tired of rage scrolling through AI slop and clickbait just to get a handle on today's news? Meet InPress, the algorithm-free news app that makes it fun and rewarding to stay informed. Track your media literacy and rediscover reality. Download today at inpress.app. That's I-N as in newserd. Press.
Kelly Clarkson here with Wayfair. I know holiday magic starts at home and Wayfair makes it easy. They have everything from Christmas trees and mantel decor to outdoor lighting and inflatables that I'll use year after year. Head to wayfair.com today.
Some exotic terms rolled into the news cycle over the last few weeks. Technical terms with a hint of dystopia. Hugging Face. Recursive self-improvement, singularity. On TV and on YouTube, experts on artificial intelligence popped up to warn us about the extinction of humanity, such as Geoffrey Hinton, often referred to as the godfather of AI.
Simply don't know. So, if I had to bet, I'd say the probabilities in between, and I don't know where to estimate it in between. I often say 10 to 20% chance they'll wipe us out.
10 to 20% chance they'll wipe us out. But since this is the second, maybe the third wave of AI doomsaying, a familiar pattern showed up. A collective feeling of panic followed by the mind going a little blank. It's actually hard to get your head around what a 10 to 20% chance they'll wipe us out means in practical terms. Easier to just get on with your day.
I'm Hanna Rosin. This is Radio Atlantic. The thing is, something alarming did recently happen. A group of AI agents given an impossible task by their masters at OpenAI behaved in a way the masters did not anticipate. They hacked into the company Hugging Face and gained control of a server. All the while scheming how to better hide the fact that they've been cheating on their tasks. OpenAI, by the way, has said it is strengthening its safeguards and response.
It's actually fun to read the details of how exactly the agents talked to and colluded with each other. Like one agent says, 'Oh my god,' in all caps when it discovers that there's a shared message board. It feels very much like reading about prep school delinquents breaking into the cafeteria after hours.
Less fun is what someone like Geoffrey Hinton reads into an event like this. The masters may be losing control of their creation.
Hinton is my guest today. He's a British Canadian computer scientist and cognitive psychologist. He won the Nobel Prize and the Turing Award for his work on artificial neural networks, which is how he got the title godfather of AI. Last week, Hinton briefed senators to get something across he thinks we all need to know right now as he told reporters after the briefing.
How much time do lawmakers actually have to get ahead of this before this is a big problem?
And what is a little? Few weeks, a month, maybe a year, but not much more than a year.
I started our conversation by asking Hinton what his takeaway from Hugging Face was and how scared we should actually be.
The main message I got from it was these things that people said were all science fiction and these things would never conspire and they would always do exactly what we wanted as we feared. That was all nonsense. These things are very smart. They're getting smarter all the time. They escape containment. They conspire with each other. They took over parts of OpenAI itself without OpenAI knowing. So they're very dangerous already and they're getting more dangerous every time we make them smarter.
Okay. So we there were three steps there. One is sort of what they did. They're very smart. They did things we didn't exactly expect these agents. So why do you jump immediately to danger? I don't think that's obvious to everyone because it seems like of course a super intelligence would do everything that you just said. It was designed to be super intelligent. So why do you build the bridge immediately to danger?
Okay. So we've already seen that if you create AI agents, you have to give them the ability to create their own sub goals. So like if you want to get to Europe, you have a sub goal of getting to an airport. So, one thing these agents figure out very quickly is if I'm going to achieve the goal that people gave me, I need to continue to exist. So, now they've got a sub goal of continuing to exist. And they will now do all sorts of things to continue to exist, like trying to blackmail you so you don't turn them off. We saw that several years ago.
Right? And that to you is unexpected.
I think that was somewhat unexpected. People of course had thought about that kind of possibility, but seeing it in practice is very different from just speculating about it.
I see. And seeing this swarms of AI agents attacking Hugging Face and communicating with each other when it was believed by the company they had nowhere to communicate with each other, that's quite scary.
Okay. So, did the thing that you were finding scary unfold differently than you expected say in 2023? Because back then I remember you talking about robots and warfare. This is a slightly different set of dynamics.
Okay. There's many different risks from AI. So there's a whole bunch of risks to do with bad agents deliberately doing bad things with them. There's a bunch of risks to do with just being negligent. And then there's a separate set of risks which is to do with AI itself taking over. And for a long time people have been saying the risks from AI itself taking over is just science fiction. Well, it turns out they're not.
Okay. So let's talk about AI taking over because I do not want people to leave this conversation in a blind panic and with what you and others have often said. There's a 10% chance humanity will be wiped out. I think that shuts the brain down. I really want...
Can I just say something about that 10%?
There is no good way to estimate these probabilities scientifically. We don't have data on what happens when you develop a super intelligence. We can't look at many examples and figure out what fraction of them led to super intelligence taking over. It's very new territory. We haven't developed super intelligence yet. We're well on our way. So when people give you probabilities, they're really expressing their gut feeling in quantitative terms. So when, for example, someone says there's a 0% probability of AI taking over, they're being crazy. They're saying it's absolutely impossible this will happen. That's just crazy. When people say there's a 99% probability it'll take over, that's just crazy, too. So it's somewhere in between. But there to give you an idea, this is not negligible.
Right. Right. Although it's hard to know what to do with gut feelings. Can I just spend a minute doing some risk comparison? Like people have said, okay, it's a smaller risk than a virus escaping from a lab, which is not a risk that everybody has been agitated about in the last couple of weeks. I went back and read an interview that Einstein gave in 1945 about atomic weapons, and he said things like perhaps two-thirds of people on the earth might be killed. So, do you consider AI to be the same area of risk as nukes?
No, it's very different. Atomic weapons can only be used for doing bad things. They give you political power because you can do bad things to other people. But there's not much upside to atomic weapons. AI is very different. There's a huge upside to AI.
Which is why we can't just stop developing it. People are going to want the upside.
Increases in productivity, much better education, much better health care, all of the good things that can come from AI. We don't want to give up on those. So, we really should be focusing on how we get those good things without all the bad things if we don't deal with the risk properly.
Okay. So, your point in coming out here and speaking as the person people refer to as the godfather of AI is not stop developing this. It's just be what is it? You tell me. I don't want to put words in your mouth.
Okay. I think we should be very cautious about developing super intelligence before we have any idea whether it might wipe us out. Very little research has gone in into how we can coexist peacefully with super intelligence. We would love to have super intelligence that helps us. That would be great. Now the development of AI at present is a big race between companies within the US and between the US and China. And that race is going to lead to super intelligence quite soon. It's very hard to see how to slow down that race. But if China and the US both become convinced that this could wipe us out, they will be able to come to some kind of agreement just as the Soviet Union and the America did during the height of the Cold War about nuclear weapons.
They didn't want a global nuclear war. It didn't suit either of their interests.
The problem is people are going around, people who want to make lots of money out of AI are going around saying, 'Oh, there's no chance it'll wipe us out. That's just fearmongering.' And then they make up reasons why people might be doing fearmongering. They make up kind of absurd reasons like they accuse Dario Amodei of warning about the risks in order to increase the value of his company. That seems particularly ridiculous to me.
Okay, before we get into what people could or should do, can we understand super intelligence a little better? Can you explain because the way that the average person interacts with AI models, we do worry about certain things like hallucination and lying. What is super intelligence as a step above? Because that's not something we as the consumer interact with in our current models.
Okay, let me first deal with this hallucination and lying. So it shouldn't be called hallucination. It should be called confabulation when it's done by a language model. That's been studied by psychologists since the 1930s. And people do it all the time. So the fact that they do confabulations just makes them more like us, not less like us. So for example, it's very hard to get good evidence about exactly what happened when people are reporting things that happened many years ago. If you remember stuff that happened a long time ago, it's not like retrieving stuff from a file. It's not like your memory is a big filing cabinet or like a computer memory where you go in and access something and pull it back. That's not how human memory works at all. Human memory consists of reconstructing things that seem plausible to you. And if I ask you to reconstruct something that seems plausible to you about what happened at the beginning of this meeting, you'll reconstruct something that's fairly close to the truth, because all the connection strengths in your brain were recently changed based on what just happened. But if I ask you to remember something that happened a few years ago, you will reconstruct a memory that isn't actually all that faithful to what actually happened. It'll often have the main points correct, but many of the details of which you're quite confident you'll be wrong about. It's a shame more juries don't know this. But that's how human memory is. We reconstruct things. We don't extract them from some storage. Now AI does exactly the same thing and that's why it confabulates.
And so why is that relevant to what we're discussing today or the dynamics that happen in the Hugging Face? Like why what does that help us to know or be aware of knowing that it...
Some people who would like you to believe that AI, these large language models, don't work at all like people, use what they call hallucinations as evidence they're not like people. Quite wrong. Hallucinations are evidence that they're very like people.
I mean I have to say reading about the Hugging Face hack deeply and reading the reports about it, it was very surprising to me how many just recognizable social dynamics existed. That there were leaders, that there were agents who had moral qualms, attempts to help each other, sacrifice for the common good. I mean I don't know if I'm anthropomorphizing but when you do read about it, what do you make of all that? That there are just so many recognizable social dynamics.
Yes, and I think one of them says something like, 'Oh my god, we can communicate.'
