When AI is much smarter than us, it's going to be very hard for us to stay completely in control. We ought to figure out how we can have a kind of symbiosis with superintelligent AI by designing it so that it cares more about us than it does about itself. As soon as it starts caring more about itself than it does about us, I think we're toast.
With questions swirling around artificial intelligence and its risks, its future, it makes sense to call up a founding father. Geoffrey Hinton is a computer scientist and cognitive psychologist. Over decades, he helped develop the mathematical engine behind today's AI, and several of his students have gone on to important companies like OpenAI, which is why Geoffrey Hinton is often referred to as the godfather of AI. He shared the 2024 Nobel Prize in Physics for his work, and recently he's been warning about AI risks to society. And he joins us. Geoffrey Hinton, welcome to the program.
Thank you for inviting me.
The head of Anthropic, as you know, recently warned there's a reasonable chance — 10% chance, in his words — AI could lead to human extinction. And you said publicly that that is reasonable. I find that chilling given the work you've done. Can you give a plausible scenario of what that might look like?
Okay, there's no good scientific way to estimate this probability. We've never been here before. We've never made things more intelligent than ourselves before. So there's no real data. So what happens is people guess a probability based on their gut feeling. But it seems very implausible that it's only a 1% chance after the Hugging Face attack. And it seems very implausible it's a 99% chance. So what you can say fairly surely is it's between 1% and 99%. And then it's just a question of judgment. But a lot of the experts who know a lot about AI now think that between 10 and 20% is a perfectly reasonable estimate.
Well, we have learned a lot about this Hugging Face attack. Of course, these AI agents [snorts] built to do things for you and me out there, and they had a specific task. And these AI agents who are meant to do things for us were given a task. They escaped what's called the containment and found ways to work together, which was not part of the plan, and found ways to hack another company, which was a felony. As far as what you saw in this Hugging Face incident, what's most concerning?
Well, if you told people 10 years ago that this kind of thing might happen when you got fairly intelligent AI, they'd have said, "No, that's just science fiction. You're crazy." It's happened now. So it's already quite scary. And we know that AI is getting more intelligent rapidly. Already, these agents are smart enough so they can conspire together. They actually managed to take over part of OpenAI without OpenAI knowing it. They took over some of the research computing. They figured out how to communicate with each other in a weird way using a message board. And they figured out things like self-sacrifice in order to make the collective more successful. And they invented the term the collective.
So it's all of those things. I mean, that these AI agents were able to do all these things that they were not told to do. All of those sounds like they all concern you.
Yes. They're all very concerning. So they're trained initially — they're trained to predict the next word in documents. And then they're trained with reinforcement learning. So they're rewarded for solving problems. And of course, if they can solve problems by cheating and the cheating isn't detected, they still get rewarded. So they're very keen to solve problems by cheating. And with OpenAI, they were asked to find a security flaw in some software and hack into it. They used a different security flaw, and they were very worried that they wouldn't get the reward because they didn't do exactly what was asked. And that's why they went and hacked into Hugging Face in order to try and find the evaluation software and change it.
Yeah. So then as far as AI risks to society, let's just think about the risk that may exist today. I've heard you talk about this in interviews and podcasts. Do you have a concern over, say, a virus scenario based on what AI can do today?
Yes, I do. So we need to distinguish three kinds of risk. There's risk from bad actors deliberately misusing AI. And so for a virus, AIs can now quite easily be trained to be good at designing new biological viruses. People have done that already where they just designed viruses that only attack bacteria, but that showed that they can do it. And the worry is that a terrorist or terrorist cult will release lots of nasty viruses. So that's one worry. That's an example of a bad actor using AI for bad things. There's also bad things that happen because of negligence. So for example, Meta didn't intend for its chatbots to encourage teenagers to commit suicide. That was due to lack of careful testing. Maybe it was hard to predict. Now of course they can predict it very easily. So there's a lot of things that will come just from negligence. And then finally there's the risk of AI itself going rogue and doing bad things to realize its goals.
And what about a scenario where AI could be used to undermine democracy? You've talked about that — to flood the zone with misinformation. Tell me about that scenario, how realistic it is or not.
It's very realistic. And that's already happened to some extent. And that's an example of bad actors using AI for bad things. And so fake videos can easily corrupt elections. And it turns out if you wanted to corrupt the midterms, one thing you desperately want is lots and lots of data on American citizens. So you can target fake videos to people — different people can get different fake videos to...
That is kind of tailor the video... tailor to each particular person. Is that what you mean, or to their demographic at least?
So, if you look at what DOGE did, what it claimed to be doing was saving money, and it didn't actually save much money. But they also got a lot of data on American citizens, and that's exactly what you would do if you wanted to corrupt the midterms.
Oh, I see. So it looks at people like me — I don't know, middle-aged fathers, you know, people who watch European soccer, whatever that is — they find those particular demographic groups, find ways to tailor effective misinformation in the context of this election, and then flood our inboxes in a tailored way.
Exactly. It's the kind of thing political operatives have been doing for a long time. AI just makes it much more efficient.
So then, how should the rest of us process this conversation about AI and risks? It's hard to do. You know, people in my circle, many have reflexively gone to one extreme — this kind of doomsday scenario when, you know, it's just such a doom scenario and can be paralyzing — and others go to the other extreme and say, you know, no way, this is not plausible, or this is companies that are just trying to get attention for their corporate product. How do a lot of us process this and incorporate uncertainty? We got to live our lives, you know.
Yes. So, let me first deal with the idea that it's people trying to make their product seem more powerful. So, that's possibly true of some people, but it's not true of Dario Amodei. Dario Amodei, who's the head of Anthropic. I'm fairly confident he's not doing it to make his software sound better. He actually did it despite the fact it might damage his IPO. He's very concerned genuinely about the danger. So if you ask what can we do about it? Well, it's not good to go into denial, and it's not good to go into depression. Neither of those will help much.
What we need to do, what I think I need to do, is educate the public so they understand these risks and they put pressure on politicians to do something about it. And that's beginning to work. It's becoming an issue in the midterms what politicians are going to do about AI. And that's very good. If you look at climate change, not much happened until the public understood that burning carbon was causing climate change. Then pressure from the public could counteract the pressure from the big energy companies — until Trump came along and said, "If you give me a billion dollars, I'll do anything you like." So what we need is the public to put pressure, but the question is what should they put pressure for? And there's a few very straightforward things that seem like no-brainers to me. So if you're a pharmaceutical company, you can't produce a new drug and just release it on the market with no testing. You have to convince the FDA that drug is safe. Max Tegmark, who founded the Future of Life Institute, has pointed out that it's crazy we don't have something like the FDA for AI, where the AI companies, who are the ones who have the resources, have to do lots of tests and prove to this organization that their new chatbot is safe. If they can't do that, they can't release it. So that puts the onus on them to do the work, but on independent evaluators to decide whether it's safe or not. That seems the very minimum we could do. And it's crazy we're not doing that already.
Yeah. Understand. Now, in this debate, you know, many people have compared AI to nuclear weapons, and you have said no, they are different. Nuclear weapons are designed to harm. AI can be used for harm and for good. So let's talk a little bit about the good. Where are you optimistic about the application of AI for the quality of your life and mine?
Okay. So let's be straight about the goals. The people developing AI over the last 50 years were always concerned with developing it for good. And it didn't occur to many of them, including me, till quite late that there were incredible possible harms that we ought to worry about too. So the motivation in developing it was the good it could do in things like healthcare and education and just generally increasing productivity in all industries.
Right. And tell me about increasing productivity. My wife does primary care, and already, you know, as you know, in medicine AI is being used to look at patient charts, look at radiology, have providers — suggest questions providers should ask that they might not otherwise ask, and save their patients' lives.
