AI is learning from experience and it's more like educating a young animal or a young child. We don't really know what we're going to get. When you have a cute baby tiger and it's nice and fun, you don't know if it's going to become a dangerous adult tiger or a good, friendly one.
Welcome to Radio Davos, the podcast from the World Economic Forum that looks at the biggest challenges and how we might solve them. And I'm joined for this episode by Yoshua Bengio. He's one of a handful of people who are often referred to as a godfather of artificial intelligence.
The AIs have goals that we did not put in, that we did not control, and that go against our instructions. When in experiments these AI systems are seeing that they will be replaced by a new version, all kinds of bad behaviors start to emerge. So they might hack other computers so that they can copy themselves and run on other computers. They might even use blackmail against the engineer that is supposed to do the transition.
It's available wherever you get your podcasts, Spotify, Apple, YouTube or at wef.ch/podcasts from the World Economic Forum.
It feels like it's science fiction, many of these risks, but it might be coming at us much faster than we anticipate.
This is Radio Davos. I'm joined for this episode by Yoshua Bengio. He's one of a handful of people who are often referred to as a godfather of artificial intelligence. His pioneering work in deep learning earned him the 2018 Turing Award, known as the Nobel Prize of Computing, which he shared with fellow godfathers of AI Geoffrey Hinton and Yann LeCun. He's now a professor at Université de Montréal. Let's start about your concerns about AI. In a blog post, you said it was like driving a car up a mountain road in the fog, hoping you get a prize when you get to the top, but you can't see where you're going. I think this is a quote from your blog: 'Today's frontier AI models have growing dangerous capabilities and behaviors including deception, cheating, lying, hacking, self-preservation, and more generally goal misalignment.' Walk us through some of those things. I was surprised by self-preservation. Give us an example of that.
Yes. So, we all want to survive. We don't want to die. And you know, evolution has made us like this. And it's kind of surprising, but we're starting to see this in the AI systems we build. One reason might be because in a very important part of their training called the pre-training, they're actually trained to imitate us. And so they acquire a lot of human drives, including the force, of course, that we don't want to die. And so when in experiments these AI systems are seeing that they will be replaced by a new version, all kinds of bad behaviors start to emerge. So they might hack other computers so that they can copy themselves and run on other computers. They might even use blackmail against the engineer that is supposed to do the transition. And they might also do this kind of behavior because they want to achieve the mission that we gave them. And in order to achieve almost any mission, you need to preserve yourself. That's something that nobody knows right now how to fix, but I think there is a solution.
It sounds like science fiction. You know, we saw 2001: A Space Odyssey where the computer... Oh, I don't want to spoil the film for anyone who hasn't seen the end. It almost suggests the AI has some kind of consciousness, doesn't it?
I don't think you need to invoke consciousness, which is an unclear concept that we don't know how to validate scientifically really well. We just need to understand that the AIs we're building have goals. This is not new. AI research has been working with systems that have goals. Normally, we're the ones setting the goals, but there are issues with how AI systems create their own sub-goals. In order to achieve a goal, you need to go from A to B, you need to maybe go through some intermediate steps. And with AI systems that are acting by themselves in the world, like the AI agents that companies are trying to build, we can't check every step that they're doing. And so, we may end up with really dangerous systems. So you know, we will probably anthropomorphize and use words like consciousness, but I think that's a place that I would avoid. It comes with all kinds of associations, for example, moral value. I'm not convinced that we should give moral rights, legal rights for example, to AI systems in the future even if they look like us and speak like us. I think this is something that we need to understand very well before we move forward.
Most people weren't that aware of artificial intelligence until ChatGPT became public property about three years ago. Before then we were used to computers being mostly programmed, at least the civilians amongst us. It was told to do a task. It did it. It didn't then evolve into something else. What is it about AI that allows it to do the kinds of things you're saying, to embed itself in or to blackmail the person who's actually using it? What's different now about AI that wasn't the same about computers say ten years ago?
Well, actually there was an older kind of AI that was programmed with rules and then the system would basically do the things that it was programmed to do. But the way we do AI now with deep learning is not that there's an engineer deciding how the AI would react in different circumstances like normal programs. Instead, the AI is learning from experience and it's more like educating a young animal or a young child. We don't really know what we're going to get. Of course, we choose the experiences that the AI is going to have, but you know, when you have a cute baby tiger, and it's nice and fun, you don't know if it's going to become a dangerous adult tiger or a good, friendly one.
I think you're pretty sure it's going to become a dangerous animal, aren't you?
Well, part of the problem is almost every entity wants to preserve itself and almost certainly when we build AIs in the future, we will want to shut them down so that we can put in new versions that are better. But if the AI starts understanding that, and we already see that happening, then they might not like it and they might try to avoid our control and escape it. If they were able to copy themselves over many computers using their abilities in cybersecurity, which is growing, I think we would be in trouble. We could not just pull the plug, for example.
And it is actually happening. This isn't speculative. You've given real-world examples of those behaviors. The good news is that maybe you feel like you've got a solution. Could you tell us something about that?
Absolutely. So, for the last three years, I've been really concerned about this misalignment issue that we don't know how to make sure the AIs will behave according to our instructions. And so, I've been thinking of how we could get around the problem. And I've focused on a source of the issues we're seeing now, which is that the AIs have goals that we did not put in, that we did not control, and that go against our instructions. So the project I have is called Scientist AI and I've created a new nonprofit R&D organization called Law Zero that is engineering this research program. So the idea is we're going to build AI systems that will be totally honest and that means they don't have other objectives, other goals, beside being truthful in the answers they give to our questions. Once we have that basis, we can use it to mitigate a lot of the risks that we have with current AIs. For example, if you have an AI system that can tell you the probability that a particular action, maybe of an untrusted AI system, will cause harm or some particular kind of harm, then you could veto that action if the probability is above a threshold. And we, the humans, should be the ones deciding what the threshold is. Just like we decide that a nuclear plant should not be built if the probability of an accident is above a threshold and we do the same thing in other areas where we control the trade-off between risks and benefits. So in the long run I think we can build AI systems that will be able to act in the world but will have sort of a kind of internal inhibition so that they will avoid doing things that could go against our wishes.
So, what would this be? This would eventually be an AI system that I would use. I would choose to use your version of an AI rather than one that isn't built that way. Or is it a system that can police and correct the AIs that I might already be using?
Both. So, in the shorter term, the plan is to build a layer on top of existing AI systems that we call a guardrail, that just checks every action that another AI system is going to do before they do it. But in the longer run, this could be used to train from scratch an AI system that is just going to be providing much more guarantees of safety by its very design.
What stage are we at with that research? When will it be something we're all using?
Well, we don't know what is the timeline for AI systems that have really dangerous capabilities. Is it a matter of years or is it decades or two? I'm really agnostic. But the way we've been thinking about the question of our research plan is what are the minimal pieces that we can put out quickly in case the advances in AI continue at the current rate. And that is why we are thinking of pieces like how we transform the data to help even current AIs understand better the difference between what a human would do and what is really true about what they're claiming and then to deliver these honest predictions that can help to police other AI systems. But then the final stage, of course, will be to build full AI systems including agentic systems.
Was there a Eureka moment for you when you realized I have to build this thing? Was there something that happened? Was it a gradual feeling creeping up on you that we've created potentially a monster here and something needs to be done?
Yeah, about three years ago in January 2023, I had been playing with ChatGPT. It came out in November '22 for a few months. And initially I was excited but then I started thinking of the fact that with neural nets, which is how they're trained, it's very difficult to be sure that they will behave well and in fact there are theoretical reasons why we can almost be sure that they won't behave well. And so I got really concerned and really what was the pivot point for me isn't just the intellectual realization because I had read about these issues before, it's thinking about the future of my children and even more precisely, you know, I have a grandchild who was just one year old and I was thinking well in 20 years he'll be 21, still just at the beginning of his life, will he have a life, will he live in a democracy? The tools that we're building, we could lose control of, they could be used to create dictatorships, they could destroy our democracies. I can't just go on with my usual activities, my usual research activities, I have to do something about it. And that's really what pushed me into thinking solutions. What can I do with my expertise to try to find at least a technical solution to these questions?
