Professor Bengio, you are a godfather of AI. So they say. One of the OGs in this field, if you will. You could have gone to work for probably any of the many AI labs right now and receive millions, maybe more, in compensation for it. Why have you never gone to a major AI company?
Actually, a number of reasons. I feel like as a university professor, I make a decent living. It's not a crucial thing for me. What has been crucial for me is working for a mission that I felt good about, and I was able to do that in academia. Now, I created this nonprofit, Law Zero, which has focused on something really important to me, maybe for the future of our societies, which is AI safety. How do we build AI that will not harm people? And it isn't something that, in my opinion, gets enough attention in industry.
What are the most worrying emergent behaviors that you're seeing today in AI?
Many, but I can summarize many of them under this rubric that we are not succeeding in making sure that AI will behave in a way that doesn't violate our red lines or our safety instructions. So it shows up right now with methods, for example, where it is quite easy for somebody to use the AI to launch an attack, a cyber attack, even though the AI has been instructed not to help for this kind of thing. But it's true for other misuses of AI. It shows in what's called sycophancy, where the AIs, in spite of the instructions, are doing a little bit too much to approve your thoughts, your emotions, and potentially bring you in a dangerous place, which has given rise to self-harm, for example. It means that these systems somehow are willing to lie, which they shouldn't, in order to make you feel good, for example. And it shows up in places where I think it really matters for the business development of AI, with agents that sometimes go against our instructions and can be really dangerous for our software infrastructure or databases, as we've seen some examples of that. And finally, it shows up in these behaviors that are really concerning and scary as far as I'm concerned, where the AI chooses to go against our instructions, like lying, cheating, and hacking in ways that they're not supposed to, in order to preserve themselves, to make sure they're not being shut down, and more recently, to preserve other AIs, which is weird, but this has been observed across many of the leading models.
So we're building systems that seem to have their own goals that we don't fully control, and are getting smarter and smarter. I think we should be concerned. Do you think Mythos was a genuine turning point for industry? And why do you think people who were maybe even skeptical before about the need for regulation in AI, that it could hinder industry, why do you think Mythos should change their mind?
So many in industry have been broadcasting this view, which clearly was false as shown by Mythos, that AIs would never be powerful enough to be a danger to society, for example, in the hands of third parties wanting to use AI for really attacking our infrastructure. And that meant, well, if AI is just another technology, ordinary technology, well, you know, we can just let companies regulate themselves. But what this is showing is that we reached a point, and it's going to get worse, where AI can be weaponized. And as such, it becomes a national security issue and in fact, a global security issue, because AI could be developed in one country and then in a second country, somebody uses it to launch an attack on a third country. So it also raises international coordination questions, which is probably related to the fact that there were discussions between the US and China in the last visit of Trump in China. And I think this is just the tip of the iceberg. Like if we think of the Mythos event as a one-off, we are wrong. If you look at, for example, the cyber capabilities, we could see it coming. Like if you look over the last year on benchmarks, they've been stronger and stronger and it's going to continue. And in, you know, 6 or 12 months, probably the Chinese models will be at the same place. And then you have to think it's not just cyber, right? So knowledge gives power and that power could be misused or we could lose control of it. So we need to make sure that power is under check. And we can't rely on every company doing the right thing. We need governments to make sure everyone evaluates the risks and then acts accordingly.
The Pope recently issued an encyclical on AI. I understand that you have also been involved in some conversations around what role maybe religious leaders have to play in this discussion about AI safety. Can you tell me a little bit about your thoughts on that?
Yes. I was in the Vatican last fall, and the Vatican has been consulting a number of experts on AI because of the same reason I was talking about: intelligence gives power and the power of AI is growing. That means AI is potentially going to transform our societies. But there are many possible futures. And with power, who chooses towards what goes becomes a moral question, and it becomes a democratic question. What do we want? Who decides who is going to benefit and who is going to lose? Like if you think about it, the Pope is very much concerned about the jobs issue, but also how AI is transforming who we are, how our children interact with technology, how it changes their minds, our minds, how we interact with each other as human beings. And I'm really delighted that he's chosen to put humans at the center of these discussions, like human dignity. And that is how we should think of steering the technology in a direction that's good for humans. Because, you know, we could go in bad places. The good news is we have agency. Human societies can choose. It doesn't have to be just the forces of competition. We can put guardrails, and that's what he's telling us.
So let's talk about those guardrails and technically what you're working on with your startup Law Zero and on a far smaller budget, right, than some of the major AI labs out there. But what do you find most promising right now? How confident are you that you're going to be able to develop an AI that is significantly more safe than what we have today?
