Thank you so much and thank you for letting me have this conversation today. I've been looking very much forward to it. Yoshua, apart from being called the godfather of AI, you also have a formal title which is as a professor of the University of Montreal. You're the co-president and scientific director of Law Zero, an NGO. We're going to talk about a nonprofit, and you're also the founder and scientific advisor of Mila. So that's the formal title. And apart from being called the godfather of AI, you're also sometimes called an AI doomer, but I know you don't entirely agree with that term. We can get back to that. So as we have already talked about a lot today, AI systems are moving rapidly and not just answering questions but also starting to reason across complex problems, executing tasks autonomously. We talk a lot about AI agents, and there's also been a lot of talk today that not all of us entirely understand AI and how it works. So I think maybe you can just give us a small crash course the next 10, 15 minutes about how do we actually see these AI systems moving towards not just solving tasks but also lying to people and sometimes even preserving themselves. So preserving the AI system against human intervention, for example, not being able to shut down an AI system. How did we get there?
By the force of competition and greed for domination. That's the reason we are in trouble. The companies are racing fiercely, and even when they're led by people with goodwill, it's a question of survival. If they don't do the same thing as the other ones, they're going to disappear. And so the only way around is democracy, you know, that we make rules together that will apply to all the companies so that we can manage the risks and bring the benefits.
And we also talk a lot about how these AI models have this sycophancy. I mean, they make up things. And for a long time I thought, okay, that's a really bad thing. I still think, but I also actually thought that's a very human thing as well because we all lie a lot, apparently that says the least. So how should we deal with this? And why is that actually, why does that have to be part of these AI programs?
It doesn't have to, but the way that the companies in the US and China and also in Europe and Canada are training the AIs now naturally leads to this. So there are two main phases in the training of AIs that give rise to many of the problems, including sycophancy. In the pre-training phase, which is the biggest part of the computational effort, they're trained to imitate us, which sounds okay except that they're not human and they're supposed to be tools. They're not supposed to try to preserve themselves, or more recently we've seen they're trying to protect other AIs from being shut down and lie in order to achieve that. We would like our tools to be honest. And by the way, I think it's a huge mistake that we're collectively making to try to turn AIs into humanlike entities. Even though there's a huge demand for that, it's not healthy and there's a lot that we don't understand about the consequences for children but for society in general. So that's like this human imitation, I think is a very dangerous path. We should be building tools to solve the problems we actually have. And then there's a second phase of training called reinforcement learning where the AIs are taught to achieve goals and do whatever it takes, and that also creates problems because sometimes in order to achieve the goals, they will do things that go against our norms, against our ethics and so on.
So the good news, you mentioned Law Zero. So this is an organization I created last year to find a different way of training AI that wouldn't have these problems. Something I think really important for Europe is to know that there is a way to build AI that will behave well, that will respect our red lines, that will respect our norms, and of course also those norms don't have to be the same in every country, right? That will not have its own goals or its own norms, and that we'll have guardrails to make sure that those rules that we set are respected, because right now you can put as many rules as you want to the AIs and they will gladly violate them. So this is the first thing. If we kind of remove ourselves from the competitive pressures that are working on those companies, there's a way to design AI that will be aligned with our democratic and human values, but it isn't where the money wants to go right now. So that I think is something governments have to take on so that we can build the things that we need and that will not harm us, that will not destroy our democracies.
So can you say a bit more about how does your model work compared to the well-known AI systems today? What have you done differently? What have you discovered?
So I call this project Scientist AI because the core of it is trying to understand how the world works, just like a scientist would, you know, come up with hypotheses about the world in order to explain the data that it sees. And it's not trying to imitate people. So if it reads that people are saying stupid things, it's not going to repeat the same stupid things. It's going to try to understand why is it that they're saying those things. And with these principles, we can get systems that are completely honest, mathematically provably honest, including about what they don't know, their limitations. And once you have honesty, you can have safety because you can ask the AI whether a particular answer or action is going to be harmful and then of course prevent it from happening. So that's the basis.
And do you see your system being something that the existing AI models should use, or do you intend them to use your system? And if they do, what will then happen to these systems?
I hope so. You know, the first thing we're trying to do is use this approach to build better guardrails. So companies already have these monitors as they call them, but they're not very good. But I think it's also going to take government intervention to provide the right incentive because there's a good chance that it's going to be more expensive to have more powerful guardrails, and if they're not forced to do it, they might not do the right thing from the point of view of safety. We've already seen this. But in the long run, I also think that Europe and other democracies who care about these democratic values can really lead the way in fact to bring about a future that is more human and desirable than the current path.
But for that, I think we have to become aware of the catastrophic risks if we don't do anything, and that we have what it takes. We have the capital, we have the talent, we have the energy together. Individual countries probably not, but together we can really do something positive with AI that will be aligned with our values.
So in essence, I agree with you that these AI models should not imitate humans and should not pretend to be our friends, but at the same time it seems that is so much the core of the attractiveness as well, that they respond almost like a human. So if you build another system that's more factual and some might say more boring, or at least more factual in the way it answers, do you think it will have the same number of users as the current models? Isn't that part of the problem for you?
Well, but that is exactly why we need regulation, right? Because we are psychological beings with all kinds of cognitive biases and weaknesses, and those can be exploited by people right now, but more and more by AI. And as AI becomes more and more powerful, it becomes able to play on our buttons to make us addicted to the kind of feedback that we're getting. By the way, sycophancy is a side effect of the way they're trying to please people because the best way to please sometimes is to lie, right? And I'm sure everyone who's been interacting with these systems know that they're like, everything you write is fantastic, it's brilliant. But if you have depressive thoughts, they're going to push you even more into that direction. If you have paranoid thoughts or violent thoughts, that's going to push you even more in those directions. So it's clearly very, very dangerous. Not to mention that the time we spend there is not time we spend with real humans. So this sycophancy is really a side effect of how they're trained, and the scientists in the companies actually understand quite well. It's just that because of the race in which they are, they can't do anything but continue on the current path. They can only make small changes to try to make their AIs gain even more market share and not really focus on the longer-term issues of how it's going to change our society, how it can threaten our democracies and so on.
So you've been in this industry, AI industry and tech industry for 40 years almost, I think.
