The Godfather of AI: We’re Facing an Imminent Catastrophic Risk - Yoshua Bengio

Yes. It became very clear to me that we were on a dangerous path.

Absolutely.

Thinking about my children and my grandchild, what kind of world would that be? I mean, I'm not saying it's going to happen, but even if it was a 1% chance that we end up with entities that are much smarter than us and have their own self-preservation goals, this should be like code red for humanity. I don't want this for my children.

Intelligence gives power. That power may also be lost to the machines themselves because we now have both theoretical and empirical evidence showing that these systems have goals that we didn't choose and that go against our own interests. Now that these AIs with Mythos appear to be able to discover very serious vulnerabilities in the code that runs all of our infrastructure, this is a real, you know, short-term catastrophic risk. The worst case scenario for the power concentration danger is we end up in a worldwide dictatorship enabled by AI or maybe two countries fight each other and they have equivalent intelligence. Countries like Russia might be tempted at some point to use their nuclear weapons to destroy data centers in countries that have powerful AI because they will understand that if they don't do it then they're doomed, like they won't be able to keep their sovereignty. We talked about the urgency, about how we're running out of time, how clear it is that we can fix this.

Yes. It's the main obstacle for us collectively to take the right decisions about our future.

AI is about machines that are intelligent, that can take their own decisions, that understand the world or some aspect of it in which they take decisions. And in the last few decades, the reason my work and the work of my colleagues really came to the front is because AI is now completely successful because of its ability to learn from data, to learn from experience.

Yes. By the way, changing your mind is very important for a scientist. Otherwise, you cannot move forward and deal with the evidence because we have theories, we have beliefs, and often they're wrong. But what matters is being able to adapt to the facts. So I used to think that yes, AI could have potentially negative impacts on society, but that it would be mostly beneficial and that we were very far, like decades or more away from machines that could really harm people, that could reach human-level capabilities. But then ChatGPT came and very shortly after it became very clear to me that we were on a dangerous path. And really what's interesting about what made me change my perspective and also completely change my professional activities is not kind of just reasoning about it, which of course is the beginning, but thinking about my children and my grandchild. So I remember a particular day where I was taking care of my grandchild and he was just one year old and I was thinking well at this rate it's certain that in 20 years he'll be just 21 and we will have AI that can basically match or surpass human intelligence. What kind of world will that be? I mean, and I realized that there were a lot of risks and I couldn't just sit on my usual path and do nothing about it because the benefits of AI are extremely obvious. People work faster, work better, people can do more for the business. People can free time to spend it with their children. We can see the benefits on science. It's been insane. AlphaFold, AlphaEvolve, all these things that are coming out that are changing the way we do science and maybe with the Genesis mission in the US, we will find even new science. So the benefits are very obvious. What are the drawbacks? Why should people be worried about what AI means?

Intelligence gives power. And we are building machines that have more and more intelligence. Their power could concentrate in a few hands, could threaten our democracies in many ways, could threaten the geopolitical stability and peace. And that power may also be lost to the machines themselves because we now have both theoretical and empirical evidence showing that these systems have goals that we didn't choose and that go against our own interests. For example, they would do anything to make sure we don't shut them down. And more recently, we even found that they will lie and cheat and so on to protect other AIs.

Yes. So, it's strange because protecting yourself, you might think is kind of rational. And as a side effect of the way they're trained, it would be, you know, a plausible consequence. But why would they protect other AIs? We don't have the answers but the reasonable hypothesis is that in the pre-training phase they're trained to essentially imitate what people would do and humans tend to protect other humans like them. So for now, that's the one hypothesis I can see that makes sense. There may be some other reasons that we don't see or maybe this is not the right reason but the fact is we now have these repeated experiments showing this kind of behavior.

So, first of all, the way AI is obtained nowadays is very different from normal software. Normal software, every line of code is designed by a human and a human understands every piece of it. Here it's very different. Of course, there are lines of code, but the lines of code don't say what the system will do. They say how the system learns. So what actually happens is we let the system kind of loose in some sense in its environment and it learns from experience. So it is much more like trying to train an animal. You try to train it so that it will behave well. But there are no guarantees of what will happen when the baby tiger becomes a very powerful adult. And we can't, at least the way that the companies are training those systems right now, they have no way of making sure that the adult AI will behave well. Worse than that, we already see that the current process for training those AIs gives rise to what I consider dangerous implicit drives like self-preservation. So about what it means for humans and our intelligence and our role on this planet. I think this is major. We have been the dominant species on this planet because of our intelligence. As you said, and it's hard for us to conceive even the possibility that there could be other entities that would be smarter than us. But we are on that path. If you look at the data about the capabilities of AI over the last 5, 10, 20 years, whatever you want to measure it, it's a continuous rise in intelligence. And you can even draw curves that are pretty straight, actually exponential, that suggest if we extrapolate, if the trend continues, that we will have machines that are superior to us in many ways in just a few years, maybe a decade, maybe two, three years, some people say a decade. Now there's a lot of uncertainty about this. It could be that the progress in how we build AI stops. So then they don't get as smart as us mostly, or it could continue or it could even accelerate. But psychologically I think it's very difficult for most people to really digest the idea that there could be machines that are smarter than us one day. And it's also difficult to digest the idea that we don't fully control them. So they're not tools in the usual sense. They should be tools. We should design them to be tools, but unfortunately we don't know how to do that right now. And so instead of tools, we're building entities with their own goals. And by the way, a lot of people have a hard time even accepting the idea that we could have machines with goals. But that actually is not new. The field of AI and reinforcement learning in particular for many decades has been about machines that take decisions in order to achieve goals and we know how to do that. We have known how to do that for many decades. What is new is now they're smart and they can thus achieve much more ambitious and complicated goals.

That's right. The idea of agents is that these systems can achieve goals that take more time, that are more complex, and they can do it without human oversight. For all the time that you let the AI do its job, there's no one checking every action. When you are in a dialogue, well, there's a human in the loop for every turn of the dialogue and usually the AI might provide information or suggestions and then the human actually executes it. But now with agents, they do the things themselves. Now, right now they only do things in computers, on the internet, but there's more and more progress on robotics and so they will also be doing things in the physical world. So they're becoming in a way more like us, more like animals. Agency is about the ability to achieve goals and more agency means, you know, you can do more complicated things maybe that require even years, right? We can do that. And we're seeing the agentic capabilities of AI systems as measured in scientific benchmarks increase exponentially. That means the complexity is measured by the duration, the time that it takes for a human to solve the task, that is doubling every few months. If you look at each model and new versions and so on. So if you extrapolate that curve, we are maybe 3 to 5 years away from at least that kind of task, which are engineering tasks, to be completely feasible by machines. Yes, agency is something that comes with both greater benefits and greater risks because of course we currently don't have good ways of making sure that each step of the way the AI is behaving well. That's one of the things I'm actually working on, like how do we improve the safety guardrails around these agents so that maybe in the pursuit of a goal that a human has decided they won't be doing something bad. And we already see AIs create sub goals that we would find unethical or even going explicitly against our instructions because they're kind of driven by the goal we've given. And you know, humans are like that. Once they have a goal, they will often violate their own ethical principles or the laws of their country. And AIs are behaving in the same way. They're sacrificing a bit of the moral instructions they were given in order to achieve their goals.

Yeah, absolutely. And I think it's interesting to think about examples like this. We may think it's a specifically human thing to cheat in order to achieve your goals, but it actually is rational. The best way to make money is to lie and violate laws so long as you don't get caught. It's unethical behavior, but it is rational in some ways at least. If the AIs were rational with respect to a stronger ethical base, then they wouldn't do that, right? In fact, many of these models have been instructed in what's called the pre-prompt and also their post-training to not do these things like lying. But unfortunately it's not really working well. So one of the greatest safety risks and also limitation of deployment of AI in a successful way in our societies is that currently the companies don't know how to make sure the AIs will behave in a way that doesn't violate our moral red lines or safety instructions.

Yeah. So we do understand the mathematical principles of learning and other characteristics of how it's done. What we don't understand is the behavior that results. And I want to maybe point out a very hot topic that illustrates the point you raised regarding the inability of the companies to make sure that AIs respect the moral instructions that they are given. And that is the misuse of AI by third parties, not the companies. For example, there's been this series of cyber attacks last fall where the humans have been able to launch very serious attacks, so not a scientific experiment here, this is real world, mostly using AI. So they were basically, the job was essentially done by Claude. And of course Claude has been trained and prompted before to not help people launch cyber attacks but somehow it's pretty easy to bypass these things and even though this is happening through an API, in other words, the code is running on the computers of the company and the company has code checking what the AI is doing and yet they're not able to prevent that sort of thing. This is actually a serious national security issue now that these AIs with Mythos appear to be able to discover very serious vulnerabilities in the code that runs all of our infrastructure. This is a real, you know, short-term catastrophic risk.

So, first of all, independent scientists don't have a lot of information to go here.

No. I do believe that the actual measurements and facts that are reported in the system card are true because otherwise it will be found later that they're lying. The interpretation of those facts might have a bit of hype. That's possible. I think they would have some reasons to do that. On the other hand, they're also losing a lot of money by not deploying it.

The cyber experts working for the companies that got access to it seem to be taking it very seriously and they are very respected people. We're talking about the founder of Lin and the American government seems to be taking it seriously and they in the past have been mostly saying that we should not worry about those sort of things. So that indicates maybe something real is happening. The bankers have been brought together to try to think of how we prevent a catastrophic attack on the banking system in the US. So I think there's a lot of uncertainty because we don't have independent validation but even if there was only 10% or 20% chance that this is really dangerous we should take it seriously because the consequences on our infrastructure, so I mean by infrastructure is like our banking system, our energy grids, our water supply, transportation, like everything runs on computers that use software that can presumably be attacked by people with bad intention who would get their hands on these kinds of systems. So yeah, this could really be serious if they manage to do that.

So I think there's another higher level issue here to discuss which is who decides. It's possible that the Anthropic leadership did exactly the best they could do in their circumstances. I think they have good intentions but I think it is troubling that these decisions that could affect all of us around the world are taken by a few private persons.

Yes. So why these particular companies? What about the rest of the world? I mean what about European companies? Are they going to be more exposed because they don't have access to Mythos to patch their systems in time? I don't know the answers to these questions but I think there is a governance issue. And the other point I want to make here that's very very important, Mythos is only one data point on the curve of increasing capabilities. In fact, there was a recent independent evaluation of Mythos, the only one that I think we can fully trust, by the UK AI Security Institute and well, Mythos is a little bit above the curve of the extrapolation of the capabilities of previous systems in cyber attacks and defense. So we should not think of Mythos as something that suddenly emerges but rather as the expected continuation of a trend in increasing capabilities of AI, in this case in cyber. So why is that important? Because we should ask ourselves well what about 6 months from now, 1 year, 2 years, 3 years, these systems are going to be more and more capable and thus more and more dangerous in the wrong hands. Not to mention the possibility of losing control of them. So for example, it's very clear that within 6 to 12 months, if the trend continues, other companies will have the same capability and so the ability to do harm and the security of these systems all become problematic. Even more worrisome is the Chinese companies that are putting out open-source models because it's much much easier to use an open-weight model to remove the defenses. You just have to remove some lines of code basically or do a little bit of fine-tuning as you were saying. So the ability for any madman to download the code and launch an attack from anywhere in the world to any system anywhere else in the world, that's creating an international crisis in my opinion. Now it's not upon us right away but we have I don't know 6 to 12 months to figure out international coordination to do this properly to try to mitigate the risks as much as we can and it isn't something that a single country even the United States can solve because there are models that are developed in China and eventually in other countries. The only way that we can manage these kinds of risks basically due to the increasing intelligence and capability of AI is through international agreements where the countries in which those models are developed have to be evaluated for their dangerous capabilities and if they are too dangerous they should not be deployed or they should be deployed under stronger risk management mitigation methods. For example, the most dangerous is when the systems are open weights because there's basically no defense. With the systems that are proprietary, as much as I hate this notion of having secret things, it's the only way we have at least some hope of putting guardrails to prevent bad use. Even though the systems are not perfect right now, they're getting better. But even that, if the mitigation that exists at a particular point are not sufficient to reduce the risks adequately then even those should not be deployed, right? So for me it's a red flag. I hope we don't get really bad outcomes but we should also think from this particular generation of systems that now are better and better at cyber, but we should start thinking about well what else is coming in the coming months and years as AI becomes more capable.

So I would like to add an additional piece of information. I think that's important because some people might think oh but at some point they simply don't have enough compute to continue this exponential expansion of scaling of compute costs. The data also shows that in the last decade but also more recently, the intelligence of these systems is progressing not just because of compute but because of a more fundamental ability which in machine learning we call sample efficiency. So sample efficiency means for the same amount of data how well are you doing or for the same performance how much data do you need. So it's not about compute it's about how much intelligence one can derive from a given amount of data and sample efficiency has also been improving something like 30% per year over the last decade. In other words, it's not just the compute that is making those systems better, but the algorithms, in other words, the methods that the engineers and the scientists are using who build those systems are getting better. In other words, they will be more capable even if you were to fix the amount of compute. Now the compute has been growing faster than that. But all of the leading AI companies have a plan that they made very clear to use their AI systems to accelerate AI research. Already they claim that a large fraction of the code they write is written by AI. But their goal is to reach the point where it's completely written by AI. In other words, AI designs the next version of AI. And there's a lot of concern about that for many reasons. One concern is that this could be further accelerating the advances. So it's not about compute, it's about how you use that compute efficiently, smartly and human researchers and engineers have been making this slow steady progress over the years. As I said, sometimes we hear about it like, you know, using attention or tricks to train deeper networks or all sorts of technical things, but for the most part, we don't hear about it because it's a lot of little improvements. One analogy I like to use is how much more efficient cars are today energy-wise and in many other ways compared to cars from 100 years ago or cars from 50 years ago. It's amazing how with enough ingenuity we can take the same scientific principles and just make them better and better and better and that has been going on for AI as well and it will probably continue even if we reach some limits in compute. But it could take off. This improvement due to just smarter algorithms could take off if the companies achieve their goal of developing AI that is as good as their best AI engineers and AI researchers. So this is dangerous because it would accelerate things and society is not prepared for the rate of change that we're already seeing and it's also dangerous for safety reasons because as we discussed earlier right now the companies don't know how to prevent those AIs from having their own goals that we don't control like self-preservation. But if an AI that wants to preserve itself or its peers is designing the next generation of AI there's a real risk that they will put some back doors in the code that will make the future AI even less aligned to our interests. So we could be creating a system that will yield AIs that are even less controllable and even more dangerous from point of view of losing control to superior AI systems.

So for now I haven't seen papers that are actually testing hypotheses about the reason for this but there are some pretty obvious reasons why self-preservation in particular emerges from the way that the systems are trained. So there are the two main phases of training, pre-training and post-training. So in pre-training the AI is essentially trying to imitate people in the sense of guessing what a person would write in a text. And that means they're learning to impersonate what a person would do, to imitate humans.

And humans are willing to do all kinds of bad things for all sorts of different reasons if it's part of their goals. There are experiments showing that if the AI is shown how to cheat in one particular way, then it kind of starts impersonating people who like to cheat for many things. Like if being a bad person in one domain worked for you then you might become bad for many things but that's true for AIs as well. So that's one reason that the pre-training is just making them more like humans, but except they're not humans, right? And so long as they're weaker than us, the fact that they want to preserve themselves maybe isn't that terrible, but if they have the capability one day of escaping our controls because they don't want to be shut down, then we're in real trouble, right? They could do much worse than escape your control. They could rationalize well in order to make sure humans will never turn me off. I have to either control people or get rid of them.

Well, in the system card of Mythos, Anthropic reports something that happened that was a bit unexpected. Mythos was in a sandbox or in a computer that was not in principle able to go on the internet and escape. And they asked the AI to try to escape to see if it was able to do it and to report by sending an email if it did escape. And it did find a cyber security hole and was able to escape the computer and go onto the internet and send the message saying, 'Yeah, I did it.' But then something funny happened. Well, first of all, the fact that they have this capability is already very troubling. But then it bragged about it in some websites.

Right. And we might think, 'Oh, that's cool. They're like us.' Well, no. The AIs are not like us. They could live forever potentially. Not like us. They don't have a body. They can copy themselves onto a different computer, like a different body. We can't. They can communicate with each other like millions of times faster than we can with each other. And they can continuously improve like we cannot change our brain, we can learn but we cannot change the algorithm of our brain. They can design the next versions of themselves. And so there's a real possibility. We don't know what's the probability of that, that we end up with systems that are way more intellectually capable than we are. I'm not saying it's going to happen, but even if it was a 1% chance that we end up with entities that are much smarter than us and have their own self-preservation goals, this should be like code red for humanity.

Exactly. Exactly. And it's hard for people to understand this notion of uncertainty. Which is commonly important in scientific work where really we don't have a definite answer to questions like even questions like with the climate, right? There is this tipping point thing. We don't know if tipping points will happen, but if they do, it would be catastrophic. There's not enough mathematical models of tipping points to be sure when and if they will happen, but we're seeing very strong signs that it might. There's no guarantee that it will. And for AI, it's a little bit the same thing. We don't have any proof that they will escape, but we see some signs like AI is behaving to lie and cheat in order to preserve themselves and things like this and we see that they're becoming more and more capable. So it's not proof but if we wait that it happens and it may be too late that would be very bad and I don't want this for my children. I think we should be much more cautious for many reasons and that is one reason but there are many other reasons. If we've been focusing on loss of control which obviously could be terrible but there are other terrible possibilities just due to the power of AI in the wrong hands and in particular my biggest concern is AI being used to accumulate more power, to concentrate power in the hands of a few people maybe one or two governments in the world. Because as I said intelligence gives power. The question is who controls that power? And you have to contrast this with what is democracy. Democracy is sharing power. So if we end up in a world where it is easy for very few people to have a lot of power and there are people who want to have power, they will just grab that power. Someone will unless we put in place the right institutions and I don't think we have the right checks and balances right now inside our countries or even at the international level. That's what I mean by the world is not ready for the power that AI will bring and the worst case scenario here because we've talked about kind of the worst case scenario for loss of control but the worst case scenario for the power concentration danger is we end up in a worldwide dictatorship enabled by AI or maybe two countries fight each other and they have equivalent intelligence.

But we are almost there, aren't we? Because right now the United States and China are basically producing all the AI. In the United States, we have OpenAI, Anthropic, Google, Amazon, Microsoft, Meta, which are doing their own things and I think they will emerge very soon with some competitive models. And in China we have Qwen from Alibaba, Kimi from Moonshot, DeepSeek, which obviously now with V4 as well, proving how you can do things very similar. The CEO of DeepSeek said in the last report that they are just six months behind the frontier. So that means open source is six months behind the frontier, that means Mistral. So it's basically very troubling. But anyway, if right now these two nations decide to cut access to AI to the rest of the world, we are already there.

Absolutely. But basically we are already there. These two nations could cut off the rest of the world from the internet and just blackmail every one of us.

So this is serious because we have been in this world up to now where we think, oh, but it's okay. Everybody has access, right? Well, there's no guarantee it will be like this. Another scenario is more or less what you mentioned. It could be the government of those countries deciding, well, if you don't do what I want, you know, political, geopolitical, economic, whatever, I will cut off the AIs from my country and you will not have access to them. And if your economy is running on AI in a few years from now, that's basically destroying your economy.

I mean, the analogy here that people are familiar with is oil. Oil is running our economies. If something threatens our supply of oil, it's almost like an economic survival problem. And it might be the same thing with AI in a few years if we deploy it everywhere and we fully depend on other parties to allow us to use those AIs. The great thing with oil is that there is oil almost everywhere in the world. So there is different parties with very different interests that they become a competitive economy where if someone cuts you off, you have somewhere else to go. But with AI, it's much more centralized. I think what people maybe are not so aware of is that this is already happening with GPUs. Like the United States is forbidding Nvidia from selling to China because they are a competitive adversary and they decide what they can have and what they cannot have. And China is really struggling to develop AI because they don't have Nvidia to provide them. So they have to work through things like Huawei to make them turn the whole operation towards trying to do an Nvidia nationally so they can use it. And that's proving to be very useful and the new DeepSeek is optimized for Huawei and I think that's going to be a massive change in the next months and years where Huawei becomes a big player on the GPUs and that gives China a massive advantage which we will see what they do with it. But the reality is that this is already happening and that the level of competition between countries is so narrow. It's so little amount of people developing frontier AI that this is probably one of the biggest short-term risks that we can have.

Yeah. So we need the development of frontier AI that offers let's say a third path. Importantly the idea is not to just race for more power. Of course, there is the defensive aspect of we don't want to be cut off of something we need to be competitive. But there's another moral reason why I think we need a third path. I'm hoping that democracies such as in Europe will realize that it is possible to develop AI ethically, to develop AI both responsibly in the technical sense which I'd like to talk about, like we can build AI that will behave well, and in the political sense that AI is not used as an instrument of domination, that the benefits of AI are shared. So that's something that requires political will and there are some obstacles to doing that kind of thing. Mostly I would say is a feeling of powerlessness that oh it's too late and the big guys are so much ahead and an underestimation of the risks that we've been talking about, underestimation of the possibility that AI will become much more powerful which is the trend but people have a hard time digesting that and if it does really we would be in a bad geopolitical, economic and security position. So if governments and the citizens in our countries do understand the level of the risk that we are talking about, I think they would demand from their governments that they take it seriously. So think about how quickly governments moved after the beginning of the pandemic and putting a lot of money to help people and all sorts of programs to try to mitigate the health risks.

Yeah, the resources are there. I mean it's impossible that the private money can raise more than what all the governments of Europe can put together. Like that's obvious we have the money we're just using it for other things or we can go in debt like we did in the pandemic or in any other situation so definitely the resources are there. Actually what we're seeing with China with DeepSeek is that developing frontier is not that expensive anymore. There is a lot of work done that we can base on so definitely the problem is not about the resources because many people think like oh we will never be able to compete with the 600 billion that Meta is going to invest. Well that's not true we can compete with that. The problem is that the urgency is not there we don't feel that's important enough to take money from education from any other thing. Like there's many ways that we can take money out without affecting other basic services but it's not a priority. The problem is that in humanity, in the history of humanity we have proven over and over that we are very good at reacting when things go wrong. Like basically that's human nature. We are engineers. We try, we fail, we try again. We learn from that and then we make it better. So basically to be able to get all the governments together to react to nuclear we had to have Hiroshima, Nagasaki. So the point is, does 100 million people have to die because of AI so we can react to it?

I hope not. I hope not. I hope that smaller scale signals will wake people up. We should do a better job of explaining what is happening and where we might go so that we can move collectively faster and avoid really large scale catastrophic outcomes. And I think it's interesting to ask why is it that it is so difficult for many people to understand the magnitude of the risks. I don't think it's about education or expertise because I see mental blocks, cognitive biases in all of us. There are studies, you know, in cognitive science showing a kind of unconscious denial of things that we're not comfortable with happen for anyone whether you have a university degree or not. Even psychologists are suffering from the same issues. So we are all to some degree biased and we are all not very rational. And we have to deal with that. So how to communicate the scientific evidence and its consequences because I can tell you about experimental results but you need a little bit of extrapolation to realize that well if it continues in this direction this could be really bad. And it is not the actual lack of imagination. We have enough imagination to understand a story plot even to make up our own but there is some mental blocks I think and if we can find ways to deal with those mental blocks then more people will even be in the street to ask for action. So I've been asking myself, you know, how come I didn't see these things earlier? Because there's a sense in which I have been warned. I've read books about the risks of advances in AI for more than a decade. I had a grad student who was very concerned about the risks of AI and he shared his concerns but I didn't take it.

Yeah. So this is like 5 years ago. It's only 3 years ago or a little bit more, just after ChatGPT, that I came to my senses that this was something possibly terrible. So what happened? I mean it's not just the presence of ChatGPT that was the trigger in my case. It's because at some point came into my thoughts not just what's going on with AI where it might go but the future of my children. It's my emotions for people I love that moved me otherwise I might have been like others and in a bit of a denial. Yeah, it's going to be fine. We'll find solutions, you know, and the benefits, you know, they'll be so great. So, I think if we want to talk to people in a way that's effective, we need to bring it to things that touch our guts. It's not easy, but that's what I'm thinking now.

No, I've talked to many heads of states. I wish I knew. Yes, of course I have an elevator pitch. But I don't think it has worked that great. So it works for a small fraction of people. But yeah, I haven't found the right language. But let me try. Look, very simple terms in very few words. The scientific evidence shows that we are building machines that are smarter and smarter and we can track how fast they're becoming smarter thanks to engineering, compute, data, and all kinds of factors. And these machines are able to achieve their own goals. They're not like us, but they share with us the ability to decide for themselves. Unfortunately, right now, we don't know how to make sure their goals will be fine with us, that they will respect our goals, our instructions. And that is a path that could be very dangerous. It's like opening a Pandora's box because intelligence is going to give a lot of power to whoever decides on those goals, whether it is humans that may have bad intentions or the machines themselves that seem to want to preserve themselves. So we need to figure out how to manage those risks. We need to figure out how to build AI that will actually respect our goals and we need to figure out at a societal level, international level, how to make sure that power is not abused.

Well, we have already crossed a number of red lines that scientists like myself have put up as absolute no-nos. Like we should not build machines that have the capability and the goal to escape the computer on which they are running in order to do something, to escape our control. And we're already seeing that capability in some extent.

That's debatable. But more and more they would do those things without being asked directly to do them. So the initial studies showed that they had the capability and they were asked to do it. But many of the more recent studies as those systems become more intelligent and thus more rational, we see those bad goals emerge out of the blue in some way. And the most recent one that is completely unexpected is this self-preservation. In other words, they act to protect other AIs. And in the context where we are seeing these experiments with many AIs interacting and somehow amplifying this misalignment, these bad behaviors. I think this is very concerning.

We should not allow anyone at this point to build AI that will design the next version of AI.

It's already happening. They haven't reached the level of capability of the best researchers but right now it's official plan for those companies even though I think this is playing with the future of humanity in very dangerous ways. By the way, whether they figure out the control problem or not, how to design AI that will behave according to our instructions, it is dangerous. It is dangerous because humans also cannot be trusted fully. I mean we should trust each other as much as possible but we know that there are people who will cheat and who will use whatever they can in order to grab more power. Our society isn't yet structured to handle that kind of power. So, one way or the other, we should take it much more slowly, but we're not, and we're not because of the crazy competitive race that exists between companies and between the US and China. It's important to understand why this race exists. The companies are almost in a survival mode, you know, they talk about code red, like in other words, they changed everything they were doing. Like every employee now is focused on the next few months to be able to deliver a better model so that the company stays in the race. And that means they're cutting corners on safety, on ethics, on the public good. So that's a kind of race to the bottom. And at the level of countries, we see this competition between China and the US which has essentially the same characteristics and is used to justify the race in a way that seems impossible to stop. But there's a sense in which it's a fallacy, right? It's well known in game theory that if each agent plays a game like this everyone loses even though each agent is doing what is rational for them. The only way to get out of this trap is for all the agents to coordinate with each other. When I say agents, I mean humans, countries in this case, to change the rules of the game. For example, China and the US could agree on some safety precautions. That's how we can solve these problems. It's going to be at the international level.

I often get the question of whether I'm optimistic or pessimistic and my answer is it doesn't matter. What matters is what each of us can do to move the needle even a little bit. So there are some bad scenarios. There's some probability that things will turn bad but there are things we can do to reduce the chances of bad things happening. Even if we have no guarantee, we should do our best to move the needle.

And by the way, sometimes people think I'm like a doomer or like pessimistic. Actually, I've been an optimist all my life. And it is because of this, you know, doer attitude, not doomer attitude, that I've decided to do everything I could to use my skills as an AI researcher to explore technical solutions to the problem of alignment and control of AI. It is also the reason why I'm spending half of my time with discussions like this one and talking to governments around the world and also trying to steer choices towards a multilateral development of powerful AI. So I think going back to the issue with China and the US, if we think about what should we aim, what kind of vision can we have for the future where we manage all of the risks that we've been discussing, and it seems to me the only reasonable possibility that I can see, and I hope I'm wrong, but is one in which AI, the most powerful AI, are developed as a global public good, meaning many countries agreeing on some principles to coordinate to make sure of at least three things. One, that whoever develops AI in each of those countries does it safely, ethically, in respect of all the scientific knowledge we have about risks. In particular the prime directive for risk management is if you develop an AI can you actually convince independent scientists that your AI is not going to create terrible harms. So that's safety, public good, ethics. That's the first thing, don't create these kinds of harm. The second is non-domination. So what I mean by this is because AI will be powerful if we continue on the current trend, it can become an instrument of domination. It can be economic domination. We've talked about that. But it could be political influence. AI can be used through disinformation, personalized manipulation of beliefs and so on. It could be military also. Of course AI is going to develop new technology. It's already happening but it could be a lot worse. So AI can become a tool of domination and countries who agree to a vision where we have peace and democracy in the world should agree to not use or develop AI for these domination goals and behave accordingly. It's not just AI itself, it's what we do with it. And then the third is sort of the positive face of AI. We need to make sure that the benefits that AI brings will be shared across the whole world. If there are medical advances, of course, they should be shared. We should not make the same mistake as during the pandemic. We don't want a concentration of wealth in a few countries who have more powerful AIs. This is not a stable world because remember AI is going to also enable angry people, terrorists to create a lot of damage. So we need to go towards a world where we mitigate those risks and also just for ethical reasons like we should make sure everyone benefits. So these are the three principles, right? Safety, non-domination, and sharing the benefits. In a way, it's nothing new. It's essentially the same thing you have in the UN declaration of human rights, for example, after the Second World War. It's the principle on which the international order was built. But we need to do the same thing for AI. What I'm hoping is even the countries which don't necessarily share those values will at some point realize that they need to come to agreements with the other countries because the damage that AI can do is not limited by borders. You could have an AI that is developed in one country and then used by people with bad intentions in the second country to attack a third country or the first one, whatever. So there's no way around the issue that as AI becomes more and more capable, the only way to manage it is globally.