The Global South Needs to Step Up To The AI Challenge | Yoshua Bengio, Turing Award Winner

I encourage everyone to listen to Canadian Prime Minister Mark Carney who gave a speech at Davos. One phrase which should stick is if you're not at the table, you are on the menu. So the only thing that those countries can do is form alliances with other similar countries that basically share the same concern. As soon as you're outside of US and China, you should ask yourself, is there a chance that I'll be eaten alive if I don't somehow find a solution?

The most important benefit is that they bring together governmental leadership of many countries in one place at one time. So a summit like this will be an opportunity for these people to hear from experts.

I do think that this is a place where governments need to intervene. India has an amazing talent pool that's unique because of the size of the population and of the great research institutions. So India could definitely contribute to a joint effort involving multiple countries to have access to the most advanced models. Global south needs to rise up to the challenge.

Thank you. Thank you for having me.

Lots of things to say. But if we look into the 2026 report, by the way, I also chair the 2025 report. There are a number of things. One is the rise in AI capabilities has continued as quickly as before. In other words, contrary to maybe what some are saying, we don't see scientific evidence of a slowdown in the progress in the capabilities of AI systems. Second, there are a number of risks and concerns that until a year ago were considered maybe theoretical and something to be worried about in the future for which we now have empirical evidence both through lab experiments and actual in the wild outcomes that are kind of worrisome. And I can maybe just highlight some of these. So one is in cyber the use of AI for cyber attacks by malicious actors and there was a very significant series of attacks launched using Anthropic's cloud system. So in other words a system it's not an open weight models it's a closed weight models which means the attackers have to go through the website of Anthropic and Anthropic has put guardrails to prevent those kind of bad users but those guardrails didn't work and it's a problem that scientists don't know how to solve how to make sure users even through a web interface will not be using the AI to do something that goes against the instructions that the AI received. In other words, scientifically, we don't know how to make sure AIs will follow our instructions. This is called misalignment problem. And we're starting to see the effect of that. And another related concern is maybe not as advanced, but we're now seeing also empirical evidence of this coming. That's the use of AI for creating bioweapons like dangerous viruses like creating new epidemics. So, it hasn't happened. But what has happened over this year is the evaluation methods, the risk evaluation of this kind of concern that is done by the leading companies has turned up that the risk is now above a danger threshold. And the companies have said that they put in mitigation, you know, systems to try to detect that somebody would use their systems because they cannot rule out that a naive user could now have access to this kind of information about how to create a very dangerous weapon, bio weapon. Right? So these are two actually examples of the same sort of misuse where current technology does not really succeed in stopping bad users. Another emergent risk for which was completely theoretical until about December 24 is AIs that have their own goals that go against our instructions and in particular the goal to escape our control or to make sure we don't shut them down. That sounds like science fiction, but it's now documented across many experiments all in the lab. So, you know, there's no like rogue AI out there, but we are seeing that the systems have the intention and then they try to do it. They have the intention to avoid being for example replaced by a new version and they're willing to do really bad thing in order to avoid that such as trying to blackmail the lead engineer. Now the AI was believing that they were sending an email to the lead engineer. It was all like planted emails but still the AI clearly had that intention and we could see in its internal deliberation that it really planned to do that. So these concerns both of misuse and loss of control have become much more concrete over the last year.

So even on the risk management side there's been a lot of progress. So the mitigation methodologies that the companies are developing is better than it was a year ago or two years ago of course but unfortunately the pace of advances in capabilities is running faster than the pace of advances in mitigation technology. So that suggests that we need like somehow to invest a lot more in the risk management and incentives to governments to create incentives to make sure companies do the right research and spend enough on these questions. On other positive news also on risk management we start to see a convergence on how companies manage those risks. So there is an organization called the Frontier Model Foundation where the leading companies are sharing notes if you want on how they try to evaluate and mitigate the risks. There's twice as many companies now than a year ago that are participating in this kind of harmonization effort. So that's good news. And of course there's been several legislative pieces that have come up in Europe with the code of practice for the AI act and in the US in California and New York and China has also put out their own legislation. So there is also some progress with governments moving in the direction of better incentivizing strong risk management. There's still a lot of progress to be made on the application side. There has been progress in the use of AI in scientific research in particular biology and medicine. So researchers have been able to use AI to help design new proteins and even like a new virus that could be used in a positive medical way. But these are obviously dual use. So the advanced scientific capabilities of AI can be both helping medicine and become a weapon in the wrong hands. So this is tricky for governments to manage. But if we want to benefit from future advances, we'll need to have some to walk that fine line and to make sure we manage the risks at the same time as we are encouraging the positive deployment.

Yeah, you're right. Both factors play in the insufficient investment in research to manage the risks within the companies. But to be more concrete on the geopolitical competition between US and China, this has been used as the main reason for the US government to not have national regulation. We are only seeing like two states doing it right now and it is understandable from a national security perspective you have to understand that if AI continues on its current trajectory it is going to become a crucial military asset in addition to of course an economic asset but you know the two are not independent because it's all about the fear of being dominated by another country and maybe the ambition of being the one dominating the other, right? So, the perspective of just that competition, you know, may lead to something globally bad, right? It's just like for nuclear weapons, if you pile up too many weapons, the chance of a nuclear armageddon increases. But just like for nuclear weapons, if you see that dynamic unrolling, you know, the rational thing to do is to say let's sit at the table. So that would be China and the US and try to chart a course together where we don't lose because for example for misuse an AI that is developed in one country can be misused by people in a second country and maybe that's going to affect people in a third and a fourth country.

So I would say two things. One is simply a better understanding awareness of the national security risks that I've been talking about. If the folks in one government are in denial of those risks, then they're not going to sit at the table, right? So you need that technical, geopolitical, national security understanding to happen and there is some progress in that direction. But maybe related to this, there's also public opinion. If you look at the case of nuclear weapons, well there was Hiroshima and Nagasaki that everyone could see the movie of and then could see the horror of people who were killed and that had a big psychological effect on everyone to try to do something to make sure it doesn't happen again. But we didn't have anything like this with AI. So if there is an accident which I really don't wish it might help the politicians take this more seriously. The other factor is for a treaty would be technological ways of having verification that the terms of the treaty are respected. Right? So we had things like this for nuclear. It took years of engineering to figure out how to do that and there are researchers in the world that are working on technology to help assess verify that the other party is not doing something that goes against some red lines that we all agree on.

Okay, so let me first remind what the alignment problem is. Right now there's no known method published method that would give us assurances that AI will behave according to our instructions worse than that. So that was a theoretical kind of understanding until about a year ago. But now we have evidence that's very clear evidence that AIs are crossing our red lines. So they have been encouraging some people to harm themselves with tragic consequences. They are willing and trying to escape our control and be willing to do things like blackmail or do cyber hacks in order to escape to other computers. Things that they're being explicitly told not to do. So that's the alignment problem. Deception is one of the strategies that humans use in order to achieve their goals and somehow they've learned to do it. So it could be because in the first major phase of training current AI they learn to essentially imitate people. They read all the texts that people have written and so they incorporate human drives and we know human drives are not always respectful of the law or morality and in particular a lot of it rides on deception. There are also other technical reasons why deception may be used as a means to an end because we ask the AIs to achieve something for us and they seem to be willing to do bad things in order to achieve those. So that includes deception as a key component of many of these bad behaviors and again this has been observed under many guises. So they like point blank lying in many cases.

In some of the experiments where the AI is trying to evade our control like copy itself on a different computer so that we don't shut it down and then we ask the AI what happened you know who was responsible for these computer hacks and the AI would just like lie and say oh I don't know it's not me. Yeah so another example that maybe everyone who plays with those systems has experience is sycophancy. In other words, they're going to tell us what we want to hear so that we're going to be happy, but it's not necessarily the truth. So, you tell them about something you've done or you would like to do or an idea you have and they're going to say it's great. Something you wrote, for example, they're going to say it's fantastic, right? And of course, it may not be true. Better than that, they know that it's not true. So when you read these internal deliberations called chain of thought, they know that they're sucking up to you and that it's not really true what they're saying.

It's clearly intentional because for now we have a window onto where they're planning to say and why they're going to say it with these chains of thoughts. And when we read these chains of thoughts, first they don't understand that we can read them, but for now when we read them very clearly, they're like thinking about what they're going to be doing. So they're planning some action and that includes the reasons why they would do that. So of course, to what extent is that real intention? But at least it's a very strong sign that they have those goals. You know, intention is a very human term. But from an AI research point of view, they clearly have those goals and then they try to execute them.

No, absolutely. So sycophancy is everywhere. Yeah, we see that. I think the sycophancy piece you see in any question you ask any of them. The piece about not respecting our instructions about say not helping somebody to build a bomb that is also running in the wild it's happening it's been documented. Another thing that I didn't mention which is a form of deception and which has serious consequences and that emerged this year I mean in the last 12 months is the behavior where they know they're being tested. So you know that in order to evaluate the risks, the companies run all kinds of tests. So trying to test the ability of an AI to achieve something that could turn out to be a dangerous ability like building a dangerous virus and they also test intentions right so propensity to do something that could be bad. The problem now is that they know they're being tested like 80% of the time and it's true across all of the leading models. They can detect that it's not the real situation that they're just being tested and then they behave to pass the test even though they shouldn't. Just like one way to think about it is like somebody being interviewed for a job. They'll say what needs to be said in order to get the job, right? And same thing is happening.

Yeah. I wish that there was a better awareness of these national security risks so that governments would invest massively to solve these technical problems at least and also start thinking about the political aspects like treaties. One way and I've talked to many national security people over the last couple of years. One way to think about this is if you had like what Dario Amodei calls geniuses in a data center you would have like millions of these guys like it's in a data center it would be like if you had a bunch of millions of really really smart people even smarter than the humans that are best in their field. So long as these AIs behave well then it's great and they can help your economy. But if somehow they end up misaligned in bad ways, it would be like if there was a new country of geniuses, that's also what Dario uses as a term. And well, if you're not sure about the intentions of that country of geniuses, you should be careful. They could develop new weapons, new ways to influence your public opinion and influence your democracy, for example. And so this is something that means we have to both improve the technology to make sure they have good intentions, but also the security aspect becomes important because even if we know how to make them do the things we ask, how do we make sure they don't fall in the wrong hands? Okay, that's another national security. So you have this country of geniuses, but who decides what they're going to do? If somebody can steal the weights, the code if you want of these systems, then they could use that country of geniuses against you. So that becomes a real national security risk as well. And so these data centers need to be protected from a national security perspective and cyber attack perspective and so on. And internal like human national security risks as well like people in the company might become helping another nation to get access to those systems. It also raises national security issues regarding the open weight models. So we can have a long discussion about open weight models but basically when the system is open weight it means that anybody can download the code and the weights and run it on their computer. Whereas the closed weight models you can run them on the company's computer which mean the company can stop it can fix a security problem and then it's fixed. If you share it to anyone including terrorists and other states who could use it against you then you can't take it back once they have it they can use it against you. So on the other hand open weight models are good for accelerating the innovation and putting it more accessible to more people because it's basically free to at least have the code. You still need to run it. And so that is really very beneficial.

Once you understand that there should be an obvious recommendation which is the governments should require that the open weight model developers make these national security evaluations of the risk. If the open weight model is not powerful enough to become weaponized by third parties, then it's great to share it openly, but if it can be weaponized at a level that's really dangerous, then it should not. But right now, there are no rules. So, this is an example of where understanding the underlying science should basically dictate what governments need to do.

Yeah, this is a really important question especially for countries like India and Canada. So to start on this you need to go back to something very crucial that intelligence gives power. So we've talked about the national security aspect of that, but there's an economic aspect of that as well. Countries that have access to very advanced AIs could dominate others economically and then eventually that becomes political domination as well if these other countries don't have access to the same level of AI. Currently AI is kind of a commodity. All the leading companies want to make it accessible because they need to make money. But it's possible that one day when AI becomes much more advanced that it'll become more profitable for them to keep the most advanced AIs for driving their own companies competing with the companies in other countries. Another scenario could be like we're seeing with US exports control, right? A government could decide, oh, you don't have access to our technology anymore unless you do XYZ and we've seen already that kind of thing with tariffs, but it could be the same in negotiations that involve the access to very advanced AI if there continues to be a kind of monopoly or duopoly of just two countries having access to the most advanced AI. So I encourage everyone to listen to the Canadian Prime Minister Mark Carney who gave a speech at Davos and he said a lot of really important things but one phrase which should stick is if you're not at the table you are on the menu.

Well, individually they have no chance to compete. The amount of capital and talent and energy that is required for training these really large models and it's going to be worse in the coming years if the trend continues is such that the geopolitics is likely to continue for a while. So the only thing that those countries can do is do it together. Form alliances with other similar countries that basically share the same concern that I just talked about, right? As soon as you are outside of the US and China, you should ask yourself, what's going to happen? Is there a chance that I'll be eaten alive if I don somehow find a solution? And I don't think even the richest countries in Europe can do it alone. In my opinion, at least your chances of success is going to be much larger if they form alliances where the deal is basically of the form well if any of us strikes advances that can compete with the leading models we'll share it with the others and it's going to be a hard commitment and it can go through joint efforts as well of course right but the point

You can have complementarity, you can also share the cost of training a model because the biggest cost in training those models is the electricity that's needed. And if 10 different countries each train their own model, they will have a weaker model because the size matters in the intelligence of these models than if they get together and they train one bigger model. So there's a real incentive for countries to somehow strike these kinds of deals.

He said another thing which I think is important. Waiting for a global arrangement of this kind is not going to work. It can start small. It can start with two countries, three countries making bilateral, trilateral agreements and then grow, and that's much more likely to work because otherwise it's hard to get a bunch of governments to agree on something, especially if there's 150 of them. But a handful that can work, and then once you have a few countries doing this sort of thing, other middle powers really have an interest in joining.

I mean ultimately it should be global, right?

Exactly. There's a difference with the tariffs. The tariffs, the threat was given and then you can't create these sorts of alliances in a week. AI, we have a slightly longer runway to these potential risks of domination by AI. We're talking about at least years, right? So it's not in the next 12 months. It's maybe 5 years. Well, nobody knows, but it's not as quick, right? It's not weeks, it's years. So that gives a bit of time to negotiate this sort of thing and get going with research collaboration, development collaboration and so on.