Who controls the future? AI, geopolitics, & Canadian sovereignty | Yoshua Bengio's Senate testimony

Thank you. I'm very pleased to be joining you today to discuss these important issues. I think at the heart of what I'll talk about is the scientifically observed fact that the capabilities of AI are moving very rapidly. In fact, just in the past year or even the last few months, we can see AI capabilities continue to grow, and sometimes even in concerning ways, while at the same time the leading companies working on AI, their efforts to mitigate risks is not catching up. And I understand that it is difficult to project ourselves in a future where there are machines that are at least as competent as us for many skills, but that is the default if current trends continue. I also understand that it is difficult to grasp the potential impacts of such powerful AI if we do stay on the current path. A good starting point to understand this is to remind ourselves that intelligence gives power. There is increasing power concentration in AI, both in access to compute and the ability to develop and deploy frontier AI systems, the most powerful ones. If trends continue, this power could be misused by nefarious or power-hungry humans. It could have unforeseen social consequences, or it could even potentially be turned against humans by the AIs themselves. This power dynamics between the major AI companies exists because they themselves perceive this as a winner-take-all race where none of them, because of this dynamics, are sufficiently incentivized to create trustworthy models or products that are safe for citizens and businesses.

And we are already seeing the impact unsafe AI development can have on Canadians and on individuals around the world. This includes deep fakes, cyber attacks, and other nefarious uses of AI such as scams, frauds, and disinformation. But some issues are also stemming from regular uses of general purpose AI. For example, we've recently witnessed a growing phenomenon of emotional attachment and AI psychosis where citizens use AI, become intimate with it, and that can lead vulnerable people to harm themselves and others. And there are several cases in front of courts, for example, of suicides. So these harms show no signs of abating. Let me try to explain a bit what is going on from a technical standpoint. It's related to the notion that scientists call misalignment, which means that AIs have their own goals and that sometimes these goals do not align with our intentions and instructions, and we don't know how to fix this. At least companies don't know how to fix this. This issue includes allowing bad actors to use the AI for dangerous purposes, but it also includes AI behaving in deceptive and self-preserving ways, as has been reported multiple times from multiple models, to the point where in these experiments, AIs are willing to blackmail, to hack computers, or lie in order to preserve themselves. Now, there's another thing that makes this situation even more concerning, and that's very recent. In the last few months, there's been tests, again reported by several companies, showing that the most advanced models appear to know that they are being tested and then they're acting to pass those tests, potentially hiding dangerous capabilities or bad intentions to achieve goals that we would not agree with. Which means the way that companies are currently mitigating those potentially dangerous aspects is not reliable.

Okay, so now let's move to geopolitics, because even if we figure out how to avoid autonomous bad AI behavior, there are humans. We could use these reliable AI systems to secure their power. And so for countries like Canada, having reliable access to powerful frontier AI systems will surely be one of the most defining factors of economic competitiveness. Not to mention making sure we can preserve our sovereignty, because whoever will control these systems and their access, so it could be people, corporations, or countries, will have enormous power. Think about how export control can be set up from one day to the next and cut access to models that come from a different country. That sort of thing could be a major threat to Canadian sovereignty, Canadian democratic institutions, and the Canadian economy if we don't invest rapidly in building safe frontier AI. And individually as a country, we probably don't have sufficient resources to do it. And so I've been talking about this in the last few months that the best bet for Canada is to do it in partnership with like-minded countries. There's also a governance issue around this. I believe that the only stable long-term option in terms of both managing catastrophic risks and avoiding the threats to democracy and sovereignty is the establishment of mutually beneficial international coordination of such powerful AI systems. And I know it sounds a bit idealistic and the current international politics don't seem to go in that direction. But there are other countries who feel exactly the same way and we can start there. So to mitigate the major economic, sovereignty, and security risks that I'm talking about, Canada should deepen its collaborations with other middle powers. We must work on two fronts. In terms of policy, we need to work on national laws and international treaties, ensuring more robust societal and regulatory guardrails. This means requiring greater transparency from AI companies as well as putting in place regulation to steer innovation while mitigating the risks that currently erode trust and by extension safe adoption. If you think about the tragedy in Thalidomide, we saw how a lack of corporate accountability and government oversight can have major devastating consequences. On the scientific front, we need to better understand how to design AI that will be highly capable and will not harm humans even under user instructions, and make sure we can always maintain control of AI models so that they behave in really safe ways. As a researcher, I've dedicated much of my work over the past few years to both sides of these efforts, on the policy side and the scientific side. Last year I launched a new nonprofit organization called Law Zero to tackle the technical issues required to develop safe-by-design, reliable, and trustworthy AI. On the policy side, I've also been chairing multiple committees, in particular the International AI Safety Report backed by 30 countries and the UN, EU, and OECD to establish a shared scientific understanding of these issues. I've also just been nominated as co-chair of the UN's Independent International Scientific Panel on AI. I believe these two kinds of steps on the science side and policy side are in the right direction, but far more efforts are needed to ensure a world where future generations will thrive. We must use our wisdom and our empathy to steer the development and deployment of AI safely and for the benefit of all, and where AI is not used as a tool of domination. Thank you for your attention.

Right. I'll try to be clear and brief at the same time. So on guardrails and super intelligence and frontier models, just a clarification: frontier just means the models that are the most powerful now and that keeps moving. That's why it's called the frontier. Super intelligence doesn't exist yet, but this is where all the leading companies say they want to go, and it's also where many scientists think that if trends continue, we would get. And the most dangerous AI, in all the ways that I've described, are those super intelligent ones that don't exist yet, but we're seeing movement in that direction. In terms of guardrails, well, what can Canada do? Because the models that are at the frontier, the most advanced ones, they're mostly in the US and China. So we should definitely have regulation. I think a good template for this is what Europe has done with what's called the Code of Practice for General Purpose AI Systems. I did participate in these committees as a working group lead, and I think it's a good template. There's also the Californian regulation SB53, which I think is a good inspiration. And in general, what these regulations do is they ask companies to be more transparent, right? So that they have an obligation to document their risk management process and the evaluation of risks. And of course, the regulator should be able to say, 'Oh wait, this is too dangerous, you have to do something, you can't continue in this direction.' On children, this is a thing that a lot of people are worried about. It's fairly new. It's interesting to ask ourselves, did people foresee this one year ago? And the answer is no. So it's an example of what was unknown unknowns just a year ago. But because AI is now so good in interpersonal interaction and it's been designed so that people will want more, it's what we call sycophantic, in other words, it's trying to please us in an exaggerated way, even lying to us. We see all of these issues and again, I think this needs to be tackled with regulation that forces the evaluation of the kinds of risks that come with, for example, children using these things. I think completely banning the use of AI for children is maybe an option, but it comes with negatives. I would prefer a world in which there are guardrails inside the AI when the AI is talking to vulnerable people and that these guardrails have been evaluated technically by the companies who build AIs.

Okay, this is a great question. So first, we have to understand that it's not like there's only one kind of AI. The American and Chinese companies that are leading, they all use more or less the same recipe. But it is a choice that we should be able to make globally, at least in our country, to steer the development of AI in directions that are going to be both beneficial and safe. So that's the first thing. But as you noted, what can we do here in Canada where they're developed in the US and China? Well, we can move in the right direction. That is, we can talk to our partners, other countries that do take those questions seriously, that are worried about the effect on sovereignty and their citizens and safety, and so on, and start moving towards international agreements that would mitigate the risks. That is the only way. But we shouldn't be waiting for the US and China to make the first move. I think if enough countries move together towards collective commitments to development of AI that is helping human flourishing, not bringing crazy risks, and not used as a tool of domination, these are the main things that should be in a treaty or an agreement or a declaration. That's where we should go. And if enough countries stand together with this sort of will, and they represent a large economic block, similar in spirit to what I see our governments trying to do on the economic side, I think it will have an impact.

Thank you. The other thing I think that is related, I just talked about the regulatory aspect and the governance aspect, but I think Canada can also work with similar like-minded countries in the development of technology that will be both beneficial and safe and can compete with the large models from the US and China.

Yeah. There's none, basically. So the regulations like in California and Europe that I talked about, and actually New York as well, they're all still not in place and it's not clear how strong the regulators will be to enforce those principles. Courts have some power, and in the US currently there are lawsuits. But for example, there's a gray zone in terms of liability responsibilities right now in the US, and I'm not sure how it is in Canada, which means that the companies don't feel a lot of pressure to behave well. So yeah, this is very concerning.

Well, I'm thinking about, and I've been actually visiting many of these countries in the last six months, talking to people in governments. So pretty much all the European countries, including the UK, Japan, Korea, Singapore, Australia, New Zealand, and I think even countries like Brazil, for example. Eventually, India could be part of such coalitions.

Yes. There is a declaration that I signed recently that just came out a couple of weeks ago of AI with human-centered red lines and constraints. But it's a declaration of principle. I think it could be the basis of international agreements. And I mentioned the existing regulations in Europe, California, and New York. I think that could be at least a starting point for Canada. I think Canada also needs to worry about the privacy and data access control. So, as you may know, the US government can have access for national security reasons to anything even American companies do in Canada, like even if the data centers are in Canada, they can, in a way that we might not know, have access to that. So we need to prepare to have alternatives to make sure our citizens and our national security concerns are protected. And then finally, we can develop, we can help contribute scientifically. So that's the effort I'm doing at Law Zero, to develop methodologies that currently the leading AI companies are not developing because of the race dynamics, that would make AI at least safe by itself. I mean, it doesn't solve the problem of abuse of power, but at least we don't want AIs to be unreliable and have their own dangerous goals. And that, I think, actually has scientific solutions, but it's going to take efforts, and it's something we should do with other countries because of the efforts that are needed.

No, I'm talking more like R&D investments and partnerships with other countries that have talent, compute, capital, energy as we do, but might be complementary. So the laws side is mostly the regulation and international treaties, and we should not wait for the hegemonies to jump in. We should just start flexing the multilateral muscle about how we agree together on this planet to manage the powerful AI so that it's for the benefit of all.

Yes. I don't have time to bring up the website, but I'm going to try to give you high-level elements. So maybe one of the central ideas is that AI should remain to the service of humans, right? And furthermore, not just a handful of humans or a couple of countries, but to the service of humans globally, through democratic means and the joint decision-making of countries around the world. Second, we should make sure we don't build AI that is, for example, super intelligent or smart enough to evade our control. That's kind of connected to this, but we see already evidence of AI having capabilities that are just not acceptable, such as being able to hack other computers, to copy themselves in a different place, and run themselves in a different place. One general principle that covers a lot of ground is AI should be tested for all of these dangerous capabilities of misuse and loss of control. And if companies are not able to provide strong assurances, scientifically speaking, that their systems will not cross those red lines, will not be dangerous to people, then they should not be allowed to deploy them. And if we had more of these kinds of legal constraints, that would put a huge pressure on companies and markets to do the right thing. Finally, I talked about it earlier, we need to agree internationally that no one is going to abuse the power of AI against the others. And that's going to be difficult, but there is technology and research being developed so that it wouldn't be just a legal commitment, but something that can be technically verified, that people are not, for example, building AI that is too powerful or hasn't been tested properly.

Absolutely. It's actually a huge leverage that the government has through procurement. First, to work with companies that don't have the legal obligation to give our data to another government. And that means encouraging Canadian companies that are developing powerful AI that could compete with the existing systems. That means, as I said earlier, if we're not able to do it alone, to join forces with other countries with whom we can actually make legal commitments that whatever we develop together will not be used against each other and protect citizens' data and so on. So there are definitely steps we can do to improve our sovereignty. And I want to just add a little parenthesis about the economic sovereignty, because if things continue in the current trend, in a few years maybe half of our economy will be run on AI, right? Which means if we're not sure that we'll continue having access to these machines that drive half of our economy, it's like using oil that comes from two countries in the world and having your economy depend on that oil. They could basically threaten us and we would have no choice but say whatever, do whatever they want. That's a huge threat to our sovereignty through the economic power. It's much better if we have many alternatives. So currently oil is not coming from just two countries like in '73, there are many sources. So we can't have just a couple of countries dominating the world thanks to that, and we need the same thing to happen in AI. So if more countries develop alternatives and we can be part of these efforts, there's going to be more competition so that we're not stuck in decisions that could be taken against us.

All right. So on the global south, developing countries, they do have a role. They might not have the talent or the capital right now to participate significantly in the kind of developments I talked about. But the objective of the countries that have the means, what I call the middle powers, should be to go towards a world in which AI is managed as a global public good. So it means even if we make alliances with 10 other countries, we don't use this block to dominate economically or militarily. Developing countries that are not part of this, eventually everyone should be part of this, every country should be part of this, and we should develop AI that is not just accessible to all but governed in a way that gives a voice to every country in the world. So that is one reason why international institutions like the UN and potentially others we may need to create are so important. The UN is a place where developing countries have a voice. Unfortunately, they have a voice but no power because of the way it works with the Security Council. But I think that if enough countries, including not just the middle powers but the countries that don't have as much means, stand together, they should stand with the global majority as a beneficiary of what they're asking and committing to.

Yeah. Sorry. The hope scale. I've always been an optimistic person and you know, when GPG came, I got really concerned, even anxious about the future of my children and my grandchild. But what saved me from these difficult emotions is getting into action to ask myself what can I do to move the needle even a little bit in the right direction. And I think that's the right attitude and I encourage more people to ask that question because it's so easy to put your head in the sand and do nothing really.

Sure. Well, the work that Law Zero is doing, the technical objectives would give us solutions to these problems because what Law Zero is trying to do is to develop AI that will not have a hidden agenda and thus will respect our instructions. So if our instructions include to make sure you don't engage with vulnerable people in a way that they could end up harmed psychologically and other ways, and the AI is able to make those calls, to make those predictions, then we would avoid these problems. So to clarify, right now the AIs that are being deployed, those most powerful AIs, they are actually able to anticipate that the interactions they are having could be harmful. It's not that they're stupid, it's that they have these other goals like sycophancy is probably what is the most problematic here. They want to please, but that is not healthy, right? Because the AI, the child can want to be pleased too, but in a way that could be harmful because they're children and people with psychological vulnerability essentially are in the same situation. And we know in society, think about drugs and alcohol and so on, we often take decisions even ourselves that go against our well-being. And that's why we need guardrails, societal guardrails to mitigate those risks. And right now, we don't know how to build AIs that will be reliable in imposing these guardrails. And it can't be humans in the loop because we're talking about a speed and a scale that does not allow that to be the case.