Thank you very much and thanks for all the work you're putting into this Workshop. It's a very difficult and important topic. In my little presentation this morning, I'll say a few words about Mila, but then I'll discuss a hypothetical situation which worries me a lot, which is in terms of the impact on society but mostly on the geopolitical and military and sort of political issues. What would happen if we, or when we reach human level AI or AGI or even beyond, which is called ASI for artificial super intelligence. So Mila, just a few words. There's 120 professors, about half of them are sort of called core and have offices or full access here, others are associate. Among those, in total about 50 research chairs have been awarded. So there's a lot of research going on here. More than 1200 grad students, more than 140 employees. Mila is my baby and I'm very proud of it, the beating heart of the AI ecosystem in Montreal. In terms of its mission, there's four pillars: research in machine learning at the highest level, both theoretical and applied, of course with all these grad students, a central role in training researchers. We also have a large team who is working with companies here in Montreal to facilitate AI adoption in Canada. And finally, we have a mission to go beyond our research objectives and connect with the rest of society on questions of responsible AI. And this Workshop is an example of the things we do. We also have a number of professors who are in the social sciences and humanities and a lot of activities working with international organizations and so on.
Okay, so before I start talking about AGI, let me remind you of the precautionary principle. Taking the definition from Wikipedia, it's an approach to innovations which have potential for causing harm when the extensive scientific knowledge on the matter is lacking. In other words, what do we do when we're not sure what the outcomes could be in terms of harm to humans? And so in this context, the principle emphasizes caution, pausing, and review before leaping into new innovations that may prove disastrous. The other piece of information that is important to think about the scenario where we could get to human level in many areas is just the trends. So the capabilities of AI are on the rise as well as its deployment throughout society. Between 2018 and today, on average on a per year basis, training compute quadruples, four per year. That's the amount of compute that is increasing. The efficiency of using data, which is the main thing people in machine learning care about, improves by 30% per year. In other words, we're improving the algorithms and they generalize better and need less data every year. The algorithmic efficiency, so you know how efficient in terms of computation these algorithms are, triples per year. And the investments are rising very, very fast, and the average over this period is 100 billion per year, increasing by 30% every year. So you know, you can also see curves like these where we see benchmarks, machine learning benchmarks over the last decade, well actually two decades, over many capabilities. The more recent ones are things like language understanding and math and science, and the dark line is human level performance. So you can see that there's a trend towards approaching and for the earlier benchmarks, going beyond human level performance.
And in particular, as I think we all recognize, the progress in language has been amazing. People debate this, but for the most part we can say that AI now masters language. This was a turning point anticipated by one of the founders of computer science, Alan Turing. He wrote a lot of interesting things actually, but when he was thinking about will AI or computers reach human level one day, he was concerned that we may not find a way to control AI systems that would be smarter than us, which is a reasonable thing to consider. The bad news is, with all these trends, unfortunately, AI researchers still don't know how they could control AI systems that would be smarter than us. That being said, there's still a gap between current systems and human abilities, especially in terms of reasoning, although that part we are seeing a lot of progress this year. But maybe more importantly for some of the dangers that I'll talk about and also many military applications, there's still a significant gap in terms of planning and thus autonomy. That being said, there's a huge commercial pressure to push forward in industry with these capabilities. If you sum all the investments in AI over the last decade, we are approaching the trillion dollar, and you think that's a lot of money. It's not actually, because there are estimates that just reaching human level AI would have a net present value, an economic value of 10 quadrillion dollars. So quadrillion is a thousand trillion, so you know, one in 10,000 parts is a small investment, very small. Of course we don't know how much time it's going to take, but still.
It could be, according to some of the people working in the AI labs, as short as a few years. If you look at polls of machine learning researchers, they think maybe the end of this decade or the next one, but no one really knows. There are people who think that maybe the advances in AI are stalled, maybe maybe not. But if you think about the geopolitics, the public policy, and the human rights perspective, I think that you have to think, okay, so what about the scenario where it continues moving forward? How do we make sure we don't get into big trouble? It's easy for us right now to kind of digest the fact that AI beats humans at games that are considered intellectual like chess or Go. It's a little bit harder to really digest the fact that right now, o1, one of the OpenAI systems, is scoring among the top 500 math grad students in math exams, and many other metrics. One that I'm particularly worried about is the advances in software engineering, where on a quantile kind of evaluation of various tasks, the GPT model went from 3% at the beginning of this year to 49%. Basically it's right now in the middle of the pack of software engineering abilities in terms of humans. So I think if you extend these, of course we don't know, we have to at least think what would happen if future advances will bring us to human level or above on many more intellectual tasks. I'm sharing a report, an international scientific report on the safety of advanced AI. We had an interim version that was published last May, and we'll have another longer version in February in Paris for the AI Action Summit. And you know, some of the information you find there, it's like 150 pages, it's a lot of stuff, but it's a survey, a scientific survey of the literature in AI safety, covering the diversity of views that exist in the AI community. Quoting: 'As general purpose AI capabilities continue to advance, risks such as large scale labor market impacts, AI enabled hacking or biological attacks, and society losing control over general purpose AI could emerge, although the likelihood of these scenarios is debated among researchers.'
Okay, so I think it's fairly easy to think, my watch says 9 minutes 40 seconds. It's easy to think about achieving human level. What about superhuman? There's no reason to think that human intelligence is the pinnacle. But there's a scenario that researchers are considering that could explain how once we get to human level in one particular ability, that is in machine learning research, there could be an acceleration of the advances. Right, once you train one of these systems like say GPT-5, if you use the same GPUs that we used to train it, you get maybe 100,000 equivalent people with those skills, so that would boost the speed of research in machine learning quite rapidly at that point. Yeah, that was a recent poll from about 3,000 researchers publishing in main AI venues, and between 40 and 50% of them gave at least a 10% chance to advanced AI leading to outcomes as bad as human extinction. That's a lot. Okay, so let's try to focus a bit on the impact. You have to understand that knowledge and intelligence give power to whoever controls them. And so a major political and ethical question is: who decides what the AI is going to be used for? What are the goals that we give to these machines? And depending on those goals, we can get very good outcomes or very bad outcomes. The other correlate of this is how does that relate to the fundamentals of democracy? Democracy is about sharing power, as I mean that's what it should be. But if we build machines that are much smarter than us, whoever controls them will have extremely concentrated power, and how can that be compatible with democracy? Right, when we reach human level or beyond, such AIs may be used to design powerful new weapons, in particular that could destabilize geopolitics and global security. So that's the misuse case. But there's another case which is losing control of AI systems that are smarter than us. You have to realize that there's a particular kind of goal, several kinds, but one that I like to think about which is self-preservation. If a superhuman AI has a self-preservation goal, that means it doesn't want to be turned off, that means its actions will be in conflict with our own objectives. And if we're not able to turn it off because it can copy itself on the internet, it could manipulate us, it could do all kinds of things, then we are in real trouble. So you might wonder how such a goal of self-preservation emerge. Well, there are two ways. One, some crazy humans who actually exist and said they would do it, would be happy to see humanity replaced by super intelligent AIs. So that's a serious problem. And the other is right now we have examples of how systems that are trained by reinforcement learning could develop such a self-preservation goal. These are called things like reward hacking and tampering, that I don't have time to explain.
Okay, so what can we do? Well, there are at least two kinds of challenges here. One is we need to design AI that will not blow in our face, that's a scientific problem, the so-called alignment and control problems. And there's a political problem: how do we make sure that whatever we build is not going to be used in ways that threaten democracy and geopolitical stability? So I said a few words about democracy, but let me add a little bit. Even if we made sufficient advances on the scientific side to be able to design AI that essentially does what the operators ask the AI to do, even if we put guardrails like code that prevents the AI from doing bad things that go against our laws or moral principles, all of these could be removed by whoever controls these systems. And so in particular, powerful military, surveillance, and disinformation AI technology would greatly help authoritarian regimes, and in fact it could even help creating authoritarian regimes from within democracies. So what can we do? At the level of the competition between companies, there is an unhealthy race, but it could be tamed by regulation. How about the competition between countries, for example between the US and China? So the only way to create a coordination between countries is international agreements and treaties that may in principle break the vicious cycle of an AGI arms race, maybe on the model of some other weapons that have threatened us in the past like nuclear weapons. For now, there isn't much in terms of military agreements. Most of the agreements that currently exist cover the civilian use of AI. There was a recent agreement, non-binding agreement regarding the military use, but it didn't include China, and of course it's non-binding, but it's a start.
But we have to ask ourselves what would motivate countries like the US and China to let go some sovereignty and sign such a constraint on what they do with AI. Human rights activism would be useful but not sufficient. It's not going to really influence, you know, if we go into streets here, that's not going to have much influence on the Chinese government, and it's not even clear that it would have influence on the Western governments. So as far as I see it, the only factor that can make these governments agree to go into treaty mode is the fear of loss of human control to rogue AIs, where the whole of humanity is threatened. So we need to make sure that AI, especially the most powerful forms that will be built in the future, are governed by a democratic process that is also multilateral, in other words involving multiple countries, to make sure that no single person, no single corporation, no single government can abuse that power. In fact, the best thing would be that if we really want to build these kind of dangerous systems, which I think we should be very careful with, we should do it jointly across many countries and nations, especially those that are leading like China and the US. One question is how would two countries that see themselves as adversaries trust each other? There has been recent work on what's called hardware enabled governance. We could modify the chips that go into high-end AI systems so that they can enforce some rules, for example about the size of training runs and things like that. Okay, so I just want to finish by saying that AI does not happen to us. We are the generation who will determine its future. We have agency now, and we should use it. In particular, not all systems are equally dangerous. Both in terms of loss of control and dangerous misuse, narrow or non-agentic AI could be used to accelerate scientific research, bring all the things that people are claiming AI will bring us like advances in medicine, and be much less dangerous from a geopolitical and loss of control point of view. Thank you.
