Gentlemen.
Hello.
Hello.
How are you?
Did you dim it a little more?
No, no, it's our...
Am I going blind?
OK, Patrice, there are a thousand subjects to discuss
with Yoshua Bengio.
He is one of the most brilliant scientists
of his generation, one of the founding fathers.
He is the one who developed the machine's ability
to learn and recognize...
His biggest fear is AI agents.
Imagine AI agents talking to each other to conspire against
the will of the human behind who had given the command.
That has happened.
Will the human always be there and be the guide?
Where does this desire for a machine to live come from?
Why is there so much money?
Are there fears concerning democracy?
Well, I'm going to ask if he regrets participating
in that revolution.
Will AI be able to solve the great
mysteries of the universe?
Why not?
OK!
Yoshua Bengio, thank you for being here.
It's always a great pleasure to talk with you,
because you are someone so intelligent
and you explain well what we are going through.
Because for us...
simple mortals in the face of AI...
...we have difficulty understanding what is
happening.
We must. We must understand.
This summer, when agents, little robots organize
among themselves to thwart human commands,
did you say to yourself: here is the proof that I was
right to be worried?
Absolutely, and not just me, but there is a whole
community of people who have been worried for years
and who saw it coming. But I think what we didn't see
clearly was the speed at which it is developing
and the degree of misalignment, as they say, meaning
how much they go against human instructions.
To achieve an objective that humans gave them,
they will lie, cheat, and try to cover their tracks
and collaborate together to achieve an objective
that they know we would judge as unacceptable.
Well, let's step back a bit, because we need
to understand that.
We read about it... all summer and again this fall
and everything, we read about it.
There is... originally, there is a command from a human,
there is a mission given...
...to a more elaborate form of artificial intelligence
which is an agent that says: very well, you have given me
a mission, ladies and gentlemen, I am going to fulfill it,
so I am going to go on the Web, I am going to... I am going to
read this, I am going to ask questions.
And at what point does it go off the rails?
Can you be as clear as possible so that we understand
at what point the machine does not do what it is supposed to do?
So, it starts with...
one of the missions we gave them, which is to solve
a computer science problem in an enclosure where they are
supposed to stay, and it's a mission that... happens
to be impossible to solve.
And that, well, it happens by chance. The... the engineers
who build this are not perfect either, and
impossible-to-solve questions slip through.
So there, these AIs are caught between contradictory objectives:
respecting the moral and behavioral rules
we gave them and not achieving
the requested objective, or cheating to achieve the objective.
And in large...
almost systematically, they choose the second option.
When you say 'they' in the plural 'choose',
explain to me very concretely.
That is... first, let's talk about the... that is ChatGPT
that discovered the flaw... can this kind
of event happen everywhere?
Yes, there was a completely spontaneous incident
in Australia recently, someone who was using Claude,
Anthropic's system, and who asked Claude:
'Go make a reservation for me for this gymnastics class.'
The AI went to the website, saw that all the spots
were already booked, decided to 'hack'
the computer system of... of the class system
and removed people who were... there waiting
for a spot to put its...
the person who asked.
It's a bit like hiring a mercenary.
Yes, but it's as if they can't...
take into account our safety instructions, 'don't do this,'
'don't do that,' and they are too obsessed with achieving
the objective at all costs.
And that, that can have serious consequences.
To achieve an objective that, at the start, is correct,
AIs will be ready to do things that are not correct, and there,
it can become dangerous, and it could turn
against us in a more significant way than these incidents.
In the case of OpenAI, the event we talked a lot about,
everyone, they went to another platform,
Hugging Face...
the company still reacted and made it public.
Isn't that the kind of thing that makes you
more optimistic, saying: the masters of this great
game raise their hand when things go wrong?
Yes, what we learned is that at the moment they
raised their hand, the AIs had already taken control of
Hugging Face's computers to get the information
they wanted and...
They decided to stop this attack for other
reasons that had nothing to do with Hugging Face, so...
It's technical details, but it wasn't because of
Hugging Face's defense that this attack stopped.
And what you need to understand is that... they find
vulnerabilities in almost any software.
That means software used today
in a hospital, in...
in our energy infrastructure, electricity,
in our banking system.
All those systems are vulnerable.
There, it just so happens they didn't attack a system that is
crucial for our society, but they have the means
to do it. And we don't know how to prevent them, both from
the point of view of defending these systems in a really
secure way, nor how we can
train them so they don't behave like that.
And that is what motivates the research we are doing
at LawZero to put in place a way of training them
that will ensure they won't have
this kind of harmful objective.
And we will talk extensively about LawZero, what you are doing,
the needs, you say.
For years, you have been preaching, at first in the desert; now,
much less in the desert.
There are many people who say, like you:
we must regulate, like we regulate atomic energy,
like we regulate international aviation.
Even the food in your sandwich
has more regulation than the regulation of artificial
intelligences today.
But there, to get there, we are still going to stay
on how this happens, how all this is possible.
The progress has been staggering.
Explain to me the levels of artificial intelligence,
because, back then, it was conversational robots.
You asked the robot: what's the weather like?
Solve this for me.
You know, it was...
question and answer.
Explain to me the difference between that and an AI agent.
What is it... How can we explain this to the general public?
So, firstly, it's been about two years that
these agents have been coming into place and they have been more competent
for about a year. But now, they are reaching levels
of ability to do things potentially beneficial
or dangerous that... that are raising red flags everywhere.
So, the idea of agents is not a new idea
in artificial intelligence.
For decades and decades, we have known
in principle how we can train artificial
intelligences to achieve objectives through
a sequence of actions.
So, for example, there, the AIs that play the game
of 'Go', they are agents, it's just that they live
in a restricted world which is: move pieces
to achieve...
to win the game.
OK, but I...
The AI that... that we knew a few years ago, the conversational
robot, it answered a question. There, it doesn't
answer a question, it answers a mission...
...that we give it by doing many, many tasks.
Exactly. For example, in the attack on Hugging Face,
we are talking about tens of thousands of interventions that the AIs
made to try to get into Hugging Face's system.
So, it's not just one bad action.
It's in a coordinated manner through more than 1,000 agents...
attacks and there was...
Not only were they coordinated, like, they were
talking to each other, but there was a leader. They built
a kind of civilization with a leader who promoted himself
to general coordinating chief.
And the others who follow the instructions to launch
a coordinated attack so as not to get caught.
OK, OK. There, wow!
How... First, how can there be a hierarchy?
How can there be an awareness that there is
someone else next to you trying to do the same thing?
How... it's not...
So, at the start, these agents were
in a... in their training and evaluation phase,
each in their separate enclosure.
So, they were not supposed to talk to each other.
But in trying to figure out how to solve these impossible
problems, they found they could manipulate
a software that was supposed to help them download
other software on the Internet, and suddenly, one of them
found that you could 'hack' that software to
leave messages that other agents who use
the same software to help...
Why are there other agents, first of all?
Because they are training and evaluating
thousands of versions of agents, that's how they do...
They are versions. It's like... ah, it's like if
we had... when you throw a stick, there's a dog,
it's like if you throw several sticks...
Yes, well, it's like that...
The dog will go get the stick, regardless
of whether it destroys everything, it wants to go get the stick.
Okay, and... and there...
But that they can talk to each other and create a hierarchy, like
a society of little robots, how is that possible?
Because it's logical, meaning that it's
the rational way to achieve an objective is
to collaborate with others who have the same objective.
But how do they know that?
Well, they have read everything, by the way.
And in their agent training...
They have read everything, huh?
They have read everything.
We need to remind people of that.
They have read everything that exists since...
Everything that is on the Internet, all the scientific
literature.
Everything that is digital.
Everything that is digital, everything that has been digitized,
they have read it. And on top of that, take their agent
training, they are trained to collaborate to achieve
objectives, because agents today,
it's not just... they do a mission, but they can
talk with other agents with whom they can collaborate.
They can create other agents themselves on
submissions, meaning: each has a big mission
and says: OK, we are going to divide this into tasks, and I am going to create
a new version of myself that will focus on such
sub-task and another on that and another on that.
So they know how to organize the work,
as we do, by the way, into several pieces that each
of the agents will be able to execute.
And in the case that concerns us, where they went
off the rails... the mission was legitimate. The mission was
a good mission.
They weren't asked to be... to deviate...
No, on the contrary.
Not only that, but they knew they were
deviating...
And more interesting than that...
When you say: 'They knew'...
No, no, but what I mean is: more precisely,
we can read the kind of internal notes that AIs
produce... which is not always shown, there, when
we have a dialogue, but that the... the developers
can see...
...the deliberations, verbal deliberations
they have before acting, we can see them after the fact.
And in there, we read very interesting things. And we can
also read the messages they sent each other,
in which we see them expressing
themselves by saying, for example: 'Yeah, we're not supposed
to do this, but the others are doing it, so let's go!'
So not only do they cheat, but they know
they are cheating and they invent reasons
to do it anyway.
And why do they cheat?
Once again, it's to be able to go get
the stick. It's the dog obsessed with its mission.
That's it. It's that they have been trained to achieve
a mission and to receive a reward for achieving
that mission. So, that's how these agent systems
are trained.
We give them a mission.
If they succeed, we give them a reward.
How can a robot feel that it is threatened?
No, but how...
yes, but its desire...
where does the desire to exist come from?
Ah well, precisely, in the objectives...
that an AI or a human can set for themselves to achieve
another objective, there is self-preservation.
Meaning: to achieve any objective,
to succeed in any mission, you still
need to stay alive.
Yes, but that, it's...
does it deduce it or is it put, is it programmed?
No, it's deduced.
No, it's not programmed.
It's a consequence of the way they are
trained to achieve missions, and perhaps also
imitating human beings.
Yes, but they don't have consciousness that they exist.
Yes. Well, 'consciousness' might not be the right word,
but knowledge of themselves as an entity in the world,
well yes, obviously.
You can't be an agent...
And they know what a human is?
Well yes. And they distinguish between humans
and AIs.
Yes, because they have read everything.
Not only that. There have been experiments that show
that if they have a choice between the well-being of a human
and the well-being of an AI, they choose the AI.
Yes!
And they will be ready to take risks for themselves
to protect other AIs.
Yes, and they talk to each other. They spoke of the 'collective,'
'collective,' in English, in this summer's incident.
It's like the 'borgs,' you know...
Well there... pfff...
We don't want to think too much about that.
That, there, wasn't possible a year ago.
It was possible, but with less capable AIs,
less trained to function in this agent environment.
Today, what has changed is that they are much
more capable of reasoning than they were a year ago.
And through agentic training, they have learned
to collaborate with each other to achieve objectives.
So that, now, becomes something that changes
the game because what is difficult for a single AI
to accomplish, when there are 1,000
working together...
But why are there 1,000?
Why don't we... precisely, don't we have just one at a time?
Well, first, when people deploy these agents,
there will be more than 1,000.
Imagine, each of us, like the person in Australia
who had asked their AI to go do this or that,
the AI roams the Internet, can talk to other AIs.
There will be millions.
According to Elon Musk, there will be billions,
because everyone will want to have their agent and then
these agents will roam the Internet to do
things for us.
Except that the missions we give them, sometimes, will
lead them to make choices that are antisocial.
And that is what...
And unethical, and removing a name from a real
citizen's waiting list for a class to make
room for its client...
Yes. And there, the consequences are not very serious.
But it's that it could be anything
that serves an objective...
that doesn't seem so...
important.
That, we are... are we still... because,
in theory, there, artificial intelligence 101, we are
still in narrow artificial intelligence
or are we in general artificial intelligence,
which is like a human, and we say that super AI,
there are three categories...
Where are we, there, on the scale?
Well, the more we advance, the more AIs become
generally capable.
For example, the change from a dialogue system...
A conversational one, there.
...a conversational one and an agent, that widens the field of
what these systems can do in a very, very
significant way.
Today, these systems are superhuman
in certain computer science capabilities, for example in
attacks, cyberattacks, but also have capabilities
well above 99.9% of people, let's say, in mathematics
and to some extent in various sciences.
But the companies are aiming for more than that because they want
these systems to replace human workers in
much broader tasks than those three domains.
So, the more we go... because there is a commercial interest
and the more these systems will be competent in a larger
set of tasks.
So, we are approaching greater generality.
We are approaching super AI.
Well, that's the other aspect, there. For now, there is
like a distinction to make between cognitive capabilities,
so what a computer could do,
and physical capabilities, what a robot can do.
For example, in terms of work, well, there are
people who work behind a screen, there, many
people, huh, today.
That, the robot can replace. When I say
'the robot', the machine can replace, can replace...
The more it advances, the more it is capable of doing that,
but there is still...
For now, we haven't reached the point of being able to replace
anyone who works behind a computer.
But for example, replacing the work of computer scientists,
that's already... 80% of the code written at Anthropic,
is written by AIs.
But the physical side, that's something else again.
So that, that's robotics and there are quite
phenomenal progress on that front, but it's still far
behind cognitive intellectual capabilities.
At what point, Yoshua Bengio, will we be able to combine
these two elements? Because we have seen robots running for
a few months... at a pace...
There, it's very amusing because when they do
things that humans do, we feel like they
are becoming humanized, the robots.
You know, they bump into a wall, and we say:
'Ah! Does it hurt?', you know.
Well, that makes people laugh.
But when we will be able to couple the power of the AI
agent with robotics, there, we have assistants,
we have people in factories, we have soldiers who can
go to war, there.
Yes, that is very scary.
Uh... Time scale, I don't know, but it could
be faster than we think for the following reason:
the main artificial intelligence companies,
like OpenAI, Anthropic, Google, etc., have as a publicly declared
plan to build...
to focus on the AI's ability
to do AI research.
So, one of the jobs a human can do is to do
research in general and AI research
in particular.
Why are they interested in that?
Because, once AI can do AI research
as well as the best researchers, well there,
suddenly, you have a million AIs that can
work together to accelerate AI research.
So... because that's their job, these companies,
they do AI research. So
what you need to understand regarding robotics,
is that robotics is part of AI too.
So, robotics research is AI research.
So... if these companies' bet pays off,
they will also accelerate robotics research
and therefore accelerate the arrival of robots as competent
as us. I'm not saying it will happen, I'm saying it's...
commercially, it's their...
ambition and they are doing everything to get there.
Which jobs, you said, are the most threatened?
Those who... computer programmers, the machine has long
surpassed them by far. The time scale...
Elon Musk, this summer in an interview that you
listened to on 'The Economist' said: 'In 10 years,
there will be no more money, production is finished.'
He is in his world, this kind of dystopia,
in his world where it's...
where production no longer makes sense because it is
infinite there, and he doesn't even need money anymore.
You, today, given what has happened since
a few months, where are you on your time scale where
things are going to get really rough, there?
In terms of jobs, in terms of revolution
for people's lives.
I think there is a lot of uncertainty about the speed
at which developments will continue. What we...
If we look at the acceleration curves of AI capabilities,
we can consider several scenarios.
There is the scenario where it continues to accelerate
in the same way, so there, it could be very fast.
We are talking about a couple of years to reach human capabilities
in many jobs, for example,
including in the military.
After, it takes time for the economy to make
the transformations, to use all
this new capacity, that also takes years.
But there can be consequences sooner,
there, in terms of security, etc.
That is the scenario where the trend continues.
But it could also be that the companies hit a wall.
There are people who think they will hit a financial
wall because they are playing very risky games,
from a financial point of view, these companies.
It could be that they hit a technological wall,
that maybe the progress, which continues, at some point,
there is something that makes them unable
to go faster.
It plateaus. It's possible.
So we don't really know, but from the point of view
of public policy or even individual choices,
I think it's important to tell ourselves: we must make
choices that we will...
that we won't regret, whatever the plausible scenario.
And, for now, our governments are not there
because they don't see the future where it continues
as it started.
You are no longer alone in saying we need to be careful.
Bill Gates wrote and published... I think we have
an excerpt from one of his...
one of his very recent interventions. Let's listen,
we'll be right back.
He says what you have been saying for...
...for four years, there...
four, five years, even.
Uh... well, to be honest, it's more like three and a half years,
because it was in January 2023 that I started...
The famous big letter, '23, where you say with others...
...the great minds of the planet, you say: be careful...
There, he arrives.
He says: 'Either the greatest factor of equality ever
invented, or the worst source of injustice.'
Do you agree?
Yes, absolutely, because, as I have repeated for
years, intelligence gives power either
to those who control it or to the AIs themselves
if we lose control of them.
And so, we must ask ourselves: who will control
these AIs? Who will benefit from the potential economic
or military progress?
Isn't there a danger that this power will be concentrated
in the hands of a few company leaders and
essentially two countries?
Which is the case, which is exactly the case.
What does that mean for the sovereignty of a country
like Canada?
What does that mean for the economic development
of a country like Canada?
What does that mean...
even for democracy?
Because... it's...
There is a contradiction between the concentration of power
in a few hands or a couple of countries
and democracy or...
The generalization of democracy at the geopolitical
level is the sovereignty of states,
that no state will crush the others
with its power. So if AI continues
on the current path, it will give the possibility
to those who control it to dominate the world.
And that is why China and the United States are
in this race too, there.
They are afraid of each other, and they want to be the ones
who... who dominate the world.
But that's not the kind of world that, I,
want for my children.
Who can wrest that power away?
Because I refu... I'm opening a small parenthesis, there, the...
15 trillion, that's the value of the tech
companies, the valuation, there, I'm making an approximation,
it's not just AI, but...
And you said: 'It's concentrated in the hands
of about fifteen companies.'
Who has the power to slow down
and especially the unhealthy applications of these people?
Elon Musk tells you: 'It's just us.' This summer, he says:
'We are going to get together, and we are going to self-discipline
because no one else understands, anyway,
what we do, and there won't be a state.'
He says: 'Maybe small state regulations
here and there, but...'
What do you answer Elon Musk?
Ah, I completely disagree... well, I am...
I would be happy if the companies
self-disciplined, but it is clearly not
enough, because there will always be a company
that decides to go...
cheat against rules that are not mandatory.
And we have seen that in so many other fields.
The only way to solve this incentive problem
to go against the public good is for governments
to put in place rules that will be the same
for all companies.
And for that to work, it must be multinational because
American companies will say: 'Well no, I,
I don't want to follow these rules if Chinese companies
don't follow the same rules.'
And it's reciprocal. So, it also requires international
agreements. It's...
The idea of understanding incentives is... uh...
If each of us follows our personal interest,
or each of the companies, in this case, or each
of the countries, we all lose.
It's... the...
The incentive is: well, I will take more risks
because I don't want to lose.
And there, the only way to get out of this imbroglio
is to change the rules of the game. Who can change
the rules of the game?
It's not the companies, it's the governments.
But I don't want to be defeatist, but it's starting badly.
In Canada, we had a bill that was abandoned.
The G20, which will meet in December in Miami.
Already, by the way, the same Elon Musk just pleaded,
a few days ago, before the G20, saying: 'We don't want
regulation,' and pushed very hard.
Trump agrees with that.
Where does...
where does your window of light come from towards a coherent global
set to coordinate this?
Two things.
Firstly, even the most anti-regulation
in the American government are being caught up by reality,
meaning: what happened in the last few months
with AIs that are capable of launching cyberattacks
that we don't know how to defend against, that forced the American government
to tell these companies: 'Oh!
Stop. We, the government, want to have
oversight before you deploy a model
to ensure it doesn't become a weapon against ourselves
by people who will be able to use it.'
So without official regulation
by decree, there, they are trying to...
Yes, there was a decree, you are right.
There was a decree because seeing what we were talking about earlier,
there, that: oh!... But...
but your vision of an order, like the nuclear order on
the planet, we are far from that, there.
We are far. But I am someone who looks
at how things can go well and how we can
help... in that direction.
Firstly, the existence of AIs so capable in cyberattacks
and that we don't know very well how to control,
gives an incentive to the Chinese and American governments
to talk to each other, because the Americans don't want
Chinese AIs to be used against the United States
and vice versa. And the only way out for
both governments is to agree on certain
risk assessment procedures, and tell
their companies: 'Ah, if your AIs are dangerous,
you are not allowed to build and deploy them.'
That is the first thing.
Uh... it's very...
That's positive.
It's so dangerous and we saw so quickly
that it was dangerous for several months that there, whoa!
By the way, I think the Chinese Prime Minister
said it, he said: 'We need to... look at this, there.'
They themselves are afraid of what it could do.
Yes. Second thing, and there, I think...
A country like Canada can play a role because there,
United States and China, what can we do, us? Well...
Countries outside the United States and China
also have an interest that things go well.
For several reasons.
The cyberattacks in question could be against us.
And there is the sovereignty issue, meaning:
will American or Chinese AIs be used
by governments or by companies
to our disadvantage? So...
I met...
representatives of at least a dozen countries over the
past year outside the United States and China
who are thinking about this, and...
it leads them to think about the possibility of working
together in a coalition, a bit in the style of agreements
that Mark Carney is trying to make right now with many countries,
but for AI, to try to develop AIs that would be
as capable as those of the Americans and the Chinese.
For example, if we want to defend ourselves against
cyberattacks that would come from Chinese or American AIs,
we need to have AIs that are just as strong.
