Intelligence gives power and who has that power is a question that really matters. It matters for our democracies or within each country but it matters for the geopolitical stability of the world. And currently the path on which we are is a world where if AI continues to become more and more capable there will be two countries calling the shots. So what about the other countries? Like how do we make sure that everyone can sit at the table and that we choose a future for humanity collectively that isn't the domination of others thanks to the power that AI will give.
Welcome to the most interesting thing in AI from Rethink, the Atlantic's creative marketing studio, collaboration with PWC, a leader in helping organizations use AI. If you were to pick the person who's most responsible for modern AI, there'd be a lot of choices, but a good one would be Yoshua Bengio. He wrote several of the foundational papers on neural networks, on attention, on some of the key concepts that make modern AI what it is. And then when ChatGPT came out and when this current boom started, he became one of the most articulate critics and voices of concern back then. He said he worried a lot about AI safety. Then in the last year, he has changed a little bit in a way that I'm going to talk to him about where he's become more optimistic. He still has his fears. He still has his concerns. He still sees the flaws in AI, but he's also been working through an organization he created called Law Zero to build something called scientist AI. So he's both a creator of AI, skeptic of some of the hype about AI, still a builder making tools to make AI safer. Here's my conversation with Yoshua Bengio.
Yoshua Bengio, welcome to the most interesting thing in AI.
I want to start in 2003 with one of your papers, quite a seminal paper where you have the realization or you published the realization that the meaning of words, sentences and ideas can be embedded in vectors in strings of numbers in math. Tell me about that insight and when you published it, the consequences that you could see of that.
So first some early version of that insight actually came from work in the 80s by Jeff Hinton where he was thinking of symbols that could be used as input for neural nets. But what I did with my collaborators is take this to the level of language modeling like if a neural net sees a sequence of words can it actually make sense of these words and reproduce grammatically and semantically meaningful sentences. And of course in those days we didn't have the amount of data, the compute, but we were able to show that these neural nets with new architectures that we proposed to do that could actually match or improve on the existing statistical methods that computer scientists were using to capture how sequences of words could look like. And that was the beginning of gen AI as we call it today.
Really, I mean it's this extraordinary realization that you can take all the complexity of linguistics and turn it into math which is great because then it allows you to do all kinds of things like 20 years later make chat models but it also has this consequence where you've taken this thing we can understand and see language and put it into this system that is harder to understand. So it's this both extraordinary and massively consequential paper. Let's say that it's had a lot of citations. You are the most, are you not the most cited computer scientist in the world?
According to Google Scholar. But in the work at that time I think what's interesting to highlight is the ideas that I and others had that neural nets could internally capture meaning and abstractions and that with deeper and deeper neural nets we could have more and more complex abstractions emerging without us humans having to tell the machines what those abstractions should be. So what is going on inside the neural net is the neural net is constructing representations of the things we feed an input and output.
One of the things that interests me the most about AI and we've talked about it a little bit on stages before is that it's hard for us when we look at an AI system to understand what is exactly happening inside of it. And my question for you is, do you think that our ability to understand what's going on in these vectors and these attention systems will ever catch up to the speed at which these models improve?
Well, at the current speed, I'm afraid we are in trouble. The good news is there are also other ways of understanding what is going on without having to give meaning to particular elements of like which artificial neurons is doing what which I think is more or less hopeless. The beauty of mathematics is it can give us handles on understanding and even on the behavior of these systems that we may not intuitively think possible.
You know, a priority I remember once interviewing Sam Altman and he said that the thing that would make him most confident about good futures with AI, not bad futures, is a neuron by neuron understanding of what's going on. And what you're saying is that's not possible.
It would be great, right? But it's not or it's certainly not likely, but there's so many other things we can do, which I think actually squares with Altman's position now. We should definitely try to understand what is going on. And there's some nice work in great part out of Anthropic but now the other companies also to try to get some interpretation of what is going on and that can be useful from a safety perspective. But I'm afraid it's always going to be a very rough understanding of what was happening because of the sheer complexity and the way that information is distributed in our brain as well as in these neural nets.
Let's move ahead to you know ChatGPT comes out December of 2022 and the two years after that you give a lot of commentary where you're helping to move the world to understand some of the risks. Tell me about that period, the risks you saw and what you were trying to most importantly accomplish.
So the risk I saw is that we were on a path to build machines that would be intellectually at our level or superior to us in many different areas and that it would be like opening a Pandora's box. And furthermore that at least at that point we had no clue how we can make sure these neural nets would do the things we actually want because these are not like normal programs where a programmer decides oh in such and such circumstance the computer will behave this way. No, you don't get to decide this. You only like give it experience and then you get what you get.
Yeah. And if we were to give experience to an entity that is smaller than us and it ends up having bad intentions as far as or intentions simply that collide with our values or our wishes or our future, you know, we would be taking a huge risk and in addition even if we were to master how we can make AIs do what we want there's the question is who decides, right? So, it's a political question like who gets to decide how that intelligence is applied because intelligence can be a source of power and what is democracy if not sharing power.
So, you're concerned both about, you know, we're raising this baby tiger. We don't understand that it could become a large tiger and it might eat us as opposed to just doing the tricks we ask it to do. And secondly, we don't even know when we have this large tiger who's going to have the whip, who's going to be feeding it. And it won't just be one large tiger. It will be, I guess to stretch the metaphor, it'll be a large tiger that can breed extremely efficiently.
As many tigers as you have electricity to feed them.
So you spend 2023, 2024 warning about these tigers, right? But then you become a little more optimistic, right? And in some of the interviews I've heard of you, you start to feel like maybe there's a way. So explain. Was there like a moment, a conversation? Tell me when, you know, Dr. Bengio started to feel a little more confident.
I'm more confident on the technical sides, but the politics of it is still something that gives me nightmares. The problem is what scientists call the alignment challenge. The alignment problem like how do we make sure AIs will behave according to our instructions or at least will not cross our red lines and this becomes a problem when those AIs have a lot of autonomy as we're seeing with AI agents and it becomes even more of a problem when they get very smart. So we've seen recently with Claude the model from Anthropic that seems to be able to discover how to attack our software infrastructure potentially in the wrong hands a serious national security threat to our banking infrastructure to our energy infrastructure and so on. So why is that happening? It's happening because the companies who train these systems haven't found a way to make sure the AI will not obey the requests to do something really bad that can harm us all. And that's a problem of alignment. The alignment problem can be basically showing up because we don't know how to prevent the AI from following bad instructions or we don't know how to make sure the AI by itself will do bad things. So we've seen in the last few years what was like a theoretical idea that is AIs that have their own self-preservation drives for example that don't want to be shut down and things like this which seems straight out of science fiction but is actually happening and being replicated in many labs.
All of these are AIs that behave with goals that we did not choose. They're behaviors that we would like to shape, but we haven't been able to do that because the AIs have these drives they inherited from training where they're trained to imitate people and people have all sorts of issues. They don't want to die, for example. And weirdly, science fiction, right? They've never read about like their actual existence as AIs, but they've read all the science fiction and learned from it. One of the most troubling things along those lines recently is AIs that are willing to take risks for themselves to protect other AIs, right? Even AIs from other models, right? It's bizarre.
So what I did for a while is just read the literature, think about it, look at all the experiments have been done, try to understand and I came to the conclusion that it's mostly because we don't know how to make sure the AIs will be honest and will not have a hidden agenda that we didn't choose. And then I realized that there is something in the world that doesn't have these problems. It's not humans. It's the process of how science is built. Science is built to understand the world. Period. Science doesn't have political goals. Science is just, you know, use the facts, the evidence and come up with the best possible theories that explain it. Whether the theories can be used for bad things or good things, science is blind to that. And we can build the AI that operates in the same way that's just trying to make sense of the world and give us the honest consequences of the theories that it comes up with just like an honest scientist would. It can be used to provide advice to people who have values or have goals like in a democracy or politicians maybe but the science itself doesn't have any agenda. I mean that's an idealized science, human scientists of course do have an agenda. So things are not as simple but the principle could be applied to train AIs that would be completely honest. And if we can build AIs that are completely honest, we can then get rid of this issue of hidden agenda and behaviors that go against our instructions.
So you take the tiger, you put it in a lab coat, you put it in a science reading room, and in some ways you give it the values of a scientist and the tiger acts differently.
It's not a tiger anymore. It's not even like us. It's not even like an animal. It's more like math. It's more like detached, like maybe an ideal connected to this is like Buddhist sort of detachment. You just try to understand what is going on and you don't have any preference for the world. And it turns out that there is a mathematical way to do that and to train neural nets that would behave like this. And the thing that gets me really excited these days is that if you train them in this way, so just purely try to understand the world and not do anything about changing the world, you can get mathematical guarantees that they will not do bad things with astronomically strong probabilistic guarantees. That is something if you had asked me two years ago, I would have said, 'Well, that's impossible.' Like we can't get guarantees from neural nets, they'll do whatever they do and because we don't understand what's inside and we don't control what they do but actually there is and that's fantastic as far as I'm concerned.
So there are two ways one could do that. One would be the foundation models could all agree that they would focus on these scientific outcomes, scientific processes, scientific values and then the second way which is that way hasn't yet happened. The second way is what you're doing. You created a new company Law Zero, you have a project scientist AI and the idea is to create a system that will supervise and give values to the foundation model AI.
So if you have a completely honest AI in other words something that can answer your questions with the proper level of humility unlike the current AIs, in fact quantifies how certain it is about any particular question. You can use it as a safety guardrail for any other AI, right? Because for every action that the untrusted AI would propose, you can ask your honest scientist AI, what's the probability that a particular kind of bad outcome could follow. And if that probability is above a threshold, you just reject that action, right? So now you get guarantees that bad things won't happen, you know, with some high probability.
The challenge though is the scientist AI has to be as smart as the other AI, right?
You're building scientist AI and you've raised tens of millions of dollars.
You'll be managing these other AIs that have raised hundreds of billions of dollars.
So how's this going to work?
At this point, our plan is to demonstrate the methodology, right? And you don't need billions to do that. You just are going to do things at a small scale. And you're going to show that for equivalent size models you're able to get the same ability to answer questions but without the hallucinations or the dishonesty that we see in current models.
But you're not going to build a foundation model because you can't do that with the money you have. Are you going to essentially train a custom AI on top of like Llama, DeepSeek, Qwen? How are you going to do this?
So, first you have to realize that companies already use extra models, external neural nets, maybe it's another smaller LLM to do the job of guardrail that I talked about. So, they already have these pieces in their software systems. But what we're saying is we're going to add one more such piece that we think and we want to demonstrate is going to be something we can trust because it's going to be honest and so it doesn't need to replace anything. It's like an additional safety piece in those systems in the long run. However, I think in order to get the maximum safety you also need to change the underlying AI, right? And the reason for that and maybe connected to your question is a sufficiently smart AI with the wrong goals either coming from humans or by itself could defeat a smaller less capable guardrail. So I'm also thinking of how to do that like how do we go from an honest predictor to an agent that we can also trust and have the same sort of guarantees and I'm pretty sure this is also feasible but that's more research ahead of us.
So the idea is to do a demonstration project show hey this is how you can make this much safer and then hope that the big AI companies license your technology or build their own version of the technology build it into their models. It's not like you'll have a separate AI that I as a consumer will bolt onto my version of Gemini.
I think that the path you're talking about is the most plausible because we don't have the kind of capital to train a competitive model. So a more likely thing is our ideas or our smaller systems are used and we mitigate some of those greatest risks that we see forward. The problem with current systems here in this scenario is they've been trained with this general drive to please and we get sick of it. The honest AI that I'm talking about, the scientist AI that Law Zero is working on would not have any preference for you being happy about the answer. It would give you the honest evaluation of the risk and that's what we want in situations like this. And I think another interesting example where sycophancy currently is harmful is psychological effects. People may feel depressed for example and current AIs seem, and the evidence is, they seem to agree with the user that the world is terrible and that maybe they should put an end to it. And of course, if you were to coldly evaluate, not try to please the person because they want to hear this, but coldly evaluate the consequences of nudging the person towards those harmful thoughts, and say, what would be the consequence for the person of me giving the information they are asking or like they would like to feel reinforced about, then you would avoid a lot of these issues that we have right now.
Okay. Okay. So now this is getting really complicated because the scientist AI which we've bolted into all of these systems has to be aware of the societal impacts but also the personal impacts. Right?
So who decides what scientist AI cares about? Right. Passwords, dependency on an AI model, nuclear weapons. How do you determine that roster?
So we don't want the scientist AI to care in any way. That's a mistake. The scientist AI looks at information. So what we care about is what information we are willing to provide as input and there are privacy issues and so on. But then we want the honest answer given that information what are the risks of such and such bad things, bad outcomes happening. Right? This is a social choice. It is going to be a maybe a legal requirement or it may be a personal choice or maybe a company rule. The humans are the ones who should be deciding what is right and wrong, what is acceptable. The AIs like the scientist AI don't have any preference. That's the beauty of it, right?
But someone has to write the preferences in.
Yes exactly and of course the people who write the preference have a lot of power and that raises other questions. But yeah, ultimately we don't want the AIs to decide for us what is right and wrong.
But so then what is different between Anthropic has a constitution. They write in a whole bunch of values. OpenAI writes in a whole bunch of values. Grok writes in a whole bunch of values. Very different values, but it writes in values, right? What is the difference between that process and what you're proposing?
The difference is they do write a set of values, a set of don'ts, set of dos, but the AI doesn't quite obey those instructions. That's the problem right now. And we see it in a lot of the mishaps of self-harm or misuse and the inability of the AI for example to make the difference between the validated instructions like you should do this or you should not do that and some instructions that are coming from a malicious source maybe through an email or a document and these kinds of problems are serious security issues right now with the AIs that we have.
All right, so I just want to make sure I understand this. So right now the current AI models, they have values, they have instructions and it's not enough, right? We've seen lots of examples where it's not enough. And the idea is we add this kind of smaller model, different model, bolted on supervisory model that has other values or has more formal values embedded and we think that's explicit instructions about what we don't want, for example.
I can see that working right now. It's a wonderful idea. What happens when the base models become a billion times more powerful than they are right now?
