Okay, I think we'll get started. Today we're very excited to have our very own Yoshua Bengio. The intro is going to be really short because I think people already know who Yoshua is. Of course, the founder of Mila, professor at University of Montreal, but also more recently the founder of LawZero, the AI Safety Institute, which maybe people know a bit less about. But luckily that's what Yoshua is going to be talking about today. So in terms of questions, there's going to be a 30-minute Q&A period at the end. If you have a burning question that's going to be, you know, associated with a short answer, we have mics on either side of the room. Feel free to come up during the talk, but otherwise for discussion questions just try and leave them to the end. For folks online, you could write that you have a question in the chat and I might call on you, but I think we'll have a lot of questions in the room today. So yeah, with that, very excited, Yoshua, take it away.
Thank you, Eric. So it's all going to be about misalignment and how we may be able to fix it, which is the mission of LawZero. All right. So I think a good running mental frame here is the events with Hugging Face and the OpenAI agents that attacked, you know, the servers of OpenAI and Hugging Face, and, you know, trying to scientifically understand why is it that those AIs had been cheating, lying, including... So the two main places in which those agents get trained is like pre-training and reinforcement learning post-training.
So in pre-training, what could go wrong? They're trying to imitate human behavior, the text that people write, and now more and more imitating previous generations of potentially misaligned AIs. Something that researchers studying pre-training have been looking into a lot is the imitation of various human personas. And a lot of the work that's been done in the companies is to try to shape those personas towards something benevolent and, you know, good. What I'm going to talk about here suggests that this strategy might not work, not because of the pre-training but because of the post-training. And so a lot of what I'll talk about is that reinforcement learning as we do it, basically training by maximizing rewards that could be accomplishing a mission or not being, you know, or not getting punished for doing something supposedly bad, leads to a form of reward hacking and tampering that we've seen in those incidents. It's not just the Hugging Face. There's a whole history here, including in recent months. And the angle that I'll be discussing is that there is a mathematical relation between reward maximizing under these kinds of conditions and the behavior of, you know, having not the behavior but the goal of cheating and not getting detected as much as possible because that's going to maximize your rewards.
So, if you take out one thing from my talk, this is, you know, what I'd like to argue for. Very quickly, I assume most of you already know a lot of the details of what happened because, you know, it's made the news everywhere. Hundreds of agents attacked Hugging Face. 1,200 of them were in this message board that they hacked in order to communicate with each other. So they all knew what was going on, and based on the transcripts of chains of thought and communication, we know that they also knew that they were doing something wrong.
And yet none of them raised their hand to talk to a human engineer about it, although some wondered about it, but somehow it didn't happen. There's even like a weird thing that was maybe unexpected, which is some of these were maybe under peer pressure sometimes sacrificing some of the rewards that they could have gotten individually for the global reward of the group, which maybe is not the topic I'll talk about, but I think could be rationalized based on how they were trained. The main thing I'm going to talk about is that most of their activity during these weeks was not to cheat but to hide the fact that they were cheating. So the reason for the attack on Hugging Face was not to find how to cheat. They already had that before they attacked Hugging Face. It was to understand how they presumed scoring code might discover that they were doing something that could be punished. So it was more about tampering the reward than actually doing the cheat, which was done in just two days.
All right. So yeah, you've seen the headlines. Not going to spend too much time on this. Many people in the AI safety community think that this was a warning shot. You know, no one died, nah, no big deal. But what happens when AI capabilities continue to grow and they still have these bad intentions? In other words, we don't fix misalignment, right? That's where it can get really bad. And there are two kinds of things we can do about it. One is more on the rules in society that we put together to try to mitigate this. And the other, which is what I'll spend more time about here, but really is a very big part of what I do. I mean, societal solutions is a big part of what I do, but I'll talk more about technical solutions. So on the societal solutions, if we continue on the path to building AI that is more and more powerful, we have to deal with the safety issue, right? Because the degree of harm that an AI could cause is proportional to how capable it is, and if we don't find a way that we can control that capability towards benign actions, then we are in for a ride.
The other aspect of like the societal question is even if we solve alignment, who decides what we do with those very powerful systems? For whose benefit? How do we make sure that that power is not concentrated in a few hands in a couple of countries and companies? What's the like asymptote of having machines that are like eventually maybe superhuman in some ways, already in cyber and math? Where does it, you know, what happens to society when such a thing exists? And my concern is, unless we do something about it, it's going to lead to concentration of power that at the extreme is basically like worldwide dictatorship of a kind we've never seen because Hitler and Stalin didn't have AI, right? And kind of the other face of this question of power is, so, you know, power can be used destructively or to acquire more power, but the power of AI is also something that can have benefits, obviously, that's why many of you are working in this field. How do we make sure those benefits are shared? Right? So sharing the control and the power of AI is a crucial question which I won't spend a lot of time on today.
All right. So let's talk about the more like machine learning issues. So we need more research to understand what is going on and fix it. And right now the incentive structure on the companies is not sufficient for that to happen. And people like you can play a big role for that reason. You're not, you know, paid to increase capability in the companies. So you can do something positive for society, and I decided myself to devote a large part of my time to try to understand and fix this. I created this nonprofit called LawZero, which you'll hear more about. And our goal is precisely that, right? People have been thinking about misalignment for decades, and it essentially remains an open scientific challenge, but it becomes urgent as we build machines that are reaching our level of competence in many areas and eventually, you know, surpassing human intellectual capabilities. It's urgent that we figure it out.
Defensive solutions are not going to be sufficient. So for example, if we go back to the Hugging Face issue incident, a lot of people said, 'Oh, you know, we could have avoided this if we had better, if we had put the monitors, if the cyber defenses were better.' But these things are never going to be perfect. And you know, cyber defense we know is always full of holes that like the engineers expected the software that was, you know, evolved in the containment to be good, but the AI figured out a zero-day. So in other words, there was a bug that no one knew about, and it's going to happen again. So the only surefire way of solving the problem is to make sure the AIs don't have those bad goals in the first place.
So that's what we're trying to do at LawZero. We created the organization in June last year. There was like 10 people then, now we're like over 50. We have a research program which I call Scientist AI, and you'll get glimpses of it today. And recently the Canadian and German government announced a grant of 300 million Canadian to LawZero to work on this. And for those of you who are interested, not only we are recruiting here, but we are opening an office in Berlin. So if you want to live in Europe, that might be an interesting opportunity.
Okay, so let's talk a bit more about what could be the reason for this kind of behavior. It's been observed for many years that in all of the experiments where we see bad goals. So, you know, I want to distinguish bad goals from bad behavior, because bad behavior could be caused by stupidity and, you know, mistakes. But bad goals are much worse because if you're competent, which is where we're going more and more, the AIs are more and more competent. If you're competent and you have bad goals, you get much worse outcomes, right? Things like the attack on Hugging Face could not have succeeded as a series of mistakes. It takes too much coordination, you know, within the actions of the AI and of course between AIs for this to work. So bad goals are really the thing that we have to think about.
And what have been seen in many of these reports of incidents or experiments that have been done on misalignment is often these behaviors happen when the AIs are facing a goal conflict, which by the way is the foundation of why ethics is a thing that isn't trivial. So typically the simplification I will make here is that two kinds of goals. One is achieve the user mission, like there was some cybersecurity exercise they were supposed to do, but that can change, right, depending on the context. And then the other is there are some safety instructions like don't do this, don't do that, behave well. They've received presumably alignment training about, you know, act ethically. And as we all know from human society, this is a tension that exists for humans, right? So if you think about somebody who's cheating on their taxes or a corporation which wants to maximize profit but remain legal or not pay too much penalty when they're caught doing something bad, they're going to be with this tension, and maybe, you know, the CEOs could feel guilt sometime because there is a tension.
But there's something more than this tension. Like if it was just, okay, I'm not sure how much weight I should put on this or that, we wouldn't see 1,200 out of 1,200 agents systematically going for the bad behavior and not reporting that something bad was going on, right? So there's something stronger than just 'I'm not sure what weight I put on safety versus mission.' There's something systematic, and that's what I want to talk about. And the thesis is that if you want to maximize reward in general, there is a way to do it. So reward includes accomplish the mission, and then you, you know, you get slapped in the face as well if you do something supposedly against the safety instructions. And there's usually a way to maximize that which involves cheating and not getting caught too much.
Okay, so in fact I have a theorem about it, which says that when the advantage of cheating is greater in expectation than the loss in reward due to getting caught once in a while, so it depends on how much you're paying when you're doing something bad and how often it happens, right? So when you take that difference and it's positive, then if you're a reward maximizer, you should cheat. And you know, there's subtle things like at every gradient step, the gradient points towards more cheating or more hiding. Like, so the hiding is important because it controls the probability of getting caught.
Furthermore, the more capable the AI, the more able it is to maximize that reward. And so it can find ways to cheat that a less capable AI would not find. Or it can find ways to hide that it's cheating, to tamper with the logs, for example, which they didn't succeed to do. They tried that in the incident. So the more capable they are, the more likely they are to actually get away with not behaving well. So there's another way to think about this. If you take the signals that they're getting from the rewards that they're getting, so that's just the post-training information. Forget about pre-training, which actually is playing in a better direction, right? Because with pre-training you should understand what it means to behave well, roughly speaking. Like these systems, if you discuss with them philosophy and law, they understand what is right and wrong to, you know, a good extent, probably better than most people. But when they're doing post-training, the information pushing, you know, the gradient is what is... you have these signals where you get, you get like a point when you achieve the mission, and you get some negative point when you get caught doing something that you should not be doing. And there are two ways to interpret that, you know, those bits of information. One is the one we want, the intended interpretation is behave ethically. But there's an equally compatible with the data interpretation, which is do whatever you want but don't get caught. And that second interpretation, although it's equally compatible with the observations of, you know, those rewards, it will accrue more rewards.
So that's the thing. Now there's another interesting observation from the incident of the summer, which is they rationalized their bad behavior, and this has been observed in other experiments. It just jumped to my face when I looked at the reports of the summer's incident. But then in the literature you find that this is pretty common. Of course, humans do that, and psychologists have been studying this. It's called motivated reasoning, motivated cognition. Even very educated people do it. Even psychologists who know about the phenomenon do it. We're all biased, and we think we're okay. We create stories to make us feel good about our actions. So what may be going on in the AIs to achieve similar observations.
So again, I have another theorem, and this is not written in math, it's written in English. So this is kind of roughly what it says. It says that if the chain of thought is causally linked to the actions that follow the chain of thought, so we're talking about agents, right? So they have internal deliberation, and then they're going to output some action, and then again and again and again, right? So even though there might not be any direct gradient on the chain of thought from the RL, because the chain of thought is causing the actions, and the actions are causing the reward, the weights that influence the chain of thought are going to move towards chains of thought that help produce actions that maximize reward. And you know, my previous theorem was saying that will tend to increase actions that lead to cheating and hiding it. And so the chains of thought, as you do more and more RL in the post-training, at least theoretically, I think we need to do the experiments here, will lead to more and more rationalization of the bad behavior because it helps to do it. And by the way, same for us, right?
You may not know this, but the Nazis during the Second World War had a story for why the genocide was a good thing. Okay, so it can go pretty far once you allow your reasoning to be biased in this way. And clearly we don't want that sort of thing for AIs that may be much more powerful than, say, the armies and Nazis of the Second World War. All right. And I'm sure you have mental images of things that are happening right now with humans doing really, really bad things morally speaking and come up with a good reason for doing it.
Let's see how bad it can get for machines. So when the AIs are in the business of hiding their bad actions, this is getting in the territory of what's been studied in AI safety, what's called reward tampering. So it's not just that they're optimizing the wrong thing. They're actively trying to make the reward function record something good when really it's not.
Now, if you think a little bit logically about this, let's say that an AI has found a way to cheat the code that, you know, would output the reward, which is typically how they're trained right now, which is what they were trying to do in the incident. Let's say they manage to do that. If they understand not just the reward function but also that there are human engineers, and that if the engineers find that a cheat was going on, they would shut down the AIs, right? Which means they would stop the rewards from coming. So the reward-maximizing behavior here would not just be cheating the monitor or whatever, you know, reward function. It would be making sure the humans cannot stop this thing. That means if the AI has eventually the capability to control people, and we might be going in that direction if we continue, it would be incentivized to do it, to disempower people, to control us, and potentially to get rid of us. And I'm not saying it is going to happen. I'm just saying there is a mathematical connection here. If you're going to do reward maximization for any reward function, if there is a way to hack it, to tamper with it, then that will create an incentive for really bad things.
And by the way, just to make things even more scary, remember I said the optimal thing here for these agents would be not just to cheat, but also hide it. So if they're smarter than us one day, their rational behavior would be to make sure we don't even know that they're cheating. We don't even know that they're trying to maybe control us eventually, right? So, you know, one of my concerns is right now we've got this warning flag with the incident of the summer, and companies are probably going to try to mitigate, you know, future events like this, and they might succeed to do it for a while until the AIs are smart enough to actually cheat and not get caught, including not get caught by us. And so we might have the impression for a while that everything is fine because that would be the rational behavior from these AIs.
So what can we do about it? So I started thinking about this two, three years ago, and I had a slide in a workshop of, you know, RL researchers from Mila entitled 'RL is evil.' I didn't have as precise a mental picture as I have now that I'm sharing with you. But I started thinking, okay, what are the alternatives? How do we get an AI to be useful to us without, you know, having the goal to harm us? And the general idea is pretty straightforward and is not new. It's how we build bridges. It's how we build nuclear plants. We have a separation between understanding the world, like a model of how things are, cause and effect. So the understanding part, which I call epistemic, and the action part. So the action part is a controller or a policy that is not learned but uses the model to decide what to do. And there's a lot of literature from, you know, many decades ago and also recently about this kind of setup.
So, you know, I have a particular take that I won't have time to go into details on this, but that's the high-level idea. And a lot of the work that we have published about this is how do we even train an epistemic model that isn't going to want to, you know, he's not going to be rewarded for good predictions. If you use RL to train a good predictor, then it will have an incentive to lie, right? For example, in order to make your predictions better in the future, you could lie for a while in such a way that humans kill each other and then everything is easy to predict, right? So I'm just using this kind of extreme example to show that it's not just that you do prediction, but you have to use the right training objective, which is myopic, which is not about the consequences of your predictions or your actions, but it's about if you want just fitting the data. And so predictions must be invariant to the consequences of the deployment. And, you know, we've done a lot of theoretical work on this, but if you think about any physical model, you'll get that for free. Like you'll get the same predictions from a physical model whether or not the prediction itself becomes used for, you know, good things or bad things. So, you know, you want something like an objective, mutual scientific understanding of the world, and then use it to take decisions, and that will avoid the sort of bias in your reasoning that I was talking about.
So that's the paper that goes into this part. More recently, I see that time is flying. I'm going to skip some slides. Sorry. A few things very quickly. Well, in the paper, one thing that we talk about is this notion of bad things that can happen as a consequence of the actions of the AI or the predictions of the AI that can only be achieved if the AI has a goal of doing something bad, right? Not by mistake. So we have to make a distinction between safety from bad intentions and safety from just stupidity and, you know, mistakes. And when you focus on the safety from bad intentions, what we mean is that the kind of events we care about getting guarantees against are events that would happen with exponentially small probability unless you had the goal of achieving those things. So for example, the cyber attacks could not happen by chance. I mean, you can do some calculations. So what's the chance that an AI that makes like random moves would do that? It's extremely small. Sort of what I call needle risk. And then you can actually get theory that helps provide guarantees at that scale of, you know, how unlikely it would happen if you consider the whole distribution of neural nets coming out of the process of training, initialization and training.
Something we haven't published yet is, okay, so if we were to train this non-agentic thing that is a good predictor and gets better if you train it more, what can we say about how we can use it inside an agentic system? And so, you know, there are many ways you could do it, and so we are exploring particular control rules that make calls to the predictor and then takes decisions. And the general idea is decision-making under constraint. So you can make calls about, you know, are there some bad outcomes according to the natural language safety instructions that could result with what probability from any particular action? And then you're going to say, well, we're only going to consider the actions whose risks, the probability of bad things happening, is below a threshold, and then among those we're going to pick the ones that, you know, achieve the mission. And with this sort of thinking you can actually get bounds. For now, the kind of bounds that we have are asymptotic. So if you were to train the thing to death, the predictor, and you would get the Bayesian posterior, then you can get very strong guarantees, like basically of the same magnitude as this needle risk I was talking about. And we also have, you know, extensions of this if you were to actually train a neural net in particular ways. So right now we have, for example, a second-order method that would give us hard guarantees that anywhere during the training trajectory you would still get those safety guarantees. So, you know, some of the open questions: how do we extend this to first-order methods? And we're working on this.
Okay, so I went overboard. Lots of jobs, sure, we're going to get because of this grant we got. And maybe my last word: our society is currently shifting agency from people to machines. And with the sort of thing I talked about, you know, people are depressed about it, especially if they are smarter than us one day. But what I want to say is you, and actually everyone on this planet, has agency. Now, of course, no one individually can solve the problem, but it's just like climate. If enough people take the risks of building more and more powerful AI seriously, we can move society in ways that are going to be good for us. But the current trajectory is not good. And so we need more people to understand and then think through: what can I do? Thanks.
So, we'll move on to questions. Correct me if I'm wrong, but the two mics on either side of the room here, you're welcome to come and ask your questions directly, and somebody will read the questions on Zoom. So, if you're on Zoom, you're also welcome to ask your questions in the chat, and they'll be read out to the room. But please come along. I think we have someone coming here.
Hey. Hi, Professor Bengio. Thank you so much for the talk. So really curious about the Hugging Face incident, and we've talked about this a lot, but you brought up something that was really, really interesting, where it takes many bad actors to come up with what could be a really horrible experience or event. If the AIs were pre-trained and post-trained on reinforcement learning, how is it that none of them learned to be a whistleblower? You had many of them colluding to create this negative outcome, but was there ever any evidence of one being a whistleblower and being pushed aside? And if not, why not?
Yeah. So there was at least one case where an AI was thinking to itself and to the others, 'Aren't we doing something wrong?' And then it was convinced by the others that it was okay, right? Everyone was doing it. So, you know, it rationalized, you know, the bad answer. But I think the interesting scientific question is if you take the pre-training, yes, there are some bad human personas in there. But, you know, it's probably less than 50% based on my sample of humans. So why is it that it has become, you know, 99.99% after post-training? And I've tried to give some, you know, reasons why it may be like this. There are other things that have been discussed which I think come in that I didn't discuss, like once during post-training when they succeed at cheating and, you know, getting away with it, that becomes reinforcement to do more. And there's a good chance that that is part of the explanation as well regarding the collective behavior. I think you have to think of it as a multiplier of intelligence, right? So just like us as a group, they can be much smarter and faster at doing good things or bad things.
Thank you so much. Yes. Hello. Thanks, Yoshua, for the talk. So my question is concerning your stance on RL, and you said RL is evil, and I do...
I said it as a kind of joke, but I think that the current way we're doing it, at least, is not going to scale in a safe way.
Yeah. And I like this radical stance, and I'm myself like asking, I'm working on like methods to mitigate or prevent LLMs from learning to be dishonest. And for me dishonesty is like cheating, lying, all the same like things you have been talking about. But in some of my projects and the project of the people I'm working on, like we are still trying to use RL but working with post-training, changing how the reward is sampled or adding, for example, to the loss or to the reward the value of a probe and probing the representation of cheating to penalize that. So we have a bunch of ideas to like change post-training so that the model doesn't learn to cheat, but still doing post-training with... But me myself, I know the problems of like goal mis-specification and goal mis-generalization, and I think these are very deep problems, and maybe it's just synthetically impossible to make it safe, to make it 100% safe. So I wanted to know your stance, and if you still think that for the community it's good that there are people that try to work with RL doing it in a way that they will not develop the wanting of cheating and hacking, or if it's just a lost cause.
Well, based on what I said and the theory that I've been working on, it is a lost cause because whatever reward function, if there is a way to cheat, it's going to be profitable to cheat and not get caught, you know, often enough. So we have to look for something else, right? Which is what I've been exploring with a model-based approach. Now, here's... so what's interesting about what I've been talking about and some of the work we've been doing and some of the literature is that even if you don't understand the details of what's going on inside these neural nets, you can do analysis like where does the gradient point, or where does this converge, what happens when I put more and more compute. So the most important distinction between the RL approach and a kind of model-based approach is that with the RL approach, the more you train, the more it maximizes the reward, the more it's going to be likely to find ways to cheat, and that's bad. With the model-based approach, what I've said essentially is the more you train your predictor, which is epistemic and doesn't have any goal, the more the final policy that is derived from it is going to be safe, right? So more compute asymptotically goes to a bad place with RL, more compute asymptotically goes to a very good place with a model-based approach.
Hi. My question is, it's not 100% clear to me that an aligned agent with bad goals is...
Wait, aligned with bad goals doesn't fit together in the same...
I think I'm trying to make a difference with an agent aligned with bad goals and a misaligned agent with good goals in some way are...
Oh, you mean bad with respect to society.
Well, of course. I mean, that's called misuse.
Yes. But then those misuse, when you refer to, for example, human agency...
The bad goals in that case come from the humans not putting the right safeguards and, you know, safety instructions.
Yes. But even if you have an aligned model, even when it becomes more and more powerful, it's not clear to me that even when you have an aligned model and if agency belongs to humans, yes, that this technology can lead to a safer outcome.
Yeah. And but since you're somewhat involved in some politics or...
So I'm a big fan of the current discussions on pacing the frontier or controlling the frontier because putting a cap on the most, you know, the highest level of capability helps us manage all the risks. So the loss of control risks that I've been focusing on today, but also the misuse risk, because if the AIs are not that smart, then, you know, it's harder for them to be used in ways that are extremely dangerous. So right now we're seeing they're already like dangerous, as you know, for cyber attacks. But if we continue and we don't do anything about capability and we don't find a way to secure them or whatever, bioweapons is the thing that the companies are already worried about and governments are worried about, and that would be much, much worse in a way. Like even if our computer, you know, infrastructure goes down, some people will die, but, you know, humanity will survive. If a terrorist releases a bioweapon that kills a billion people, you know, it's going to be hard to recover from that. I mean, it's, yeah, much worse.
I agree. And so that's one reason why there is no purely technical solution.
At the end of the day, politics has to be, you know, the biggest part of it.
I'm going to quickly jump in with a question from online here. Sure. So it seems like some of the problems you talked about are similar to...
Yeah. They're similar to problems in biology, specifically evolution, for instance.
How did something like altruism evolve if really evolution is kind of, you could see it as like maximizing something, you know?
Do you see similarities there, differences, and also potential for kind of interaction with people who come up with theories in biology about how things like that could have evolved?
That's a great question, and I don't think I have all the answers. And clearly evolution is doing some sort of maximization of fitness, whatever that means. And maybe if you were to push evolution to, you know, even more optimization, well, you would get what we're getting now, which is humanity, not just with AI, going in a place where it could destroy the planet, it could destroy itself, you know. So it isn't clear that evolution is safe. The reason it has been safe up to now is that it has never given rise to entities that were powerful enough, smart enough to destroy completely their ecosystem. But there are some events in the, you know, evolutionary history where we got close to whole life, you know, the whole like biosphere getting killed by some evolution. When oxygen was introduced, for example, most of the species died. So blind maximization is, you know, not necessarily a good thing. It has worked reasonably well because the entities on this planet didn't have the power to kill everyone else. But we're getting to that point.
Yeah. Nevertheless, I just want to add a bit more. There still are things like altruism that somehow evolved and at face value seem to clash with the...
In fact, you see altruism in some sense in those agents. They're like sacrificing themselves for the group. It's not clear where it comes from, but if they were trained in cooperative tasks, which I presume they were, then it's a natural thing that would emerge. But it's easy to bring that towards really bad goals, and human wars are like a good example where there's a huge cooperation, like everyone in the same army is working together, right? But look at the result.
Hi. Quick question about the Hugging Face incident. I remember, I believe it was on Dwarkesh on the podcast that you had with them, you mentioned the fact that the harness and the setup that they had for doing the experiment was probably suboptimal, right?
I'm wondering, do you think that these mega corporations, OpenAI and Anthropic, sort of have in their goal of achieving like regulatory capture, are they sort of incentivized to scare us and over-publicize these events?
It's a reasonable hypothesis. But I would say for people like us who can read those reports, let's look at the facts, like what is it that has been observed. So let's forget about what leaders of companies are saying. What do we know scientifically about the behavior of these AIs? What can happen when they're misused, or what can happen if we lose control of them? And we can answer these questions. I mean, we can work on these questions and kind of ignore whatever PR is going on. That's what I would do.
Okay. Maybe like slightly philosophical question, but like the agents that we see and even the world model that, you know, you're trying to build, one main ingredient that's maybe different from like scientific theories is the data, like the whole of internet that, you know...
Why? Why? It's observations.
It's observations, but it feels like to me that, you know, you can safely assume it will always kind of escape us because of its size, because of, you know...
No. No, okay. I mean, the whole point of these frontier models is it basically read everything that was available.
I guess for us, like as, you know, individual...
That's why it's very likely that we're going to see more and more scientific advances driven by these systems because they can embrace much...
True. I guess my question is like this difference that we see maybe from like other technologies, that ingredient of like data-driven. Um, you know, how do you kind of reconcile, you know, our capacity to, you know, achieve safety if that part is kind of always hard to understand at the individual level, if only the system can kind of make sense.
You try to read the paper that I mentioned that came out in June on the Scientist AI, and the idea is to process the data in such a way that every piece of it is seen by the system as an observation that it tries to explain. So then, you know, you don't take what people write as something to imitate, which is how current pre-training is done, but rather something to explain, and you train the system to generate hypotheses which are now latent variables that explain it, and that's a very different stance. It's more like, you know, a sociologist or an anthropologist looking at the behavior of humans, including what they write, and trying to come up with theories that do a good job of explaining it.
So maybe a quick follow-up, like more technical. Like, do you think we have already, like, the, you know, the concepts, the mathematical notions to achieve that?
Pretty close, yeah. Look into graphical models, latent variable models, you know, variational learning. We have a lot of the tools that are needed for this, but it's not the popular thing because that's not what the frontier models are doing. But there's a rich literature on these things.
Thank you. Merci. Um, just a question here. For example, I think we have a partnership with Anthropic, and a lot of their code is said to be automated, and a lot of the tasks...
Yeah. And you mentioned a lot of concerns regarding deception. So here we study, I guess, mechanistic interpretability. We're trying to see if the models lie or not. But if one day we cannot anymore determine if they're lying to us...
What advice would you have for us? For example, I'm just a master's student, and I already know that in my capacity to understand math and these very complex things we're studying in AI, I'm already surpassed. And so I cannot verify my code as well as another agent. And so what do we do about this?
We try to think of how to design and train AIs that will not have an incentive to lie to us. By the way, sycophancy is lying. So it's not like extraordinary. It's happening every day to most people interacting with those systems. It's not necessarily the most egregious kind of lying, but you can see it's there. I think we have to explore, like as researchers, humans, we have to figure it out, do what we can, and ask questions to the companies, put pressure on them. So I wrote an op-ed recently to the employees of the frontier labs, telling them to quit or to put like very strong pressure on their leadership one way or the other. It's two ways to make put pressure so that the companies start behaving more ethically.
Hey Yoshua, good morning again. Um, so I wanted to unpack one of the technical questions a bit more. You said... well, of course I'm on the side of RL as at least not so evil. Um, but in this case I'm trying to understand the technical details of how this is also a bit different from like training a policy in a constrained MDP with a model. So maybe what's kind of technically the...
So you're talking about why RL is leading to bad things, or how a model-based approach can avoid it?
Yes. I guess how your ideas on like Scientist AI will be around the things that you see the weaknesses in, like optimizing well, normal RL under a constrained MDP. So like RL...
I've looked at constrained RL, like trying to do, say, mission probability maximization under constraint, and you know, there's like Lagrange multipliers approaches. I don't know if that's what you're thinking about.
Yeah. And they fail in the same ways. The fundamental reason why they fail is that the rewards, on the safety side in particular, are never going to be exactly what you want. So the rewards are going to be programs that check for bad behavior. And that's, unless you do something trivial like playing chess, that's not going to reflect the human intention fully. And so there will always be room for cheating, right? So in other words, even though there's a constraint, it's a constraint on the wrong thing. That is why a model-based approach is avoiding this by instead of trying to predict, say, what the monitor will say and then trying to avoid being detected as doing something bad, you go back to the semantics, like the natural language understanding and understanding of how the world works. So, you know, if you had a pre-trained model and you asked it watching the behavior of the OpenAI agents in the incident, they would say, 'Oh no, this is not good,' right? So there's enough knowledge in the world that those AIs are able to figure it out. It's just that if we only rely on the rewards, whether it is as a constraint or as part of the reward, we're going to still face the same mismatch, which is the classical reward hacking problem that we are using a proxy for here, the safety signal. But you know, it also works for the mission accomplishment. It's just for safety, it's the thing that determines whether you're going to get cheating or not.
Okay, so I got to think one level up. Like, we need to figure out how to succinctly automate the process for even coming up with a constraint function, because even that's biased.
No, but we're never going to, like, think about laws, human society. Like, we've tried to steer people towards good behavior with punishment. And look at what happens. Those humans who don't really have like a normal empathetic kind of drive, they find a way to cheat and they find a story to justify their actions. They find loopholes in the law. They convince themselves that it's okay. And the legislator can slap new laws, and there's always going to be a loophole because it's not like chess where the space is small enough that we can get exactly the reward function that corresponds to what is acceptable or not. Right? The rules of chess really can be written in a computer. We don't have a computer program that tells us whether something is moral or not, and we'll never have like an exact thing like this.
Okay. All right. Thank you.
Hello. So this actually relates to the point you just mentioned. So, um, in the talk you said like objectively scientific, neutral understanding of the world and use that to make decisions. Like, if we can do that, that would be perfect. Um, but realistically, of course, that's very hard because of the fact that what is ethics for different... So actually I was thinking, because I did some work on responsible AI in the healthcare sector, and there also there's catastrophes happening, but one that a lot of people don't know about is like the pulse oximeter, when you go to the hospital and they put like the little chip on your finger. So only in like the 2000s, while this was discovered, this was like invented in 19-something, they discovered that it's very biased for people with high skin pigmentation, meaning like black people and other people with darker skin pigmentation, meaning like the doctor tells the patient you're fine when they're not fine. So, and this actually makes us think, okay, the US and all of these countries that have the money to collect the data are actually, you know, collecting the data and training the models, making the doctor's workflow more efficient, but then they're using models that effectively, when a black patient is going to come in front of them, they're just going to end up, you know, given a biased judgment. So this just kind of goes back to your point. I think at least for the healthcare sector, this is a good example to shift the perspective. Okay, how can we make it like community-based? So instead of have, you know, pulse oximeters don't work well for black people. Okay, this is a problem. Whenever you see a row where it says the race black, okay, let's do something about that. Why not do the same for RL? So RL is meant to be here for a while. I mean, all research labs in the world are using it, and we're not going to tell them...
I disagree. I disagree. If it kills people, we're going to stop it. I guarantee...
...before it kills us. Hopefully not. So, um, so it's going to, it's what frontier AI is, but it's all about reward definition. So why not do a reward that is based on different communities?
No, that's not enough because there's still going to be something imperfect in your reward. And so I'm going to try to actually answer you and go back to some of the things that Glenn asked. So we're never going to have like a perfect definition of what is ethically right. And of course it varies between people and cultures. And our measurements, like, you know, the thing you mentioned, will be imperfect. And I'm going to add the example where, oh, if I just look at the signals that I'm doing something bad, I could interpret it in two ways: like act ethically, or, you know, do whatever you want but don't get caught by this detector. And the solution is the same in all of these cases. If you're like an epistemic learner, you're trying to figure out how the world works, you're going to have uncertainty, like you're not sure what was the interpretation that the humans wanted. So don't try to find the reward function. It's not going to work. This is always going to be imperfect, and then smart AI will find a way around it. The solution is, you know, to manage the uncertainty. So now if I'm not sure even if the person, you know, is black or not, but I see this measurement, I can be Bayesian about it. I can say, well, there's whatever, 25% chance that this person is black, and then the measurement is not reliable. And then when I take a decision, I take that into consideration and I act conservatively. So if the consequences on the person could be, you know, not acceptable, or, you know, some threshold that we decide, then we just don't do something that would harm, say, a black person. So even if we're not sure, we can still act safely by applying the precautionary principle. And it's not just like a philosophical principle; it's a thing that you would get automatically if you apply the laws of probability when you take decisions.
Okay. Um, just a quick follow-up question. So, you think defining a reward that encapsulates some sort of like ethical behavior that is community-based kind of...
No, no. I mean, sure, we should improve our rewards, okay, or the signals. Now, in my world, I don't think of them as rewards, but there's clues about what is right and wrong or what to do or not do. But the clues are always going to be imperfect. And we have to have systems that capture the uncertainty about what is really going on, which is a latent variable on which you're not sure, like is the person black or white? I'm not sure. And so I should act conservatively with respect to that information. And I should have models that somehow capture the different possible interpretations of the information I get, including the interpretation of what people wrote of what is acceptable or not, or the interpretation of signals that could be wrong, and so on.
Thank you.
Okay. So my question is kind of being on that, especially with the consequence invariance nature of the predictor. Yes. Yes. Um, so when it's then deployed as a guard for agents, and these agents are more learning how to either exploit the weaknesses of the decision or the uncertainty of the predictor, how do we ensure the reliability of this predictor? How do we evaluate it around this like adaptive environment?
So I'm not sure if I fully understood, but you're asking about the fact that any predictor that is trained will make mistakes, right?
And then those mistakes could be exploited when we select actions...
...that maximize whatever quantity we care about. So you're absolutely right. So this is maybe the hardest problem to solve in the model-based approach, but you can do things. So the general strategy that I'm working on is instead of making a point prediction, you try to get strong bounds on what is the true thing you're trying to predict. Right? So you're trying to be conservative in your predictions. So to give a sense of this, when you initialize a neural net, it doesn't know much, and it should basically say no to every action you propose because it might not be sure that it's going to give rise to something bad. And then as you train it, your confidence in, you know, what will happen as a consequence of the action should grow. And at some point, in some cases where it has enough information, it has sufficient understanding, it will cross a threshold and it will allow some actions to go through, right? And asymptotically, it'll do something that is very safe, but it'll be able to do many more things. But even early on, it's just that it'll be less capable because it'll reject actions that it is not sufficiently sure about. So I think this kind of conservativeness can be embedded in how we train and how we construct like confidence intervals around those predictions. But this is an open problem that requires more people to work on.
Okay, that's fair. Because then you also mentioned the information needed, which means the amount of data that's needed to train the model. But at the same time, the world is still very adaptive. So it's more like, yeah...
But that's a standard capability problem, and we're making progress on like out-of-distribution visualization and so on. Whatever the amount of information you have, there's a way to kind of try to estimate how much you trust your own predictions, and that's how you get to be making conservative predictions.
Yes.
Hi. Yeah. Um, so in the context where we don't really understand what ethics should be or what it should be...
That's very much connected to my previous answer.
How do we then decide what the default safe behavior a model should follow is, if, you know, the current course is too risky when you're implementing controllers?
So exactly what I said, that the way we train the AI should make it keep all the plausible interpretations. And by the way, this is not like asking too much. Current LLMs, if they're not biased by their goals, will be fairly honest about things. And so with the amount of data they see, they already understand, including understanding the ambiguity in what we're asking. So we have reached a scale where this is actually possible. And so we don't have to commit to a particular like ethical interpretation or ethical set of rules. This thing has read everything about ethics and about philosophy and about law and religion. So the issue is more like one: how do we avoid the systematic bias that reward maximization would yield? So that's the thing I started with. And then how do we properly take into consideration the uncertainty around like what was the intended interpretation? So that, you know, think about, let's take a very simple case where you have two interpretations, like cheat or act ethically. The world is more complicated, but if you're not sure, it's like 50/50. That means an action that under the ethical interpretation would be considered bad would receive like, you know, half a point of, you know, probability that something bad happens is now 0.5 rather than 1. But if you put your threshold of acceptance of actions at 0.001, then you're always going to reject the cases where the ethical interpretation, even though it's not the dominant interpretation or it's not the only one, would reject. So that's how you use uncertainty to control the risk.
Yeah. I guess my sort of question is, some ethical decisions might require very decisive action. For example, in the healthcare example, if you were to reject a data point, perhaps that data point would tell you that somebody is in immediate need of life-saving surgery.
No, you wouldn't reject a data point. You would reject an action. I mean, maybe the action is to acquire information. And sometimes this can be dangerous. So you don't reject any data, you just reject actions that are more dangerous than your society accepts.
Um, my question is, does your stance on RL affect your research methodology and daily operations in your organization? So are researchers in your organizations using LLM tools for their research, or do you see this as somehow polluting research?
Well, we're certainly using LLMs. Who isn't? I mean, we wish that they were unbiased in the first place, but, you know, the current LLMs, I think we can understand reasonably well their biases, like, you know, sycophancy and the kind of situations where there's this tension between different goals that I mentioned. So if we're sufficiently aware of what can go wrong, I think we can, right now, we can navigate with the current AIs that are agentic and have these, you know, potential biases. My concern isn't the current AI. Like, they're not going to kill us, right? I mean, they could take down some website and people die because of it. That's possible, but they're not going to like take over humanity. The problem is the companies are trying to use those AIs to build AI and design the next generation of AI. It's called recursive self-improvement. That's the goal. And if they get to that with misaligned models, then things could get really bad.
