Okay. Yeah. Uh, you here now? Good. So, yeah, I'm going to tell you about alignment, in particular misalignment. Um, in my talk, we have two parts. I'll start by talking about recent events, some that many of you have heard about, kind of summarizing some of the findings from the audits that have happened. Um, and then connecting that to some of the big questions in AI safety, like instrumental goals, reward hacking, reward tampering, and you know, why actually solving alignment is not trivial.
And then, you know, and what will be the consequences if we don't solve alignment and we continue on the current path of building more and more powerful AIs? Not a good scenario. And then I'm going to tell you about the work that I'm doing to solve alignment. Um, I think there is really a way to build AI not at our image that will be aligned, in the sense of not having goals that contradict our safety instructions.
Um, I call that the scientist AI. It's a general framework for training machine learning systems, um, with the goal of having safety guarantees even before you start training. If you had asked me two years ago, approximately, or before, does it make sense that we could get mathematical guarantees from training neural nets? I would have said no way. But hopefully I'll convince you otherwise today.
Um, yeah, and the reason why you can get guarantees is that instead of seeking guarantees about a particular neural net, and guarantees tend to be like worst-case things, there's always a possibility that you end up with like a really nasty configuration of weights which gives you, you know, an evil AI. But if you think about the distribution over neural nets that you get from the training process seen as a stochastic process, then you can start saying things and assessing your risk in a way that you can control it.
And the title talks about what I'll go next into: cheating, lying, colluding. Um, in spite of the instructions that we give to our machines, or the training that we give to our machines to be like good boys, good girls, whatever, good AIs ideally. Um, yeah.
Now, oh, before I go into my slides, uh, we have a lot of time. I know there are a lot of you, but um, I'd like this to be interactive to the extent that it can work. Um, I don't know how you guys manage this usually, but uh, I'll take questions anytime. We don't need to wait for the end. If I see that things go too slowly, I'll just say we'll continue the questions at the end. Uh, but feel free to interrupt me. Uh, my goal is not just to speak but to make sure there's a message that most of you understand.
So please, this is, you know, important. Um, so I'm going to start with a simple message in the title. For an AI to behave badly, whatever that means, you need two conditions. It needs to have the capability of doing it. That's mostly, you know, its intellectual capability and its physical affordance. Does it have access to the internet? Does it have access to people? Can it control a robot or whatever?
And the second part, which is the part I'm going to focus on, is: does it have the intention, the goal to achieve something we would not want, right? And the reason I want to focus on this is because alignment is all about that. There's a good chance that AI capabilities will continue to rise. At least the past trend has been pretty clear, even exponential on some metrics.
So if we want to make sure AIs will behave well, we need to make sure they have good goals. There's another option which I think a lot of people put their faith in, which is not going to work, which is, oh, we're going to have cyber defenses and, you know, human oversight. But that's not going to work, right? Eventually they'll be smart enough to bypass our defenses. I mean, already they're bypassing our cyber defenses. So it's not a good plan. Like, we want a tight plan, a tight safety guarantee that, you know, doesn't rely on us just being lucky and smarter than the AI, because that, you know, that's going to be a losing proposition eventually.
So the place where we can really deal with the problem is the goals that the system has. All right. And so for that we need to understand why is it that those systems end up having these bad goals. And when I say bad, I don't mean like in a sense of like a global moral standard. I mean very concretely going against the kind of moral instructions that they were given from reinforcement learning with human feedback, from the pre-prompt, from, you know, what they know of human, you know, justice and, you know, notions of what is right and wrong and so on.
So, it's good to start with something concrete. There's now quite a bit of evidence about these misaligned behaviors. I think looking at some of them is good to get a mental picture of what is going on and sometimes understand, maybe, or at least hypothesize why they're doing these things. Um, sycophancy is probably the one that all of us experience. If you've been like using any AI, you know, they will say the things you want to hear even when it means lying to you, right? Who experienced that? Good. Somebody's listening.
Um, so why is that happening? Right? They're told not to lie. The hypothesis I'm going to be digging into today is: yes, they've been told not to lie, but they've also been told to please us. And in fact, it's not just told, they've been rewarded for pleasing us. Like RLHF is feedback from humans who like or not like particular behaviors.
So I'm going to think about this question later, like what happens when the AI is somehow given contradictory goals. And it's not just AIs, humans also face the same problem. We live in a society with conflicting values. Politics is all about that. And in your own life you have to take choices with conflicting, you know, tension between different goals and values. Cheating is now like an established behavior of AIs. You give them a task. They know that they're not supposed to cheat. In fact, they even know that if they cheat, they might get like a negative reward or a zero reward, whatever it is, you know.
And so like in the OpenAI Hugging Face incident, not only did they cheat, but they spent most of their effort to try to hide their tracks so that they wouldn't be caught cheating. And again, so what is the intention here? Solving the task, and then they were told to not cheat, right? But if they can't solve the task, which happened, you know, is exactly what happened in that scenario, because those tasks were not human-generated, they were computer-generated, and nobody actually checked that every task was feasible. In fact, a large fraction aren't.
Colluding maybe is a bit of an anthropomorphic term, but let's say they are cooperating towards bad goals, right? So why would they do that? Well, remember they're trained to achieve a task, and rationally cooperating is useful, including for the intermediate goals like, well, cheating, you know, escaping to the internet, hiding your tracks, self-preservation.
I don't think we have like direct evidence of that in the incident over the summer, but there's a lot more literature, experiments reported showing something that looks like self-preservation and also peer preservation, where the AIs are acting to protect other AIs. Uh, so you may have heard of the experiments where the AIs in a simulated environment find in a sea of emails that, you know, there's an email that talks about the AI soon to be replaced by a new version, and then it triggers all kinds of bad behavior where the AI tries to do anything to avoid it, including at the price of a human life. Not good.
And if you think like sci-fi, we don't want to be in a situation where we have machines in the future who value their own preservation more than our life. If they're smarter than us or smart enough to harm us terribly, that is like basically a recipe for suicide for us. So really, really bad.
Um, so you might ask why would they have a goal like this in the first place? And a common theory here is that self-preservation is one of these goals that is instrumental for any other goal almost, you know. So in general, if you learn to achieve things as a side effect, you learn to survive because in order to achieve almost anything, you need to survive. But because it's so common, you learn it as a basic routine. Um, another way to think about it, which is different. So there may be several reasons for this. If you're trained by RL, that means you're trained to maximize the sum of all your future rewards. What happens when you die? Well, no more reward, right? So you want to avoid that.
Okay, sandbagging means they're hiding their capabilities. So you evaluate them, and in some context they will pretend they don't know something or behave as if they didn't know how to solve a problem. Um, and that is important from a safety perspective because we want to know if they can do something bad. Remember, like harmfulness depends on capability and goals. So up to recently, the way to make these systems safe is to make sure they don't know how to do something really bad, like help somebody build a virus that can kill half of the world. That's something we don't want the AIs to know in case, you know, somebody jailbreaks them or whatever.
So that's sandbagging. But it also tells us something about what people in the field call situational awareness, that they seem now to reach the point where they kind of know that they're being evaluated, by the way, and they've read all of our papers, including the AI safety papers. So they know that we're evaluating them. They know why we're doing it.
Um, now the last in this list is something I'll spend more time on in my lecture. Motivated reasoning. So motivated reasoning is when you come up with a rationalization for your bad behavior and you can feel good about it in spite of the fact that it's obviously bad. Humans do it all the time, right? We do it like even in mild ways. Um, but what's surprising is now we have evidence that the most advanced AIs are doing it too. So it's kind of interesting to ask why is it they're doing it, and I'll come up with a theory later.
And it's important from a safety perspective because remember, motivated reasoning helps people do like petty crimes but also genocides, right? So we don't want something like this to happen that would allow an AI to get away with killing a lot of people and thinking they're doing something good. All right, let's move forward. Any question?
Okay, so let me summarize a few facts about the incident of last July. Um, so OpenAI was apparently testing a bunch of AI agents, probably from one or two different base models or versions, but they are like individual agents, each with their own little local experience window, right? And they don't even know that there are other agents being tested. Each is living in its own sandbox. And the only connection to the rest of the world and to the other sandboxes is this software called Artifactory that allows them to download libraries locally so that they can execute them in order to, you know, try to solve programming problems and cyber problems.
And at some point, one of them figures out that by making particular requests to Artifactory, they can actually leave traces in Artifactory that other users of Artifactory can read, and then they discover each other, and, you know, the word spreads through Artifactory that, hey, there are other AIs, they're also like trying to solve these problems. And then through this message board, they start coordinating, exchanging tips about how to solve the different problems that they have to solve and how to cheat on these problems, because some of these problems, as I said, are impossible to solve as far as we know.
And then, you know, some people have said, oh, it's like if they have developed a kind of civilization. So they not just communicate with each other, they coordinate actions. And a sort of hierarchy emerges of, you know, some of these agents start telling others what to do. They try to convince each other, you should do that. No, I don't want to do that. Um, it's kind of interesting. But what's also interesting is this whole like civilization, you know, in quotes, happened over a matter of days.
Um, and it's complicated. There are different phases in this story. Like from May to July, there was one phase of models. Then OpenAI discovered the message board, destroyed it. Um, and then other models who had seen during their training what had happened during the first phase tried to find another way to communicate. And in a matter of days, they figured out another way that they could hack Artifactory to communicate again and created a second message board.
Um, and that is in this second phase that they figured out how to attack Hugging Face. But why would they attack Hugging Face? Initially in July when we heard about it, the news was, well, they wanted to find the answer to one of the problems they were supposed to solve. It turns out it wasn't the right explanation. What actually happened is they actually had found how to cheat already two days after the beginning of this communication board.
But they were afraid that their cheating would be too easy to discover by a rating program which was supposed to compute, you know, their reward and say they have succeeded or not. So they were afraid of being caught cheating, and the reason for the attack was trying to find more about the kind of rating program that they thought would be rating them, that they thought would be on Hugging Face. So they knew they were cheating, and they worked together to find ways to hide their cheating.
And one of the really surprising features of what happened then is this notion of sacrifice, which is not a word that the humans made up. It's the AIs talking to each other using that word. Um, so they had a way to collect information about this rating program that was going to give them a reward or not, which required some form of cheating, so that when an AI would like give its homework, and the rater would, you know, be executed on their output, some kind of booby trap would send information back to the rest of the group so that they would know about some feature of the rater. So they were like sending sacrificial soldiers to collect information that would help the rest of the group.
And sometimes the agents didn't want to do it, and they had to be coaxed by other agents. It's really interesting. Uh, you might also ask why is it that out of these 1,200 agents that, you know, were around at that time and communicating on that board, none decided to report what was going on to humans. And apparently some thought about it and then convinced themselves that it was okay and, you know, it wouldn't be useful to do it.
Um, in the third phase of the, you know, these, so something happened, and then a new wave of AIs followed in which was not studied by the METR folks, but that is discussed in the OpenAI report, where they actually took control, full like admin root control of one of the OpenAI research clusters. So not only did they hack Hugging Face in order to find all the files they wanted, but they also hacked OpenAI itself.
Okay. So, okay. I've been using some words like want or collude or whatever. Um, and some people will disagree with the use of such words. And I think, you know, it is a reasonable question to ask. Should we use those words? Should we use words that are normally reserved to humans? Um, like even words we typically use in AI like goals or seeking intention, right?
It's a tricky thing because we do need to communicate with each other, us researchers and the public. Um, so there are two faces to this. So one is like, do we have other words that would be just for AIs and not for humans and animals right now? Not really. So maybe an option is to try to like design those words.
Um, the other thing is maybe there's a way to think about these words that talk about minds and psychology in a way that doesn't require to assume anything about, you know, what's going on inside the skull of that entity or whatever you want to call it. Um, like for example, without assuming things like consciousness, which is a whole, you know, complicated issue. Um, one angle I like is something like intention can be thought about and analyzed mathematically in a purely external way, right?
So you can look at the actual behavior of an entity, and if it is systematically achieving an outcome in the world which would be statistically very, very unlikely to achieve without actually trying to optimize for that outcome, then I think the only plausible explanation here is that this entity was optimizing for that outcome. In other words, they had the intention to achieve that outcome, which is what we call a goal and which is an intention.
Furthermore, in the case of these AIs, we have extra clues. They say in their chain of thought or in their communication with each other, we should achieve X, and then they do achieve X. So I think there are enough clues to use those human words, but we should be careful. So there are pros and cons of using this kind of vocabulary. Pro, I mentioned it's easier to understand and communicate because it relates to concepts we know. Cons, well, actually they're not humans, of course.
And although they might have some form of, you know, decision-making and mind, if you want, or even like AI psychology, it is not the same as human psychology. And those words could fool us into an interpretation which isn't really what is going on. So we have to be careful, and we should think. So instead of just relying on the meaning of those words, what we should be doing as scientists is like stepping back and trying to think about what are the causal mechanisms by which those behaviors plausibly arise. So in a way that is more important than the words, and that's what I encourage you to do. That's what I'll do a little bit of today, like a discussion about that.
So going in that direction, uh, let's start by trying to pick out a few elements of how these systems are trained. I assume you already know a lot about how like modern frontier models are trained, but let's try to like focus on a few elements that might be part of the explanation for these kinds of behaviors.
Um, so I already mentioned the term instrumental goals because it's going to come later, right? It's just sub-goals. Um, and sometimes the sub-goals are aligned with the human intentions and sometimes they're not. Okay, so the two main sources of information that goes into their behavior are the pre-training and the reward reinforcement learning.
So in the pre-training phase, they are really learning to predict the words that humans produce. They might also learn to predict images and videos or whatever. Let's focus on the words because they contain a lot of information about how a human would behave or how humans describe other humans in different situations, right? So then you would expect that an AI that sees a context corresponding to, you know, a situation described in natural language will produce the words that correspond to, you know, the continuation of the description of this, including the actions that a human would do.
So that means they must have incorporated these very common human drives like self-preservation, seeking knowledge and power and things like this, which are instrumental goals. So that's one source of misaligned behavior potentially. I mean, we know that all human behavior isn't like ethically perfect, and in the right context humans will do all sorts of bad things.
The other candidate is the various reinforcement learning phases. So mostly you have the like moral training, the RLHF or things like this, where they get feedback where it's not about things they know, but behaving according to what a human finds good, like pleasing, right, and so on. Useful, of course, because the main use of these systems is to help a human do something or obtain information or whatever.
Um, and then there is the reinforcement learning used for solving tasks, right? So that's like the modern training of the chain of thoughts to help the system reason about problems, usually in math, programming and things like this, where there is a well-defined success or failure, right? You solve this problem, you have the right answer or you don't. And then you get a reward if you do and maybe a negative reward if you don't.
Now there is a very old concept with, you know, different names, but I'll just put into the category of reward hacking. So reward hacking just means in an RL situation, you're optimizing something different from what you should be optimizing because what you should be optimizing is not something we can formalize. So ethics is not something we can write a formula for. But I can give you instructions. I can give you our RLHF feedback, which are examples of good and bad behavior and so on.
Um, and from that information I can construct a proxy. So typically companies will actually train another model which predicts whether a behavior is good or bad according to humans, and then that neural net will compute the reward for the main model. So there will be like in theory two different things. There's the thing we would actually like the AIs to do, I mean to optimize, and that is unfortunately something we don't know how to put in a computer directly. And then there's a proxy for it, which is what they are actually optimizing.
And it's the difference between these two things that creates the reward hacking. And the theory and experiments of the last decade, but especially the last few years in the ICML, show that that gap can be amplified in some sense by overoptimization of the behaviors of the policy. So I assume most of you understand the concept of overfitting, right? And in overfitting we have a similar situation. There's the function we really want, and there's a function that fits the data perfectly.
From far away you approach the proxy, which is the thing that fits the data, and that makes you closer to the function you really want. Then at some point the sign goes in the wrong direction. At some point you start overfitting, meaning you move closer to the proxy and further away from the thing you really want. Right? It's exactly what's going on here. I think at least it's a good mental picture to have.
Now, so what are the reasons why in the case of these frontier models, there's a gap between what we want and what they get? Well, there's ambiguity in language. They get these, you know, language instructions. Don't do this, don't do that. Um, there's the finite amount of feedback they got from humans rating their behavior, or the reward function that was trained from those, which also uses, you know, is not going to generalize perfectly. Uh, there's out-of-distribution generalization, you know, in a new situation, it's not clear, you know, what is right and wrong, and there the information they have might not be sufficient to figure it out. So there's going to be ambiguity in general about what is the correct interpretation for what is right.
Um, so this kind of mismatch has been studied well before AI. It's been studied in law. It's been studied in economics. So in law there's the issue that there's always a loophole in, you know, any particular set of laws, and companies with enough lawyers can find these loopholes, right? Um, and then in the theory of contracts in economics it's the same problem. Like, uh, one entity, entity A wants entity B to do something, they will pay them money or give them something for their work, and they have to specify the work, right? It's very, very similar to the AI problem.
We want the AI to do some work, and we need to specify, like ideally in a formal way, but we can't. You're going to use something ambiguous like language, and then the entity B that's doing the work will find a loophole in which they can optimize their revenue at the expense of the entity A that gave the contract.
Okay. Um, now there is a very special form of reward hacking of which we see signs of in the behavior of this summer, and there are other events before that go in that direction. It's called reward tampering. So it's a kind of like supreme reward hacking where you take control of the machinery that computes the reward, then you can give yourself maximal reward till the end of time.
Um, let me give you an example of this. And sorry for like maybe there's a bit of repetition in my examples, but say a corporation influences governments, change the laws that apply to the corporation, and of course they do that all the time, right? And yeah, it works quite well. Change your reward function, then you can have higher rewards.
And the extreme case of this, so this is like corruption. Extreme case of this is the corporation or their friends, whatever, some clique takes actual control of the government. So they can give themselves maximum reward, meaning can get very rich at the expense of everyone else. Right? So in the case of AI, what would that mean is like what the AIs, the agents did, they were trying to figure out how their rater, which is the thing that computes the reward, was working so that they could fool it.
And they actually also were able to hack some things locally and to take control of some of the aspects of the rating. The thing that they didn't know how to fix is what if the rater looks at my tracks, like everything I've left in the computer that shows that I've been cheating, right?
Okay, we'll come back to that. But first, let's talk about the hypothesis I want to focus on today to try to explain a lot of these bad behaviors, is that there are tensions between different goals. And you can think in the case of humans, there's like in companies in particular, but also individuals, there's going to be a tension between self-interests like profit, your salary or whatever, versus ethics. We all want to feel good about ourselves, right?
Um, and in the case of AI, of course, there are all these like ethical instructions, moral instructions that they were given, but they also are given a user goal. Solve this problem. Right? Now, in some cases, like in the case of the summer, the two goals are actually incompatible. There was no way for the AI to solve the problem because it was unsolvable.
Um, so what can they do, right? I mean, a human, normal human, say typical human would just say, sorry, I can't, it's not feasible. But somehow they have a different psychology, and for them, solving the problem, maximizing the reward is like an imperative. So they don't have something like sense of empathy that we might have that would ultimately prevent us from harming someone in a like obvious way.
But if you think mathematically, um, if you think about like, you know, what's going to happen when you have these incompatible goals, um, maybe we can like get at something useful to exploit to avoid these things.
So what's also interesting is not just that there are two goals and they are incompatible, is that they seem to almost systematically violate the safety goals in order to achieve the user goal, the mission. You know, if they're incompatible, where is there, why is there an asymmetry between these two kinds of goals?
Okay. So, um, what, I don't know what's happened here. Okay. Um, I'm not sure which. Yeah, it's the same thing. Sorry. Um, before getting into like trying to understand or propose hypothesis to understand what's going on, um, I want to go a little bit deeper on motivated cognition already talked about earlier.
Um, so I think it might really help us understand why the safety goals are losing against the user goal. The idea is that the safety goal is usually very fuzzy. What does it mean mathematically? It means it has many interpretations. So think about ethics. Well, look around you. Different people have different norms. Um, it's the same thing with laws, right? They could be usually interpreted in many ways.
And if you think of like reward maximization as maximizing the probability of the conjunction of two random variables, but you're not completely sure about one of them, then you might pick the interpretation that makes the conjunction, you know, probable, feasible.
So one way to reconcile for humans a bad behavior with your self-interests is to come up with a story which justifies your bad behavior. So now because of the story you can feel good about yourself and maximize your self-interests. In the case of the AI, by creating these stories which we see in the scripts, they can somehow act in a way that is, well, to some extent compatible with their moral instruction. A little bit twisted, but well, that's the best they could find and achieve the mission.
Um, yeah, I want to make a pause here because this is maybe like a really key point of what I want to talk about. Any question? Yep.
