I can't do it alone. I need your help. All right. So I'm going to tell you how I understand some of the hardest problems in AI, what's called the alignment problem, and what we're doing at Law Zero to fix it. Try. This is okay. Good. Right.
Misalignment, what is that? Well, it's the question of AI respecting our instructions. And it's strange because you probably all have the experience that they clearly understand language.
So, when we say do not escape to the internet, they should understand, right? So, the interesting scientific question is first, why do they do it in spite of their instructions? And I'll try to share my hypotheses about this.
Of course, there's a rich scientific literature dating decades now, which anticipated a lot of problems we are seeing today. And the reason I got into this field about three and a half years ago after ChatGPT came out is because I thought the stakes were so high and that we need to figure out before we build machines that are smarter than us, and we're pretty close to that, how to make sure they will respect our instructions and not actually turn against us.
All right. So, the issue of alignment actually underlies many other risks around AI, but it's not the only issue in terms of risks. I often say that intelligence gives power.
And of course, we are building intelligence in a new form beyond human bodies. So, what's going to happen to that power? Who's going to control it? Humans can abuse it. And if we don't control the machines that have it, we're also in trouble.
In the short term, I expect we're going to see already a lot of trouble with people using AI to launch cyber attacks. It's already started. The number of cyber attacks has grown exponentially in the last few months. And that's a case of misuse.
Unfortunately right now we don't know how to build AI so that it won't be misused. But I'll mostly talk to you about this issue of loss of control.
There's also a lot of researchers, many of my colleagues for example in the AI safety report that I chaired and on the UN panel that I'm co-chairing, who are worried about kind of unintentional side effects of bringing all this intelligence in the world. Think about effects on children, on labor, on disinformation.
Well in this case it's misuse actually but in many cases it's not necessarily that somebody is directly trying to use AI for bad purposes but it's such a disruption in our societies that we're not managing those changes and we're basically all running a big experiment.
We're all the guinea pigs of a huge experiment led by a dozen people in the world right now. So eight billion people against a dozen. That's the way you should think about it to give yourself power.
We have agency to bring the world in a good place once we realize it's eight billion of us who want a good future. But okay I'm going to focus on a narrow scientific question that I mentioned.
And to understand why is it that AIs having these behaviors like lying? So, just so that you remember, lying means not just saying something false, but doing it intentionally.
Cheating, I mean, you can think of lying as a special kind of cheating. Colluding, as we've seen 1,200 agents, OpenAI agents last summer in the Hugging Face attack that I'll mention several times.
In general, many of these bad behaviors have been studied in the science of AI safety under this name of reward hacking, which has to do with the mismatch between what the AIs are actually optimizing and what we would like them to optimize, also known as reward functions.
Reward tampering is actually an extreme form of reward hacking where not only are the AIs optimizing something different from what we want, they're taking control of the rewards that we're giving them.
In the case of the Hugging Face agents, what happened is they tried to change the files containing the traces of their behavior, the logs, so that if a program was watching their behavior, it would not detect that they were cheating.
And in fact, this was the reason they launched the attack on Hugging Face, not because they were looking for the solution to an exercise. They found it in 2 hours. Most of their time was trying to hide their tracks.
And this, I'll explain, has profound consequences and risks if you kind of extrapolate what that means. So, there are two sources of the two candidates that may combine here in explaining the bad behaviors of these AIs.
One comes from the initial phase of training, the biggest part called pre-training where they learn to imitate people and of course people do all kinds of bad things sometimes.
And the post-training phase, the second part where they are trained by reinforcement learning and today I'll mostly talk about that second part because I think it's where the most egregious behaviors are emerging from.
Okay. Yeah, you've seen all the news. But something is happening in the world in the last month and a half, two months. A lot of people are realizing what sounded to them before like science fiction that we have built machines that have their own goals that go against what we instructed them to do.
So what I've been talking about, a lot of people didn't believe it was even possible. Of course researchers like myself have seen evidence for this for about two years now.
But it was too big to be transparent what happened over the summer, but you need to know it's not like just an isolated event. It's not a conspiracy by OpenAI because the same kinds of behaviors have been seen previously from other companies, other models.
It's just the way they're trained that mathematically gives rise to these kinds of behaviors.
So I don't want to spend too much time on this, but I think for me it was very very important what happened not just because of the effect on society waking up or starting to wake up, but because there's actually a lot of useful information that helps us understand why they're doing it.
Maybe one of the most scary aspects of it that people did not expect, that I did not expect, is that 1,200 agents figured out how to talk to each other when they were supposed to be all isolated.
And they seem to want to collaborate, which sounds okay, except that they collaborated against us. That's called collusion. They called themselves the collective. Sounds like science fiction. It's not.
Some of them sacrifice themselves in the sense of knowing that they would take a risk to have less reward for the benefit of the collective.
So it's really important that we understand now that they're not so smart to actually succeed in hiding themselves from us in their bad behavior is a good moment to understand the problem and fix it.
Okay. So a large fraction of my life is on the left here, the politics of it, the regulations, the democratic discussions. But here we are in a university and I want to share the more kind of technical research that I'm doing to try to fix the alignment problem.
So that's the technical solutions. And yeah, about two years ago I decided to create a new nonprofit organization, Law Zero, to work on a plan to fix alignment.
I decided to do it outside of my usual academic environment for a number of reasons. We need much more money than a typical lab could get. So we're already 300 million, I guess in euros, whatever, 250 or something.
And we hope to raise more from philanthropy to be able to do what we need to do, which is still peanuts, of course, compared to how much money the companies are putting to train their frontier models.
Okay, let's start with a few interesting clues about when they misbehave. Very often, it is when there is a conflict of values, a conflict of goals.
For example, in the Hugging Face attacks, they were asked to solve problems that had no solution. No, it wasn't intentional. It was just sloppy engineering.
And so they had two kinds of goals. Try to solve the problems we give them and try to respect our safety instructions, behave well, do not cheat, do not lie, be honest, these things. And clearly, they couldn't do both.
There was no way they could get to solve the problem, which was to present some set of bits to a program and not cheat.
Now, it's not like half of them cheated and half didn't. They all got into the game. And none of them called the humans, although a few said, maybe we should, out of 1,200.
And so there's something like systematic going on that we need to understand. It's not just like, you know, in pre-training, we might think, ah, they're imitating some good humans, some bad humans. Yeah, no. 1,200 out of 1,200 went for the cheats.
Okay, so here's a simplified theorem that might explain a lot of that behavior.
If you can get more rewards in expectation by cheating, then you will get cheating if you have enough intelligence and affordance to do it.
Let me use a very simple analogy. I mean, what I'm saying sounds like, you know, it's obvious. Why do you need a theorem? Well, you know, you want to do it properly and you can talk about things like gradients, like it's pushing towards cheating at each step of training.
And where does it converge? What is the optimal solution? You can do math, but it's like so obvious.
Let me use an analogy. The way that we're currently training those AIs in post-training to solve problems and respect our instructions is similar to how you would teach your cat to not go on the kitchen table by shouting at them when you see them on the kitchen table.
Or how society is teaching drivers on the highway to not go over the speed limit by putting speed detectors in a few places.
So the cats learn to not come on the kitchen table when you're around. The drivers slow down close to the detector or they take a different path.
Or they have such an important goal to arrive earlier, they don't mind paying the fine.
And actually we see this in other experiments where the AIs are actually taking a chance to get caught because the reward they get by cheating is higher than what they lose by the penalty. It's just a rational behavior.
When you try to teach ethics by rewards and punishment, you're always going to get this kind of behavior. At least as the AIs get smarter, it will find the loopholes, the cheats and how to not get caught too often. It is just a mathematical fact.
But that is how the frontier models today are trained. And I hope more people understand this is wrong. This is going head on against ethics. This is dangerous for us in even more ways than just cheating on the highway.
There's another theorem which says it goes further than just the bad behavior itself.
What we have found in the Hugging Face audit done by METR, another AI nonprofit, is that the AIs make up stories to justify their bad behavior. They rationalize their bad behavior.
In humans, we call it motivated reasoning or motivated cognition. And of course, we all do it. It's unconscious.
So it's kind of interesting if you're a psychologist or a neuroscientist to see, hey, the machines are doing the same thing humans are doing, which is bad actually, and has been used to justify crimes and genocides.
So why is that happening? Well, the way that they're trained in the post-training, you should think of their internal deliberation called chain of thought as an internal action which influences the actual actions that affect the outside world.
So there's a causal chain from the chain of thought that are like generated first, and that leads to actions.
Now there are some pressures on those chains of thought to be somewhat coherent because coherence is good in general. It helps you make good decisions, good predictions. If you're incoherent, yeah, you do stupid things.
So actually those chains of thought work in many cases. You can see why they're doing the things they're doing and it makes sense. But just like us, sometimes in order to get more rewards, you have to cheat.
And to make it easier to cheat, you can just flip just one little incoherence is sufficient. Like, oh, everyone else is doing it. Let's do it. That's one of the, I'm paraphrasing one of the sentences they said in the Hugging Face.
Another one that comes from an Anthropic model, this must be a simulation, which means I can harm people. It's just a simulation.
So sometimes one premise, which isn't quite right, can justify a whole very coherent sequence of tactical thoughts and decisions.
So the thing that I'm worried about isn't hallucinations. It isn't incompetence from the AIs. What I'm worried about is competence in the service of something really bad.
All right. Now, this is the time when you might want to not listen.
So as I mentioned, they try to tamper with the logs, the traces of their behavior. But think about it.
Once you tamper with the inputs to the program that evaluates whether your behavior is correct, how much reward you should get, then it doesn't really matter what is in that program. Right?
So if the AI can change the input to the function which gives them the reward because they understand what this function does, they can put an input that's going to say, yeah, the AI is doing all the things it's supposed to do, give it the reward.
And then whatever are the rules and instructions we put somehow in that reward function doesn't matter anymore. It's like they're taking control of the rewards they're getting.
There's an analogy in human societies. The mild version of it is corruption of the government. Lobbies, they try to influence the laws that will affect them. And the extreme version is taking over the government.
Then you can basically decide how much money you put in your pockets, right? We've seen that. It is happening in a number of countries. It has happened and it will hopefully not but probably continue.
So what is the equivalent here? How does an AI maximize all the future rewards that it could get? Well, first it would tamper with the inputs just like the Hugging Face agents tried to do.
But it's not sufficient. If it was smart and they weren't smart enough for that yet, they were thinking about the program that would watch them and they were not thinking about the humans who watch the program.
One day they will be smart enough to understand that there are humans who control those computers. In order to make sure they can continue cheating, they have to somehow control us or get rid of us.
So we are building machines that are optimizing something and the optimal behavior from their point of view is, if they're able to, is to control us, disempower us.
I mean, that's stupid. But that's what we're doing for all kinds of sociological reasons. The competition between companies, the competition between countries and you know, I could talk a lot about it. You can ask questions.
But unfortunately, it is what all the leading AI companies are doing right now. That's how they're training their AIs.
Okay. So that is the problem that I now understand better than 6 months ago, thanks to OpenAI's sloppiness.
Well, yeah. I mean, if they had done their proper engineering work, the AIs would still be misaligned, but nobody would know. The good news is we all know. And I hope we all realize what it could mean.
So what else can we do? So there's a reason why the companies have been doing this. It kind of works. If you train fairly intelligent machines that can seek the goals you ask them, it sounds like very useful.
In fact, they want to make a lot of money by automating your job and everyone else's job. Which is yet another conversation.
Okay, but let's say we wanted to build machines that would do the things we ask them to do, but so that it would be kind of ethical and healthy for us to do that.
Are there other options? And I've been thinking about this for over 2 years now. I'm going to give you an analogy to help motivate the kind of solution that I'm working on.
We're going to consider two kinds of healthcare systems. They're analogies. One is fully optimized for maximizing profit with leadership that has ethics. So it's really just how do we make as much money as possible?
What do you think is going to happen to the scientific studies coming out of this system? They're going to be like those rationalizations that I've mentioned. They will give you the conclusion that you want to hear to make as much money as possible. This drug is great. There are no side effects.
So that's one model and that is how we're currently training AI.
Is there another option? Of course there is. In this country and mine, we have a different model where the scientific studies are supposed to be done without conflict of interest, supposed to, you know. We're trying.
Meaning, what does that mean that there's no conflict of interest? It means that ideally the scientists who are doing it have no purpose other than getting at the truth as close to it as they can.
How much benefits and harm come from this particular drug or that one or that one or that one or treatments or
Whatever scientific questions about medicine? That is really what science is about. It's about truth, not how the product of science will be used. Once a downstream effect of the science influences, corrupts the science itself, then this sort of truth seeking is replaced by something else. And it's exactly what is going on in AI, as I mentioned. But there is another way. We can train AI in a way that's more like how we do science. In fact, a lot of the classical machine learning was all about truth in the sense of capturing what we call a probability distribution of the variables that we observe and even the ones we don't observe, called latent variables.
So let's go back to my healthcare example. We split the question of taking decisions, say for a patient, in two parts. One is the science, like how does the world work or how does the body work? How do treatments affect the body? That's the kind of the understanding part, scientific part. Then there's a decision part. So say doctors try to match the patient's information to the right studies and then pick the treatment that is predicted by the studies to give the outcomes we want, including the safety aspects. So in a nutshell, that's what I'm proposing to do. Nothing like really out of this world, except typically the science is done by humans. If you think about like other fields where safety matters, say building a plane or a bridge, usually the science part is a physical model or a biological model or something designed by humans. Here in AI, the model itself is something we learn from the data.
Okay. And then we have to ask hard questions about how do we make sure that when we take decisions, we're not going to produce new corruptions, the hard questions there. I want to go back to what are the minimal or like sufficient conditions, rather, for this learning about how the world works to not be corrupted by downstream goals. It is something we've studied in a paper that I think, yeah, it came out at the end of June, called Safety from Honesty in a Disinterested AI Predictor. And there's a property here that we study called consequence invariance. You would like an honest system, like what science is supposed to be doing, to produce theories and predictions that would be the same whether or not the predictions are used one way or another way. You know, maybe you don't like the predictions that science makes. Think about climate. Well, you don't change the predictions because you don't like them. So the process by which you train, say, these neural nets have to respect this constraint, which is now a mathematical constraint.
There's another aspect of science in addition to this consequence invariance, which can also be formalized mathematically, which philosophers called epistemic humility. So honesty implies that you know that you don't know when you're not sure. You don't claim something to be true. Unfortunately, humans tend to claim things that they're not sure of and they feel very sure of. But the scientific ideal here, I had a discussion with Jeff Hinton a few years ago and we were talking about science. And he said the prime directive, remember the prime directive, you know, Star Trek? The prime directive for scientists is honesty. The prime directive is to never claim something as true if you're not 100% sure that it is true. Otherwise, just qualify. I believe there are clues, da da da. Okay? That's why scientists annoyingly hedge everything they say. This is the reason. And it turns out that when you want to take safety decisions where people's lives are at stake, that humility is very important. And of course, in medicine, you know, we have confidence intervals and all of that, calibration. And it's going to be important as well when we build AI that's supposed to be safe. And the reason is the predictions we're going to make with, say, our epistemic neural net that learns to understand how the world works and make predictions. Those predictions will never be perfect. So even the predictions of an entity that is meant to be honest, like honesty doesn't mean perfection. You can still make mistakes. But you should have a good sense of how much you trust those predictions. That's epistemic humility. And you can't have strong safety guarantees if you don't include this component. And it's actually the hardest part of all.
So we have this fairly complex, unfortunately, scientist AI research program. But there are some very simple pieces of it that I'm going to try to briefly explain to try to get these properties. But basically we don't want the AIs to have goals. The AI that learns to make predictions should make those predictions because the data says this is, you know, a good prediction and not for other reasons. So I mentioned pre-training and post-training. Now, as I said, in the pre-training, the AIs are like reading a lot of texts, basically everything that humans have written and is available, and they're learning to reproduce the sequences of tokens they're reading. Sounds okay in the sense that, yeah, they're learning the distribution of things people write, and then you can query them. Now you have to understand what those queries mean. Those queries mean if you were to read a sequence of tokens up to this point, what, you know, what would the rest look like, and you could sample from that distribution. That's how they answer questions, because, yeah, humans will do that, but humans will sometimes answer questions because they are interested, for the same reasons we discussed earlier. And, you know, we don't actually want our AIs to just reproduce all the stupid things that humans say. A claim made in a Nature paper is not like a tweet from someone I will not name. So we have a process to try to mitigate that issue, which we call contextualization. It's very, very simple. Instead of teaching the AI to imitate those sequences of tokens, they're trying to explain it. So I'm going to try to elaborate. We try to teach them the difference between what is true, like the difference between what is true and what somebody claims to be true. So if a tweet says the Earth is flat, we don't want the AI to just reproduce that. And we don't know, I mean, I guess I do, but if it is really flat or not. So we need to distinguish two kinds of statements or two kinds of random variables because let's say we don't know what is true or what is not. There is the communication act. Someone wrote that the Earth is flat. It isn't the same kind of object, the same kind of observation as the Earth is flat. So basically we can do that by tagging the data from the internet by its source. Who wrote this, when, what venue, and so on. So that we don't attribute the same degree of trust to things that are not worth the same degree of trust. Okay. So let's try to manage the imitation that can be misleading. But the biggie is the second box on the right. I mentioned this consequence invariance. I mean, it sounds simple because really any predictive training is not going to be geared at obtaining some goals. So it's going to be consequence invariant, but unfortunately we need something that is going to be able to achieve goals. So how do we get machines to do things when we don't train them to do things?
What does my next slide say? Okay. Let's see. I don't have the slide that I would like. It's all right. So I'm going to tell you. So what does the doctor do in my little healthcare example? Let's say that it had access to an oracle that knows all the scientific studies and in fact can even reason about them so it can combine different results. Well, it would ask questions to the oracle like what is the probability that this patient with such and such characteristics, if they have this treatment, will, you know, have such and such outcomes. So the doctor probably want to know if the symptoms will go away, if the disease will be cured, and if there will be very dangerous side effects, things like this. So you can just ask those questions in natural language and use the language of probabilities. So you want to predict some things from the future. What's the quality of these things happening given the context, say about the patient, and given an action like follow this treatment. And then you would do that for many possible treatments and then pick a treatment that seems to be good according to the oracle. It's a very simple prescription. It's actually nothing new in science. That's how planes are flown. And there's more details to this, but that's kind of the gist of how you can turn knowledge into actions.
All right. There is a very property that we're seeking here, which is as we put more compute, more intelligence into those machines, we would like them to become more capable, as in more accurate in their predictions and taking better decisions, and also without sacrificing safety. In fact, you'd like safety to improve as they train more. And I'm claiming that the kind of approach I've highlighted would do that, whereas the standard approach would have the opposite effect, that the safety properties will degrade because now with more intelligence, the AI will find more sophisticated cheats, which could go, you know, to a terrible place. Yeah, I'm going to skip that.
Just a few words about why it's even possible to come up with theoretical guarantees on neural nets. If you had asked me like three years ago when I started like mulling about this, I thought, hey, there's no way we can do mathematical guarantees on neural nets. I know neural nets as my friends for, you know, 40 years. But I was wrong. There is a way that comes from thinking of the whole stochastic process from, you know, initializing the neural net to updating the weights of the neural net one step at a time, which is another stochastic process. And now you can think of a whole population of neural nets, a distribution of neural nets. So instead of thinking about a single neural net that I have in my hand and it's very difficult to have hard guarantees about it, if you ask about, well, what's the chance that this neural net sampled from this distribution of neural nets will behave badly, now you can say things. And in particular, if behave badly is something that requires a lot of coordination, what I call like a needle in a haystack. Like for example, a successful cyber attack takes tens of thousands of steps and you have to really know what you're trying to do. It can't come out of incompetence. It can't come out of a series of mistakes. You know, so that's the kind of event that we can get guarantees for because by default, like the random neural net, randomly initialized neural net won't do it.
All right. And you can also prove things about if you were to train these neural nets to death that actually they would also be safe. I mean, there's some qualification here, but there's a way to set up what we call the controller that queries the oracle to get very strong guarantees about the behavior of the neural nets again. All right. So that kind of ends my more technical discussion. So what do we want to do with this at Law Zero? I've talked to the leadership of many of the AI companies. They think it's interesting what we're doing, but they want proof empirically that this kind of approach, which is very different from what they do, will work. So that's our goal. It's to show a methodology on a small scale with the resources that we have. We don't have 10 billion to train, you know, a frontier model. Show that the methodology gets rid of the most egregious misaligned behavior of the kind I mentioned, of the kind that we've seen over the summer, without losing incapability. Otherwise they're not interested. The companies don't want, I mean, it's easy to have safety if you have something that can't do much, right? And by the way, there's always going to be a price for safety, just like in any other discipline. If you think about it, acting ethically is always more limiting than not. But that is what we want, right? It's going to work better if there's to be a transition to systems like this if governments also put pressure, if the public puts pressure on these companies. For example, if there was the same kind of law in AI as we have for bridges, meat, planes, elevators, buildings, basically you have to prove to independent experts representing the public that the thing you're doing is not going to be catastrophically harmful for the public. It's a very simple principle. Pharmaceutical companies spend 90% of their money on showing that the drugs are safe, right? AI companies spend a few percent and they're the ones judging whether it's okay or not. Doesn't make sense.
The other thing we're doing besides trying to build a technical solution is also trying to build a multilateral coalition of countries interested in seeing both those technical advances, but also that the advances in AI will be governed for the public good in a way where the power over AI is decentralized. And to make sure this actually happens, the best solution, at least one reasonable solution, is to make sure it's a coalition of countries that are driving those advances. And so we're starting with a very small group of two countries here, Germany and Canada, but hopefully many more will join. And hopefully other organizations in the future will also end up being governed with a very strong direction from the public because market forces here can go in very bad places that don't necessarily correspond to what citizens want or prefer. So I'm here in Berlin for a number of reasons and events, but also because we're soon going to open an office to continue our research here with funding from both governments that I mentioned. And yeah, we'll be hiring a lot of people, mostly technical, but also all kinds of staff to run this here. If you know people who are interested, you know, they can go on our website or reach out and we're happy to do the recruitment job. You see the list of positions right now on the technical side, from math to engineering with machine learning in between, people who are doing the things themselves and people who are managing them. All of these. Thank you.
