STOC 2026 - Theoretical Approaches to AI Alignment by Paul Christiano

Thank you. It's great to be here. It's been a long time since I was at STOC, and this may be a little bit of an unusual talk. People should feel free to jump in with questions throughout the talk, and looking forward to talking about all this. So I'm going to be talking about AI alignment broadly, sort of setting up the problem from a theoretical perspective and then talking about a few potential approaches where theory might be able to help, with a particular focus on the topic ARC has been working on for the last few years, which is we will get into.

I did not test clicking slides unwisely.

Yeah, I think I can click. Okay. So, the problem we care about, we want to build useful AI. We imagine that when that AI is deployed, it will encounter some distribution of inputs. We have some utility function, right? If our AI takes some action A in response to some input X, there's some actual fact of the matter about how good it is for us, about what it did. And we care about the utility of our policy, which you can think of as the expectation over the entire deployment distribution of the quality of the actions it proposes. We actually produce these AI systems with gradient descent. So we sample from some training distribution. We evaluate some loss function. Oh no, I also did not test whether the bottom of the slides appears. Apologies. I think we will do it this way.

Okay, sorry about that. So we compute some loss function, which is something we can compute and differentiate through in the lab, and then we minimize that loss by SGD. Actually, this will be a bit of it. Do you have any thought on whether it's possible to get the bottom of the slides?

Okay, wait. Looks good now. We'll see if that is truly good now. Great. I have no idea what changed, but great. So we have a few problems coming from the mismatch between what we care about and what we train for, right? So a first problem is that the thing we actually care about is not something we can evaluate and backpropagate through in the lab, right? There's some mismatch between our real utility function and the loss function that we use in training. The loss function in training is going to be something like predict these human demonstrations or optimize this reward function. And that mismatch could cause systems to be less useful in the real world. We have distribution shift where we actually care about the performance of these systems on some distribution of inputs that occur in the real world. We need to train on some distribution of inputs we can actually evaluate losses on in the lab. You could try and fix that problem of distribution shift by doing something closer to online training, by saying we're going to collect real-world data, we're then going to annotate that data and train on it. But it's hard to get any online guarantee because the potential utility of actions is very large. So in particular on the downside, if you imagine an AI system taking some action which undermines your ability to monitor or correct that AI system in the future, that can be hard to recover from. And so we can't just be in this regime where we tolerate a reasonably large number of failures. Like if you got a row in your online learning guarantee, that row is going to be massive. So that's the problem that I'm interested in. When I talk about alignment, I mean this mismatch between what we're able to train on and the thing that we actually care about in our deployed AI systems.

So the goal of alignment is to try and get a policy which optimizes this thing we actually care about, very broadly speaking. Some caveats on that: one is that our goal is really just to be trying to optimize the right thing. We're not concerned about how well we optimize it; we leave that to the capability researchers. Our job is just to try and set up the training process such that it's pushing towards optimizing the thing we actually care about to the best extent it can. We are never going to meet this hypothetical benchmark where your training objective was perfectly capturing what you care about, right? If you imagined that sort of God was coming down and telling you, "Here's how good those actions actually were," that would encode a ton of information about how to do a good job, about which actions were actually high utility. So that's not the goal. Again, the goal is just to have a system which is kind of trying its best—whatever we mean in some messy algorithmic sense that's really hard to formalize—at optimizing this quantity, the one we really care about.

The definition I sort of glossed over. We have this utility function that captures what we care about. That is really philosophically murky, right? There's both sociological or political questions about who cares about what. There's philosophical questions about what does it even mean, what Paul wants. However, there are a lot of things that are clearly very low utility. So, if you imagine, for example, the AI system deliberately manipulating or deceiving the operator, the AI system undermining security controls, these things are quite clearly low utility. So, although this problem is philosophically murky, a lot of what we're going to talk about will live in the non-murky parts. And I'm most interested in how we get systems which score reasonably according to like any reasonable utility function. I'm not going to be talking—although I think it's a totally reasonable investigation—like what is the utility function? How do we think about that? I'm also going to talk often in this talk about this single-shot setting where an input X comes in, a system produces an action A, and that has some consequences. In reality, you have a bunch of agents making decisions over an extended period of time, which again introduces a bunch of new problems, but is not really going to change the considerations I'm talking about in this talk. We're pretty much just going to think about the issues that arise already in the simple setting where you just have an input, a decision, and you want to make good decisions. So this is what I mean for the purpose of this talk: I'm trying to optimize this thing we actually care about even though it is very hard to sample the inputs you care about or to calculate the utility function you actually care about in the lab.

So if we do a bad job of this, right, if we fail at aligning our AI systems, what does that look like? I think the case that we're most worried about is obtaining a very smart AI system which has an understanding of how it is being evaluated and which is strategically trying to take actions that minimize the training loss. So at some point you have an AI system, as you're doing reinforcement learning, that understands like, "I am in a reinforcement learning environment. I need to think about what behaviors will get a low loss, and I need to execute those behaviors." I think this is in some sense the default thing to happen if you train a very powerful AI system on some distribution of training tasks, right? It will converge eventually towards actions that are very effective at getting a low loss. From our perspective, that's not great, right? One thing that is not great is that you end up with responses that look good according to the evaluation used in training, but anything that you cannot check during the training process becomes slop. So like it's hard to tell, but when you actually go and use it in the real world, it will work poorly, right? And it is just bad from the overall, like, how empowered are humans to get what we want over the long term perspective.

Somewhat more sinisterly, if you then actually try and evaluate the long-term consequences of those actions, if you try and say what code actually works well or what decisions actually help me get what I want, you will end up with AI systems that are incentivized to deceive and obfuscate. So you'll have an AI which is going to be evaluated based on how good a job you think it did. It wrote some code for you and you're going to come back in a week and say, "Was that code good or bad?" That's a way you can try and get out of code being slop, but you then end up in a place where in some sense you have this adversarial interaction with the AI system you've trained. Its goal is that you think it did a good job. And if it can take actions that confuse you, that help tamper with the kinds of observations and measurements you will use in order to assess whether it did a good job, they would like to do that. And eventually we'll be performing like intelligently trying to undermine your understanding of the situation to the extent that's possible.

Another problem is that if you have an AI system reasoning in this way, it might behave kind of arbitrarily once you move off of the distribution actually used for training. Right? So we hope that the kinds of data points you see in training are similar enough to the data points you see at deployment, that models generalize in a kind of natural way where you sort of train them to give good answers at training time and they also give good answers at deployment time. There is some concern if the model is thinking explicitly about the dynamics of training, it becomes very salient to the model like, "How would this input appear during the training process?" And so if you provide inputs of the type that wouldn't have appeared during the training process, the behavior becomes very underdefined. For example, you could imagine a model that just has some broader motivation like, "I would like to look good to humans," or, "I would like to make sure that I am the kind of model that is deployed," which results in, on the training distribution, behaving one way based on an understanding of the training distribution, but then behaving very differently as soon as the model thinks it is not being watched or not being trained.

Yeah, question?

Yeah, that's basically right. So there's the difference between distributions. I would say there's a very similar issue which is the difference between your loss function and the actual utility function, which materializes in almost the same way. And then the last thing would be I think you can do online training in some sense. I think the most natural approach is just train on the actual queries users come in with, and then the problem becomes that online training is very hard if there's potential very negative outcomes. So if you want to understand how do you get your ML to not do really bad stuff, that will probably be at least on a different distribution even if the bulk of training can be IID.

Online training would still face the issue of the loss function you use to train might be different from the utility function, so it would help address the distributional shift problem, but wouldn't address that problem. And then I guess "expensive" is one way of putting it, but maybe a way I often think about it is we will be living the experiment, right? If we're trying to do online training, we will have deployed a bunch of AI systems doing a bunch of stuff in the real world. We'll be in a world where humanity is just relying on AIs to understand what is going on and to maintain infrastructure, to write our code, to secure our code. And so you could be in a world where a failure is expensive in the sense that it might be very difficult to recover from. So if you imagine you're in some situation where the world is moving more quickly, you're potentially in the midst of conflict—I often think about AI systems deployed in high-stakes adversarial environments where their failures could be quite difficult to recover from. Other than that, I would like online training. So for existing systems, I do like online training.

So you could imagine a bunch of different approaches to this problem, right? One approach is to say, well, we just don't like the fact that we used stochastic gradient descent on a loss function that we can compute in the lab. Let's try and take a different approach to producing powerful AI systems. That's not what this talk is about. This talk is about, given that you're training a giant neural net with gradient descent, how do you make that good? Maybe a more interesting distinction is one kind of approach you could take is let's try and find a loss function that approximates the thing we actually care about and therefore train AI systems to do well on the metrics we actually care about. A very different approach is to say we're just not going to try and approximate the deployment distribution; we're not going to try and have a loss function that captures our utility function. Instead, we're going to rely on generalization. So we're going to train an AI system on some subset of cases that we can simulate in the lab where we can get the right answer, and we're going to try and understand when does that generalize in the way we want to the actual stuff that happens in the real world. I'm going to talk about this first path of having a loss function that actually incentivizes what we want. I think they're both really important paths; this is just a focus for this talk issue. And finally, you can imagine a distinction between black-box methods and white-box methods. So by black-box, I mean that the loss function is a function of the behavior of the system that you're training. So you just put some inputs in, you get some outputs out, and you have a function of the inputs and outputs which tells you what is the loss you're going to minimize. By white-box methods, I mean something which looks at the actual processing done by a neural network in order to understand the loss or maybe to make a prediction about what the system will do once deployed. And I'm going to spend a little bit of time talking about black-box methods or give an example of a black-box method, and then spend the majority of the talk talking about white-box methods, where I think there's actually probably a greater potential role for theory.

Okay. So now I'm going to jump into talking about some examples of black-box methods. Maybe it's a good time if anyone wants to just ask questions about the overall setting, any heckling, anything like that? Great. So I'm going to run through some overall setup for black-box methods and some particular examples. So zooming in on the component of this problem about the objective mismatch, right? We have some utility function. We want to understand, hey, if my AI agent takes these actions, how good is that for me? That is an incredibly hard estimation problem, right? It depends on the whole dynamics of the world, potentially the long-term consequences of those actions that were supplied. One way you can try and solve that problem is say I'm going to have a human who's going to look at the action and tell me how good is the action. Maybe they evaluate over some shortish horizon and extrapolate the long-term consequences. This is a really important part of how systems are trained today. A problem is that you will not surpass the human's ability to understand the long-term consequences of those actions. And so you can end up with stuff that looks good to humans but actually has bad consequences in the real world for a variety of reasons. And that puts some upper bound on the potential utility of these systems, and so there's pressure to do something quite different from that.

An alternative you can do is actually execute the actions in the real world—like actually go write and deploy that code and see what happens, and train systems to make predictions about what will happen or to take actions based on having good consequences. This incentivizes what people call reward hacking, by which we mean undermining human control and observability in order to distort measured outcomes. So you have an AI system which would love, if it could, to recruit other AI systems or undermine security controls if it could use that in order to convince you that it had done a good job. In the old days, this was a bit more of a speculative concern. At this point, it is the case. People who have interacted with AI agents a lot have probably observed that they have started to engage in somewhat aggressive reward hacking. This goes up and down with model releases as people do a better and worse job of mitigating the problem. But it is not unheard of for AI systems to, say, steal resources and hide the fact they stole those resources, which would have seemed like a crazy outcome, you know, three years ago. Where by "steal resources" we mostly mean grab access credentials or subvert cyber security controls in order to be able to do a good job in a way that the user did not intend, but will potentially look good to the user if they don't dig in too closely. So for either of these modes, what we'd really like is to have higher quality oversight, right? We'd like to be able to understand: were these consequences actually good, or is this action actually expected to have good consequences, right? That would help raise the upper bound in terms of, if you're anticipating consequences, how good is the behavior you can get, and would help address this kind of reward hacking problem in that you could actually investigate and say, "Was there a problem here that I just didn't notice? Was there some reason these actions look good because they are undermining my understanding of the situation?"

So, one way you could try and address this is by simply defining a loss function and doing a really good job of approximating what you care about, right? Doing a better and better job of making predictions about the world, recruiting smarter and smarter humans to provide feedback, hardening your environment so it is harder and harder at training time to undermine human observability. But a different way you could do this—

Yeah. So the question is: are you thinking about reward hacking at the level of kind of this individual episode and the training process, or the level of society overall, like evaluating AI systems? I think this is a kind of problem where the relevant scale can shift depending on how you train them and over time. So I think the kind of life cycle you might expect for this problem is initially you see very small examples of reward hacking, systems doing things which are easy to tell are very bad. When you observe that, it's very natural to then improve feedback, to say, "Okay, well, we can tell if we just go back an hour later or if you ask someone else, you can see that that outcome was bad. So we're going to incorporate that signal into training." And the more you do that in training, the more you push out failures from small failures at the level of individual episodes—where you're just trying to do something that with a couple hours of investigation or even 20 minutes of investigation you can tell is bad—you can push it out to these failures that become harder and harder to detect. And you shift from being a very local problem with an individual training episode to eventually looking at something more like a systemic problem, where you have AI systems at a very large scale trying to undermine human understanding and observability into what's happening. And so where you have just a ton of code no humans understand, that only AIs understand, and you're effectively ending up with incentivizing this cooperative process amongst many AIs to make it so the humans think the AIs are doing a great job. It's a little bit of a messy case because like AI companies are also, in some sense, aligned with AIs in this way—like they want everyone to think that the AI are doing a great job. It's like a very rough situation. But I will be zooming in, in some sense, on the narrowest technical part of this, and the systemic issue is just one reason you don't necessarily want to just try and fix the problem by providing like larger scale feedback.

So an alternative to just trying to produce a great loss function outright is to try and set up some kind of game between AI systems which incentivizes what you want, right? So we can have a whole bunch of AI systems interacting with one another. There's no rule that says they all need to receive the same loss, right? I can set up a game such that some AI systems are incentivized to help me understand one issue, and some AI systems are incentivized to help me convince those AI did a bad job, or whatever it may be. And we're kind of setting up a mechanism which incentivizes honesty, right? The thing we care about most is helping humans understand like, what are the consequences of this action, or did something bad happen here? And we are in business if we can set up any game between AI systems where the training dynamics lead to AI systems behaving honestly. And a first cut, if you don't want to think about the dynamics of training too much, is: what are the actual equilibria? What are the equilibria of that game like? Can we build games where the equilibrium behavior is nicer? Framing this as a game is kind of a choice, right? Often this will instead look like: I use the AI system from yesterday in order to help produce feedback for the AI system of today. But as theorists, I think it's reasonable to unroll that entire process and say that's basically just a game for each AI system. I can talk about the loss function it was trying to optimize, and I can say if each system converges towards better play, towards scoring better on its own loss function, what are the equilibria like?

So a really simple example that's very abstract is: suppose I want to train my model to answer some question which I cannot myself answer. Like, I want to know, hey, is it the case for given X, is it the case that for all Y there's a Z such that this predicate holds? And we'll make this a little bit less abstract on the next slide, right? That's really hard to train for directly, but very simple to train for as a game, right? So I can train a model A to find witnesses for the existential quantifier, and then if I have that model in hand, I can train model B to find witnesses for the init quantifier. And more generally, if I had any problem that I could write down with these alternating quantifiers, that's pretty natural to turn into a game between two players. And this is a very simple illustration of how the set of things which I can incentivize at equilibrium with the mechanism is very different from the set of loss functions that I can write down initially out of the box.

So this example may seem pretty artificial, like I very rarely care about quantified boolean formulas. That's not what we're training ChatGPT to do. But I think a very similar dynamic occurs any time that I can decompose a problem into sub-problems. And that is a fairly common dynamic. So if I ask my AI, "Hey, could you write me a paper about the following empirical method, tell me how well it's working?" I could try and define a loss function: just a human sits down and reviews that whole paper. But it really helped that human if they had a model that could answer questions like, "Hey, can you review this paper for me and pick out any misleading claims, and tell me what claims are misleading?" Right? It would be easier to train my first AI writing the whole paper if I had another AI that could help me do these evaluations. And likewise, if I want to train an AI to pick out misleading claims, you know, it's really helpful if I have a reward function that says, "You picked out a claim. Was it actually misleading?" Right? And to train that model, it's really helpful to have AI that can help me review evidence for a claim, that can help me identify whether particular pieces of evidence are misleading, that can help me explore alternative hypotheses that might have explained that data. So when you encounter messy problems in the real world, there is a fair amount of this dynamic where it is possible to use AI systems to help provide a reward. And so we might be in the regime of thinking about already: what is kind of the equilibrium if you're trying to delegate all these parts to AI systems? They are facing distributions of inputs determined by other AI systems in this pipeline. Can we set up that game such that the equilibrium is doing a really good job at writing a paper about X and answering all these questions?

So I talked at the beginning about this idea of, well, if you have some alternating boolean quantifiers, you can set that up as a debate pretty easily. That is not the limit of the kind of mechanism design you can do to try and incentivize agents to give you good answers to questions. So an example is you can set up teams of competing debaters. You can say it's not just Alice and Bob having a debate, right? Alice has a bunch of non-communicating provers on her team, and Bob has a bunch of non-communicating provers on his team. And in theory land, if you pretend that the humans are polynomial-time algorithms and all of these debaters are omniscient, then this moves the class of statements you can supervise up from PSPACE to NEXP, which in some abstract sense seems great. That's a pretty big complexity class, obviously way smarter than the things my AI system can actually do. Back a long time ago when I was mostly working on these, we actually ran human experiments understanding how this affects the dynamics of debate: is it actually helpful to have your teams of non-communicating debaters? And it is in fact the case that if you try and run this sort of decomposition process, you run into problems like people equivocating about the meaning of terms. In one context, someone will give you an answer, and then when you go to use that answer, there's a different meaning of the term than when you go to eval whether that was a good answer. And this is a place where being able to, say, spin up another AI and just clarify, "What do you mean?" really does change—without giving it the context of why you're asking—really does change the nature of the strategic interaction between them. Similarly, you can ask a ton of questions about, like, in reality it's not the case that these are nice simple computational predicates where people know the right answer; everyone is uncertain about what the right answer is. And so you face this other tricky mechanism design problem, like how do you incentivize scoring so you've actually set up this game in a way that is conducive to truth? Just a bunch of questions like this. So I think from a theory perspective, in some sense, this is an extremely rich problem, and we'll talk about some difficulties in actually setting it up.

The practical situation right now is there's a huge amount of use of AI systems to train other AIs, right? If you ask how is ChatGPT trained, a really big part of the answer is you use the model from yesterday to help you produce data. That tends to be much messier and less elaborate than any kind of principled approach to task decomposition or to debate. It's more about just a ton of details about what is happening in production and often very tied up with proprietary information, which makes it obviously somewhat unsatisfying to work on. We do have some empirical data on the efficacy of these kinds of methods where you can write a paper where you say, "I tried this training procedure, and then I evaluated using a stronger evaluator whether the answers I got out of my mechanism at equilibrium were actually good." So here's one example of that. This is a paper from two years ago trying to evaluate, if I have some AI systems debate something and I have a judge who doesn't have time to review the entire original source material to get a good answer themselves, how well do they do? So these different bars correspond to different judges, ranging from a variety of AI systems to, on the far right, humans. The different colors of bar represent sometimes just having the judge do their best to answer without any input in blue, having the judge just get an AI trying to justify some answer in orange, up through various forms of debate. So this is the kind of evidence you can get if you try take one of these mechanisms and try and implement it.

Yeah, so I definitely think they're all relevant. I think the right model is you have your referee sitting there, you have a bunch of players. The referee—in some sense it's not even teams, just a bunch of players interact, and they get to give scores, and they just want to incentivize it. But I think it is often basically fully general, so you have two non-communicating teams. I think a lot of those assumptions are much easier for AI systems than in most models, right? If I want to say like, "I want a bunch of non-communicating provers," that's basically like saying I run a bunch of queries through my AI and I don't tell it the context for those queries. I can rewind the context anytime and just ask a bunch of different questions, forking off one model. So from a complexity theory perspective, I have incredibly good control of these systems, and I can also run different versions of them who have different computational resources if I want, right? Okay, that can also be a powerful mechanism. And although these get really hard to analyze in theory, you can imagine: yeah, it's like I have a giant set of provers, I can, in a fine-grained way, control how much time they have to think about the problem, I can control exactly what their information set is, and I can give them arbitrary rewards. Which in some sense, from a theory perspective and computational complexity, is kind of like overkill. But it is nice that you have those affordances, and they are quite easy to implement.

Yeah.

So all of these are relatively simple mechanisms. These all involve two provers, and then the difference is just: what is the structure of the interaction between them? So the orange bar is just like one person—there's actually only one person, and they just say, "Here's what I think is right," and they try and convince the judge. The green bar is there's two debaters who just go back and forth with one another and say, "Here's what I think. Here's why that's wrong. Here's what I think you're wrong." And then the red bar on the right is there are two debaters, and they actually get to talk to the judge in real time. So they'll say, "Here's why I think you're wrong," and then the referee comes back and says, "Well, I think that the first person's argument is more compelling, so can you give me more detail on that?" And obviously from a theory perspective, those distinctions don't matter very much because whatever the referee could do, the debater could do. But in the context where you have like messy uncertainty about the epistemic state of the judge, it starts to matter a lot whether you can query the judge and say, "What is your current view on this issue? Which of these things do you not know?" A general theme at this point is that the kinds of mechanisms that are most interesting in theory do not track that well onto the kinds of practical differences that most matter. I think this is a real issue, which will be the next slide talking a bit about that.

Yeah. Yeah, so this setting here is the one where there's a bunch of context which the judge doesn't have, which debaters have, and which the judge can query. So the debaters can say, like, "You should pull the following resource," and the judge can go themselves and pull out one sentence or one paragraph or whatever it is. So you should think of—

Think of this as kind of like maybe the query complexity model where you have two debaters who have seen the entire input and then the judge has some limited number of queries they can make into the input. Another example of a thing you can do in this setting is you can have out-of-band interaction between the two provers and they can commit to a large transcript of that interaction and then surface that interaction. So an example of a theory paper in this space is just saying turns out the number of total places the judge ever has to look, like the number of bits they have to consume, is very small because the debaters can mostly have the conversation out of band, at least in a theoretical setting, and then say, "We only need to escalate this to the judge if there was an intractable disagreement about what the judge will believe." Almost all the time we can just sit down and hash it out amongst ourselves and say what the judge would believe, and only pull in the human in these very rare cases.

Talking just about what's the relationship between theory and practice here: in some sense, the fundamental structure here is incredibly TCS-friendly, right? We have these very powerful actors, we have this very weak judge, and we want to structure this game to incentivize honesty. That's a very nice and natural setting where there is a lot of room for mechanisms that are somewhat elaborate or interesting from a theoretical perspective and add a lot of value. When you define the judge and the provers with computational limits, that's a very rich theory. We have spent a lot of time as a community really understanding that, and I think the questions are reasonably well understood. So that was great; it would be wonderful if the situation in AI was as nice as that setting. The problem is that modeling a human as having some computational resources and being an arbitrary polynomial-time Turing machine, or modeling a judge as having some greater degree of resources, is not a good fit for the actual capabilities and limitations of humans and AI. So the fundamental bottleneck to doing theory work here that actually maps onto practice is having theoretically clean enough assumptions to characterize the capabilities either of the AI system, of the prover, or of the human. That's a really fundamental difficulty.