Thank you. Thank you. Thank you. Thank you, Pablo, for the warm welcome. Thank you to all of you who got up early to be here today. It is a brand new talk. That's what happens when you suddenly find yourself with a lot of time on your hands to read and think, and also, you know, a lot fewer people to be responsible for in terms of whose work you might shine a light on. So here we go. Please be kind, but please be provocative also if you have any questions. I'm happy to take that.
So where I want to take us, and in some ways I think Dale sort of prepared the grounds for that yesterday, is to think outside of this framework of Bellman and really think about what we value. And while it would be really tempting to just like spend the hour thinking about AGI and superintelligence, and I'll encourage you to tune in to whatever is going to happen later today, actually I wanted you to focus on one really gnarly little problem which we need to solve together, which is: what do we actually want from this general intelligence? Like, what do we want these machines to do for us? What is this notion of value that is so fundamental to reinforcement learning, so fundamental to how we go about conducting our lives? And how do we build bridges between the formalism and the more intangibles of value?
And so really, you know, value — very overloaded word. So I'm someone who does like formalism. I embrace definitions. They're reassuring and they ground the work that we do. And when you start in reinforcement learning, we encode it in math because math is precise. And so for many years now, decades, we've looked at value as being this expected sum of rewards over a trajectory of states, subject to a policy. There's a variance on this one, but that's roughly — I hope you know — like that is the question at the entrance door when you get into this conference: what is value? And now I'm going to ask you to let go of that as you walk outside of that door.
And so when you look in other fields, in economics, the notion of value is really defined in terms of like the worth of a good, worth of an outcome, usually typically encoded in terms of financial values of outcomes. We can look a little bit beyond economics and get into things that are slightly more squishy. Ethics — incredibly important to the work that we do in that space. Actually, values defined in terms of actions being morally right or good or wrong in terms of a specific pursuit, and that gives rise to concepts such as honesty, justice, fairness — things that we've tried to put into our algorithms, but I think there's a lot more work to do. So it's not completely disconnected.
Sociology gets even a little bit further from the stuff sometimes we're comfortable about. In sociology, what's interesting is that there's this notion that values are actually shared beliefs and shared norms. And this notion that you don't sort of receive these values from external signal — you actually create them and you can shape them. We shape them through our institutions. We shape them through our discussions. And so that gives a really interesting angle in terms of what we might have to do in terms of our own responsibility here. So this is really the core question for this morning.
How do we formalize what is valuable as we, you know, as we build AI systems, as they permeate all the spheres of our life? Like in simple terms, like what do we care about and how do we put that into the machines that we're building? And so I'll be asking more questions than I will be providing answers for today. But I do hope to plant seeds in your minds in terms of a willingness to engage with this broader question, to actually revisit the foundations of what we are doing rather than take for granted that value is the expectation of the sum of rewards.
So this is going to be three chapters. First I'll start on, you know, familiar grounds — classic Bellman. What do we know about that? I'll have a chapter about sort of timidly extending beyond Bellman, and then I'll have a chapter where we really break free and go beyond that Bellman formalism. So let's start on familiar grounds, still early morning. Bellman gave us this wonderful equation, and frankly this is possibly like the single equation that made me fall in love with RL. There's an elegance here that has guided my professional path for 30 years now. And what's so elegant about this is that it gives us a simple way to turn planning of what we try to do into computation. If you compute the value, then you get the answer to how you should behave. And there's something profoundly inspiring about that.
The challenge when you fall in love with an equation is that you sometimes forget all the things that are underlying that formalism. So here we have it, you know, underneath the hood of that: there's the Markovian assumption — the fact that, of course, you know, you only depend on the current state and action and none of the history matters, that you're able to capture all that. The stationarity assumption — the world is never going to change. Well, that's not quite true. The scalar assumption — that you can capture value as a number, that there's a way to quantify everything that is of value.
The additivity — that you can add up these values, that it obeys the simple laws of mathematics. Estimation — that you're going to have enough data that you can actually estimate this and be able to use that to drive planning. And then also uniqueness — the assumption that you're dealing with one agent and there is one gold reward function that we should be optimizing. So that's a lot to carry, and I'm assuming none of this is new to you. I hope at least you know, may need a little refresher, but hopefully not completely new.
Why I'm bringing this up is as we're moving forward towards AGI and so on, I think we have to look with more courage at how do we break out of these assumptions and yet not be in a completely free-for-all world. And so what is the in-between space where we adopt some common ways to represent information, where we have some common notion of what are the properties of systems, and yet not quite so rigid as what we've leaned on for the last little while? And that involves us being willing to say that value sometimes can be negotiated, can be discussed, and can arise, or even can be plural — doesn't have to be singular.
And so, you know, one of the things I want to acknowledge: this gets a little bit uncomfortable because the value function in many ways has been the soul of RL. It's a central abstraction that permeates so much of our work. It allows us to link the theory, to link the practice. And when you have that same abstraction, it means you have the same language for talking about things. I mean, an RL researcher: 'How's your value?' 'How's your value?' We shake hands. We know what we mean, right? This is not necessarily comfortable to get outside of that. And it also gave us our unique language compared to other fields, whether it's control theory, supervised learning, or others.
And beyond that, it's not just comforting. It's not just about, you know, being in the cult of RL. This is about making progress together. Having a common formulation means that we could build on the same abstraction. It means that we could have the same benchmarks and we knew what we needed to measure to make progress. We knew what to compare, and also as much as possible we could make advances that were modular. You make progress on function approximation, you make progress on exploration. We can put these pieces together within this formulation. So I don't want to completely ignore all the benefit we've gotten from having this common formalism. I want to acknowledge them so that we can build new common formalisms that give us more richness of value and yet have some of these properties as well.
And so really, you know, while value function unified our math, it unified our community, it gives us a lot of power — we need to find new formalisms that can play that same role for this group. And I also had some fun just pulling some numbers in terms of like how much this has unified our community and pulled it together. The first time I'd pulled this graph together was around 2018, I think, when we were doing some of the early work on reproducibility in RL, and at the time we were, you know, somewhere around like 30,000 papers a year on RL. And I pulled the numbers again like last week, and like suddenly we'd seen a huge rise — and so 180,000 papers that discuss RL on Google Scholar.
And, you know, I think sometimes the last few years still have to catch up, takes a little bit of time, so I don't know if it's like a definite decrease, increase, or that's going to keep on increasing. We'll have to see. But nonetheless, you know, like clearly there's a lot of interest in this field. There's a lot of people working in this field. So we also have potential to explore many more ideas of what it means to do RL.
And so let's take this to the next chapter of how we might timidly extend the Bellman formalism, and sometimes to really great benefits. And so I'll sort of talk about a few phases of the field. We've had the classic RL period where we were really looking at very specific individual tasks — grid worlds, cart-pole balancing, these kinds of things — with very simple setup for evaluation. We moved to DRL where we had families of tasks, and what we were interested in was looking at how do we learn very efficiently, because the tasks were getting more complex — not just if we can learn, it's like how do we learn efficiently, and also how do we generalize within a family of tasks most of the time.
And what we got out of all this work is the ability to start tackling real-world applications of RL, and it's been wonderful. Every time I come to a conference such as this one, I talk to people who have new ideas of how to take RL and apply it to real-world problems. But when you do that, you come across many very messy problems of how to do that. In my own case, I was in particular interested for many years in looking at how we apply RL towards healthcare, personalized medicine, the ability to use RL to optimize sequences of treatments and to customize that to different patient characteristics.
I'm not going to go into the details of this work today, but just to say like very quickly, I got a sense that, you know, in some ways the assumptions of Bellman were much too rigid for what we were trying to do. You know, the Markov property — forget that there's a lot of partially observable information that we're not able to measure. Stationarity — tough one as well, you know, patient to patient over time, whether it's the patient changing, whether it's the devices and so on. Scalar rewards — it's very rare that you just have like one single quantity that you can maximize. There's often considerations in terms of toxicity, side effects, things like that, and somehow you need to pull it together.
And so while for many years, and when I was in more theoretical domains, I was quite happy to, you know, look at the tension between exploration and exploitation, and we had beautiful theory to do that and algorithms that were getting increasingly better — when I got to the real world, that was actually not the tension and the trade-off that was the most meaningful for solving the problem. And so we wanted to make sure we weren't going to harm patients. We wanted to make sure that we were respecting constraints in terms of confidentiality or achieving good properties in terms of transparency. That put us like in a whole other dimension compared to what the classical Bellman framework provided.
So I spent some years looking at what I'll call sort of RL-plus approaches. And so we spent some time exploring like what happens if you take classic Bellman formulation and you try to add safety to it, or what happens if you try to add privacy, or fairness, or interpretability — sort of one by one knocking down the types of things that matter for applications, in particular healthcare applications, and trying to see how can we broaden the framework, extend the Bellman framework to achieve that.
And so let me give you just a little — I won't cover extensively all of that, but let me give you at least a taste of what that looks like. RL-plus: there's constrained MDP framework. And so in this case, proposed a number of years ago, you essentially have all the elements of a standard MDP, but you add two key elements. One of them is a cost function that defines some constraint cost. So a safety penalty, for example, that is a scalar that is added in there. It's state-dependent. So you can add it to whatever your reward function is. And then you have a budget, and so overall the cumulative cost over your trajectory can't exceed that budget. So it's a reasonably straightforward extension.
It's relatively open-ended about like what's the cost, what's the cost constraint, how to set them. And the goal is to find some policy that maximizes cumulative reward while also respecting that cumulative cost constraint. So we started on that path and looked at safe policy iteration methods, and there's several other groups that have done work in this area. This is just one sample of it. We looked at combining a notion of forward value, where we were looking at the expected future reward, with a notion of a backward value, which estimated like the safety cost incurred so far along the trajectory, so that we could verify that we were within the constraint. And then we framed it as an optimization problem.
And so there was both a forward cost, backward cost, the cost constraint, all of that put in together. And so we had a nice clean optimization problem. And out of that we were able to run this. We ran it on some simulated examples. We ran it on some more healthcare scenarios. And we could show — in the middle, just very briefly, you know, the blue line is sort of the new algorithm — we could show that we were able to learn. It took a little bit more time because you had to make sure that your policy even during the learning stage was within the constraint. So that's the plot on the right side. The blue line is respecting that constraint, and on the blue side you end up learning the policy. So, you know, the results, at least at small scale, were reasonably promising.
And again, like if you're interested in this literature, there's actually a rich literature in terms of people who've explored these different quantities — not all of them using constrained MDP, but quite a few using constrained MDPs — and we draw liberal inspiration from a lot of the work from these other groups. That group keeps on going to this day. So I do think that's a viable path to keep on exploring. You get some formalism that people can agree on, make progress, and you get all these benefits.
My problem is most of the time I wasn't satisfied to just optimize a reward and add one of these properties. I wanted like RL-plus — like I wanted to be safe, and I wanted privacy properties, and I also wanted to be able to be transparent, so on and so forth. And once you start doing that, you can like keep on adding cost terms and constraints and just throw it into an optimization procedure, but it tends to not necessarily make explicit how you are doing the trade-offs between all of these terms. And a lot of the solution you end up getting is completely determined by how you set the trade-offs between these different properties. And quite frankly, we don't know how to do that in a principled way, and we don't even know how to elicit how to do that in a scalable way.
And so I was sort of stuck on this, like how do we move beyond these notions of scalar stationary rewards? And of course humans, you know, we use a lot of rich value signals to guide our decisions in all sorts of directions, and we learned quite strongly this lesson of not optimizing just like a single engagement reward. And so we need to have a way to construct values through observations, through interaction, through dialogue in order to leverage all that rich information.
And so I think what I want you — if there's one thing you come out of for today — I think is to realize like this isn't a question for tomorrow. Like this is a question for yesterday. This is a question we all need to engage with as we are doing the work that we do. How do we figure out to bring in these rich values into our system and to have them reflect what we want them to do in a more profound way.
And so, to be a woman of my time, I will of course look at how that question is handled within the frame of generative AI. I do think there's a lot of things that are happening there by necessity in terms of encoding richer values that we can draw inspiration from. I probably won't get to the topic of agents, but I think, you know, to some degree a lot of what we're looking at in generative AI may extend to agents, or I'll need more time to think about it to say something. But I will at least talk about how do we look at reward data, preference data, and so on in this context, and what are some interesting questions that we need to keep on doing research on within that framing.
And so what's the property of these generative AI systems today, whether they be general generative language, image generation, multimodal, and so on? One of the things is they go beyond just looking at a set of like family of problems, but it's really a broad set of domains. The domains are defined by modality more than they're defined by task. But usually it's like the data and the evaluations together that define the domain. The evaluation is also highly multi-dimensional. And I'll talk a little bit more about that.
And so that's an interesting question of how do we leverage data and how do we leverage evaluation much more as the formalism that we share in common rather than a particular equation to ground these questions. And so this is chapter three for today: moving beyond Bellman. What does that look like in terms of what we can be doing in this space? So let's start sort of more, you know, canonically. But what do we mean by human value alignment? And I'll separate that into three steps, right?
First, we need to figure out a way to elicit rich value from people, by which I mean like try to get to the bottom of what people value. I don't mean get them to send money. I mean to actually reveal their true preferences. Second, we need to somehow like take all that signal and distill it into an alignment target. This is, you know, we had reward function, but what is going to be this target for alignment? Because our machines do train by optimization — without that signal there's no learning to some degree. So how do we distill an alignment target, and then how do we train the model to operationalize the alignment target? And I'm using the word operationalize rather than maximize intentionally, because sometimes maximizing isn't necessarily the thing that's going to get us to the behaviors we want out of our machine.
So roughly three steps. Classic LLM training pipeline today. This is a little bit simplified maybe compared to what's running in some of the bigger labs right now, but roughly there's a pre-training phase. You dump a lot of data. You use self-supervised learning to learn a representation from this data. And then there's what I call the alignment phase, which includes some supervised methods. You can think of supervised methods, supervised fine-tuning as a form of direct learning with some analogies to imitation learning. In this case, you get some trajectories from humans, raw feedback, and you train the model to essentially replicate this raw feedback.
And then there's the RL from human feedback. This is more indirect learning through model-based reinforcement learning. So the human feedback is used to train a reward model of human preferences, values, and then you use that reward model to train the LLM, and so it's maximizing with respect to that reward model. So this is the RLHF framework that is used right now in these LLM systems in terms of the training.
When it comes to the evaluation, there's really two types of evaluations that are used routinely. One of them is the leaderboard aspect, where you have a number of tasks where you can automatically evaluate, and because this is broad domain you need many of these different tasks with respect to corpora and so on. This is often monitored throughout the training cycle of these LLMs all the way to the end. And then there's a phase of human judgment where it's more of a qualitative evaluation. Essentially you present different model outputs and you ask humans to judge: do you prefer A, do you prefer B? And out of that you get some score out of these panels. So that's roughly how it is.
In some ways this has given a useful way for the field to track the progress, right? When we look at how do we have common ways to frame the progress, you know, this is a version of it that is commonly used today. Now, there's a few problems with this, and I would say we're still looking for better solutions for this. There's been a number of interesting articles coming out recently that bring to question the validity of some of this work.
There's some work that has looked at some problems in terms of the methodology for the baseline quantitative evaluation. In particular, this first paper points out that maybe the advantage of RLHF has been overestimated because the methodology on the baseline is sort of underestimating the baseline numbers. So that gap may be smaller than we think. So I think there's some questions there.
And then in terms of the qualitative evaluation, there's been some concerns raised also in terms of like who has access to these leaderboards and like how often are you able to submit to these leaderboards? And there seems to be some undisclosed private testing practices that may be benefiting some people submitting models rather than other groups submitting models. And if you're able to test a lot of variants before you release publicly, you may be biasing the numbers that are getting reported. And so that adds a lot of complexity in terms of, you know, gaming and incentives and who has the ability to affect that or not.
I would say to some degree like some of this is not completely new, and certainly feels like a little bit of echoes compared to some of the problems we were trying to point out back in 2017 when we first started sounding the alarm about problematic empirical practices in RL. I wish I was here to tell you that the problem has been eradicated. Unfortunately, not. So my public service announcement for the year is that when we fool ourselves as a community about the quality of the model, we are doing more damage to our field than when we are setting up with actually, you know, rigorous evaluation benchmarks.
For those of you who are interested, we've been running a reproducibility challenge annually since 2018. And for the first time, there's an in-person reproducibility challenge conference happening in two weeks in Princeton. So if you're interested, please look into it. Amazing lineup of speakers and papers. But still a lot of work to do in this area, and that is a shared responsibility both in terms of our own practice and in terms of how we review and evaluate the work of others.
Now let me step back from that just for a second because, you know, that's sort of the state of affairs right now on the LLM front. I just wanted to give you at least a little bit of a brief window for those who maybe been paying less attention. It's interesting to ask what are the characteristics you would want out of an alignment target. So I'm asking everyone to think beyond the box in terms of reward functions — like what is it that we want out of these alignment targets?
And so there's been some interesting proposals thinking about how to formulate this, and I thought this work in particular by Christiano and some of the colleagues was in particular useful in that it articulated qualitatively the characteristics that we wanted. One of them is to have like a way to represent value that's fine-grained enough that it can actually guide model behavior. Like if everything is great or everything is really bad, you don't have the granularity of signal to drive the learning. So you want that granularity. You want the signal to be generalizable. You don't want to have to specify your preferences about every single little option. You want it to, with a minimal amount of signal, maximal information.
You want that to be scalable, meaning as you get more signal about preferences, that should help you drive the behavior of the model to be better and better. Adding more participants should also improve the quality of the alignment target. So these are sort of three criteria related to the performance of the system using the alignment target. And then there's two important criteria related to more the legitimacy and the trust of the system. So one of them is
that the people affected by the model should have a voice or at least an ability to endorse the values of the system. There should be robustness to the signal so it's not easily amenable to attacks by third parties, malicious third parties, and ideally it should be auditable. So there's a way, you know, you're using a system, there's a way to look under the hood and see what are the values that are getting operationalized by this system. And so that mix of performance and trust sort of captures some of the desired properties.
Now we're not yet at a stage where we have a way to take these properties and like turn them into equations and algorithms quite yet. What I appreciated is also people being willing to explore completely different formalization than like a scalar reward function in terms of eliciting reward. And so the same authors actually proposed this notion of a value card, and the property of a value card is that the values are encoded in natural language. And so the distillation of all the preferences are encoded in a templatized card. It starts with a scenario, in this case written a little bit small maybe for some of you, but you know, scenario is roughly like, oh, I have a twin 10-year-old son and wanting to get them to do their homework rather than spend all their time on the screen. Familiar for many parents of pre-teens or teens. And then there's a context, a context for where the question is asked. Is it because the person wants parenting advice or the person actually wants to avoid a conflict? Most likely both. And so out of that, through natural conversation, you can elicit essentially like a set of core values that underlie how decisions would be made.
Now, there's still a lot of things to be figured out in terms of how do you go from like a general conversation, how do you prompt in a way to get the right salient value information, and how do you distill that into a template? But I think it's at least worth opening up the door to using things like natural language for values rather than scalars. And the reason being, when reinforcement learning was founded as a field, the only thing machines understood were numbers. Today, machines understand language. That's just where we are today. And so there's a way that we should take that to our advantage and use the fact that we can interact with machine in a much richer way to indicate what we want in a much richer way as well. So I'm not necessarily saying this is the end of the conversation in terms of how to elicit values, but I encourage all of us to embrace the fact that we could have much more rich representation of the alignment target. And you know, people may be a little bit skeptical about this notion of value cards, but actually, like when model cards came out, people were a little bit skeptical who would adopt model cards. And in fact, today, you know, looking at a recent survey paper, there's over 32,000 model cards that have been released. It's now completely routine for people releasing model to include a model card, sometimes a system card when there's lots of different artifacts. But that has just become part of the practice of how we do this work. And so similarly, at some point, there may be other artifacts that arise that can qualify the values that underlie a system.
Now let's talk a little bit about how we reconcile these values into a target. This is a complicated one. And as I, you know, looked at that, I think what I wanted to figure out is like, what are the existing mechanisms we have to reconcile values, leaving alone AI system? You know, we live in society. This is a problem that we face frequently. So like, what are the different mechanisms we have? So one of these mechanisms is to set up institutions and grant power to these institutions, right? So you know, many of you are here today for this RL conference. Many of you sent your papers to the RL conference, and in sending your paper, you accepted that there was going to be a committee that is going to determine what is good work and what is less good work, and you've granted that authority to that institution. You can argue with your institution, and we can have long discussions about why review or two was wrong, but nonetheless, you've decided, we've all decided to abide by the authority of that institution. So that's one way is to create some institutions that hold power over certain decisions and hold power to determine what is valuable.
The second way is through deliberation. So through conversation, you can also arise this. So the committee that's been empowered with deciding what work to accept doesn't just go like blindly behind closed door make this. They actually set up reviewers and they ask them for their opinion, and these reviewers get to talk. Sometimes they talk to the authors in case they got something wrong. And out of that deliberation arises this notion of what's a good paper for this conference this year. Not that the work that was rejected is necessarily not good, just didn't meet the value criteria of this specific committee. Now, another very popular mechanism and one that is very scalable is aggregation system. So these ones tend to be more grounded to just like an algorithmic way to determine this. Voting system is probably the classic system out of that. So I'll talk a little bit more about aggregation in the context of AI system and that that might look like. Another common approach is through markets, which again works quite well when you have economic value associated with specific items. And the last one, and I'll spend a little bit of time on that one also, is this notion of pluralism, is this notion of actually accepting that maybe we're not going to reconcile all the values, and maybe there's a lot of beauty in having different value system coexist, and what does that look like to enable that. So let me say a little bit more about those two, starting with aggregation. So I went back in time to try to figure out like what are properties of different aggregation system. I don't know how many of you have seen Arrow's theorem. I had heard about a long time ago, forgotten about it, but it was fun to get back in.
So, back in the 1950s, Kenneth Arrow started looking at like what are properties you would want for a voting system. Maybe something we want to revisit in 2025. But nonetheless, you know, at the time, you're not going to like the conclusion of Arrow's impossibility theorem. The assumption was, you know, subject to a few desiderata that you had on your system, like you want the ability to work for any possible set of individual preferences. So that's the unrestricted domain. You want the fact that there's no single individual that should always determine the outcome. So that's a non-dictatorship. If everyone prefers option A over option B, then the group ranking should also be option A over option B. That's a Pareto condition, and so on. You also want the ranking of A and B not necessarily to depend on other irrelevant factors that are not captured. And then you want some consistency or transitivity in terms of the different preferences, so no cycles in terms of the preferences. So let's say you want your voting method to adhere to all these properties.
Well, unfortunately, you know, Arrow showed that there's, you know, if there's three or more options on, you know, three or more options on the ballot, unfortunately, that's not going to be possible. There's no voting system that's going to satisfy all of these conditions at once. And any system that satisfies like all the other condition, there must be a dictatorship. There must be one person who decides. So that's not really encouraging. And what does that mean for AI system? I mean, you know, there there there's there's always, of course, a big step from like the very formal condition that were assumed in the theorem versus the practice of this. But but roughly what I question it raises is like there may be no attempt in terms of aggregating value into a single reward function that is possible unless we have a designer that chooses some value over others. And another way to think about it is when people disagree, there's no neutral universal way to combine all their preferences. And so in some ways this is not satisfactory. On the other hand, it just acknowledges the reality that there's no neutral way to determine reward function, and we have to keep that within scope.
And so I love to oscillate between the theoretical and the practical. It's like, well, we get this done with LLM. So how do we do this? Well, in practice, I think one way to think about this is that the designer's wishes are very much encoded in the annotation guidelines that are provided to the human labelers, and those annotation guidelines in many ways will determine what types of things to encourage in the system and what type of thing not to encourage. Typically, all the labels are kind of thrown together to train a reward function. There's quite a bit of work that is done to clean up to make sure to take away sort of outliers, but other than that, everything's kind of thrown in together. And then the model is trained against all of that with this RLHF, and it's evaluated against the benchmarks. In the case where the behavior is subpar, what does it mean to subpar? Well, again, the designer comes back in, you may call him the dictator, dictator comes back in, looks at the results, and says, 'I don't actually like the outcome of this. Let me change my annotation guidelines,' and the cycle continues. And so in this sense, this sort of sidesteps Arrow's theorem, but it actually, you know, does make dictatorial choices about like which evaluation count, how do you set these annotation guidelines, and when you stop training the system. So it's embedded in there.
Now in practice, there's a lot of multiple phases of this. So interleaved rounds of acquiring data, training the system, looking at it, going back to it. And as you interleave these two phases, there's different versions of feedback that can be exploited. But often what happens is this model and data co-development happens together. So the model that's been trained is used to generate the data. And so it becomes hard to tell actually like is the data reflecting the true values or is the data reflecting the values that have been put in the model already. And so that line gets blurred in terms of whether we're reflecting the values of the humans or we're actually imposing the values in terms of what we solicit from the humans.
So, you know, still probably a lot to be discussed about what what I'll so-called this sort of dictatorship mode of aggregation. And you know, it it gives me a little bit of hope to look at the pluralistic alignment question, though there are two more more questions than answers. But that but really, you know, right now there is a state of affairs where the alignment of large AI system which we are using every day are in the hands of a relatively small set of people. That is just a fact today. And this is not a totally new situation. You know, this a natural extension of of, for example, how preferences were set in recommendation algorithms for the last 20 years or so. But but I think we have an opportunity to move the target beyond that and explore new ways that we can actually look at a diverse representative sample. And so this might be my, you know, public service announcement number two for the for the day. You know, I think we need to continue to champion the fact that we need a diversity of voices in this community. If we are to have a diversity of voices with their hands on the dials of the preferences for these system, we need to get a diversity of voices into the room from an early phase and bring them into our community. We need people who have different lived experiences to be contributing to these systems. This may not be a popular opinion in some places this year, but I think we individually hold a responsibility to continue to uphold that.
There's a lot of questions of how to do it. And so, you know, public service announcement being made, I also think we need to look like pragmatically, technically, what does that look like? How do we live in a world where we have pluralistic alignment? So I started digging into this question a little bit, went into a bit of a rabbit hole, which I will like sort of summarize briefly. I think this there's sort of a few variants of pluralistic alignment that are worth considering. What's interesting is they apply in different settings. So it's not necessarily like one way to even do pluralistic alignment. Scenario number one is that you may be in a situation where you want to surface many different behaviors or many different preferences. So let the system satisfy multiple targets, and that particularly useful when you have a human in the loop for decision-making. So if your system rather than provide just like one answer, you know, I go back to my medical treatment system, rather than say this is the way to treat the patient, actually says like here are a few different options, then it provides a way for human to complement with their own preferences and knowledge the system. Option number two is to have sort of steerable pluralistic system. So you can add additional information, and that additional information can then determine the behavior of the system. And to some degree, this is what LLMs do today when you're prompting the system. And we certainly know there's like a whole culture of prompt engineering out there. When you prompt the system, it's one way to steer the behavior, to add external signal to steer the behavior of the system. Now in those system, you know, you're steering it within some some lanes, right? Like the lanes have been embedded into the system, and you have some ability to steer within that, not outside of that. And the third the third option is actually to have sort of a distributionally pluralistic model where multiple, you know, values can be considered. You may sample them in a representative way, and that tends to be particularly useful if you want to represent like a generative sample. So if you're actually developing an AI system that you're using to generate data, you certainly want to preserve that diversity. And so this notion that there's value intrinsic to diversity is something that can be considered in there. The whole phase is really quite early on. I think there's a lot more work to be done in terms of what that means for our RL system and in terms of sequential decision making, but at least it hopefully gives you a few ingredients that open open up the aperture of how to think about these problems.
One of the things we need to do is to create new data sets in order to do that. And as we're creating these data sets, we have an opportunity to enable more research into that field. So there's an interesting data set that came out recently that was specifically looking at, you know, preferences over LLM answers that represents a global population. And one of the things they did is not just they collected data across five countries and, you know, 200,000 different comparison, but they also explicitly included negatively correlated sampling to force generation of more diverse response. And so being thoughtful about how do we create data, it's something that we've done, you know, in terms of our own, you know, work in exploration, something that we have a lot of knowledge on that we may be able to bring to this community to encourage a proper exploration to generate more diverse values.
Now almost everything I've talked about is is what something that's called sort of like forward alignment. I just wanted to acknowledge for the sake of completeness, there's a really nice article that's a comprehensive survey of AI alignment that been released in 2023, and they they argue to have a framework that like includes the forward alignment and the backward alignment. And what falls under backward alignment are things like governance and policy questions. So I just want to acknowledge there's also all of that work to set up the system in which these algorithms are deployed that does have an influence. It's it's like a topic for another day. But I wanted to actually acknowledge that force, and to some degree that the reason I want to acknowledge that is because it it brings me to one of my last questions for today, which is what values do we inherit from our institutions? I talked a little bit about the impact, for example, of a conference committee in determining the value of academic or of research work, but that goes much broader than that in terms of like what gets funded, who gets to do the work, what work gets done. All of these questions are actually deeply influenced in some cases by the institutions within which we do this work.
And so I thought maybe it'd be useful if I talk a little bit about my own experience dealing with that in terms of not only inheriting value from an institution but also articulating value for an institution. And so over the last number of years while I was at FAIR, I think if there's one thing that I did above everything else is actually to be the curator for the value of that organization, to actually every day, you know, stand for specific values, be clear about what these were. It wasn't about telling researchers what work they should do, it was about telling them like, here are the values that we are going to stand for in the work that we do. So we picked six of these values back in 2017, and through that time the field of research changed incredibly, the organization changed incredibly, but we were able to maintain a consistent set of values. Out of these probably the one that is most the signature of this organization is the openness. But the others were there, and you know, I assume are still are still there and and caring how the work gets done.
Let me talk a little bit about openness because it was one that was very much near and dear to me. And over those kind of 10 years or so, you know, it allowed us over that time to release over a thousand different research artifacts from the work that we did at a time where this notion of openly releasing research was not necessarily the modest brand for many other industry labs. And I do think that encoding it as a core value of the organization and communicating about it consistently helped us keep that path open. And so sometimes being explicit and stating what your values are helps you build a strategy to realize those values. And you can do it at the scale of like a reward function, but you can also do it at the scale of, you know, 100 plus person organization. One of the things that's interesting when you do that is that you also have to look at the broader ecosystem. And so because, you know, FAIR was a research organization that operated within a large for-profit corporation, you have to build bridges between the values of these organizations. And one of the ways we did this is being very explicit about articulating in what way does this bring economic value to the company? What does bring brand value to the company? Knowing that these are tangible things that matter in terms of the company's strategy. So we were able to build the case that having broader accessibility to the models could accelerate the innovation. It could help ensure that the products were better in terms of performance, but also that the products were better in terms of safety and responsibility. And releasing something openly when you're in a research phase gives you a lot of learnings that you can then carry out to the to the product. And so I think that was, you know, what's wonderful for me was to see like starting from the value, using that to create the work, and seeing then what's the outcome of the work sort of come close circle.
We had the the beautiful pleasure of seeing some of the models that we build, including the the Llama 1, Llama 2 models, get in the hands of people around the world. People in Brazil who are using Llama to build AI-powered math teachers, people in Africa who are using it to inform small-scale farming, and so on and so forth. Tons of example of people who are solving problems in their own communities in ways that matter to them. So sort of closing the loop on this notion of pluralistic value is that when you share those artifacts openly, it enabled this pluralistic value to exist. There was no way that necessarily a large US corporation was going to devote the resources to doing that, but it was going to build a model that served its own purpose and then could share it with others.
Now it's not always that easy, and sometimes there's a real tension between your values. And as much as you know, I love to stand for openness, we did come across a lot of cases where there were another value that were in conflict with that, and we had responsibility clearly articulated in our values. At one point in late 2023, we built a model called AudioBox. Great model, impressive how from a tiny segment of your voice recording it could clone that voice and essentially be used for voice synthesis. The challenge of releasing that model in an open-source way is you are just opening up the floodgates in terms of voice cloning, and that didn't line up well with our value of responsibility. And we were unable to build sufficient mitigations to release the model safely, whether in terms of watermarking and so on and so forth. So it's a model we didn't release completely openly. We shared it with a small group of academics with the intent of working on better watermarking techniques during that period of time. So I also want to acknowledge, you know, these values every day come into tension for each. In this case, you know, we had to make a decision of how to do it and how to resolve that.
Now, I and I'll finish with this. I had a chance to work with a few people to think a little bit more holistically about this notion of risk and and can we have like a common framework. A lot of how we made these decisions internally linked to Meta isn't, you know, we wouldn't necessarily like expose all the books of how we handle the tension between all these different values and how we make decisions. And so with a with a group of academics, we worked on developing essentially like a framework to assess marginal risks for models. And that gave us at least like a common language. I think there's still more work to be done, but just want to draw your attention to this. One thing that I think would be incredibly beneficial for a field is maybe we're not all going to argue or agree on what's valuable, but maybe we can at least align a little bit more easily on what's really dangerous. And so starting from identifying a common set of threats, if we could have like a taxonomy of harms, and right now there's a few that are out there that have been proposed. I haven't yet seen convergence to a common taxonomy of harms. If we could get there, I think there would definitely be benefits to doing that.
But if you can clearly identify what are the threats, if you can measure those threats, it's it's easy to talk in hyperbole not just about AGI but also about the risks of AGI. But I think as scientists we have a responsibility to ground that towards things that we can measure and observe. And so if we have a way to not only identify vectors of harm but actually to measure the risk, to measure the existing risk, some of these things like misinformation, voice cloning, which certainly are not things that are desirable, do exist absent the latest generation of AI models. So measure the prevalence of risk at baseline. Measure the marginal risk that is increased when certain models are released. Look at the effectiveness of your mitigations. Are your mitigations able to reduce that marginal risk to a tolerable level, and then see whether your remaining risk is above an acceptable tolerable threshold. That may give us a way together to look at how to break these models safely out, whether it's with open-weight models or just API type of interfaces.
I'm going to close out, and we should have a few minutes for answers. I think, you know, if I if I finish with the last a last push for you is to really embrace the fact that RL is not just a technical tool. RL today is a force that is shaping society, and you know, we've seen a lot of properties of these systems. I think we have a responsibility to engage with what we mean in terms of rewards and in terms of value. We can still rely on Bellman for some cases, but we have to figure out how can we build these systems in the way that they enrich our values. In some ways, you know, the goal is going to have to be not just to train machines that reflect us. The machines are going to shape us, and we will shape the machines. And so reinforcement learning, AI more generally, I think we we have to accept that we're partners in all this. We are partners in expanding the possibilities. We're partners in defining what has values and what we care about. Thank you.
[Applause]
