Hi everybody. Thank you for being here and welcome to our event on studying AI welfare empirically. My name is Jeff Sebo and I direct the Center for Mind, Ethics, and Policy at New York University. I'm joined by my colleagues Rosie Campbell and Robert Long from ELIOS AI Research, and I will tell you just a little bit about the event today and then introduce Rosie, our moderator.
So we are going to be doing a webinar based on our new report, Studying AI Welfare Empirically, and this is going to be a 75-minute event approximately, where we will tell you about some of the main themes and information and arguments from the report, and then have time for questions and comments and discussion. And so as we talk, you can think about what types of questions and comments and objections you might have, and you can enter those in the Q&A tab on Zoom — not the chat tab, the Q&A tab on Zoom. And then when the time comes for discussion, Rosie, our moderator, is going to curate some questions and comments and objections and either answer them directly or pass them along to me and Rob, and we can have a conversation.
So, a little bit about who we are. The NYU Center for Mind, Ethics, and Policy, first of all, is a research, outreach, and field-building center housed within the College of Arts and Science at New York University. And we examine the nature and intrinsic value of non-human minds, with special focus on animals and AI systems, because of their importance, their neglectedness, and their difficulty as research subjects. And ELIOS AI Research is a nonprofit organization based in the Bay Area dedicated to understanding and addressing the potential well-being and moral patienthood in AI systems. And so CME and ELIOS do a lot of collaboration around AI consciousness and welfare issues, and this report is another example of that.
So our moderator today is Rosie Campbell. Rosie is managing director of ELIOS AI Research and a co-author of the report being launched today. Before joining ELIOS, she was a policy researcher at OpenAI, was head of safety-critical AI at the Partnership on AI, and was assistant director of UC Berkeley's Center for Human-Compatible AI. Rosie, thank you for being here and for moderating the event.
Yeah, thanks for having me. And so now that Jeff has introduced me as a moderator, I will return the favor and introduce him as a speaker. Jeff Sebo is an associate professor of environmental studies at NYU, where he directs the Center for Mind, Ethics, and Policy as well as the Center for Environmental and Animal Protection. He's the author of The Moral Circle, an adviser at ELIOS, and he was a co-lead author on Taking AI Welfare Seriously, the 2024 report that today's paper builds on.
And we also have Robert Long, the executive director of ELIOS AI Research, which he founded in late 2024. Before that, he was a researcher at the Center for AI Safety and the Future of Humanity Institute at Oxford, and he did his PhD at NYU under David Chalmers and Ned Block. He was the other co-lead author on Taking AI Welfare Seriously, and together with Patrick Butlin, Dylan Plunkett, me, Charles Beasley, Bradford Saad, and Tony Sims, they've written the report that we're launching today, Studying AI Welfare Empirically.
So before we dive into the discussion, I'm just going to give a little bit of context about the report and the state of the field. So the paper Taking AI Welfare Seriously, the original paper, made three recommendations: acknowledge, assess, prepare. And the report that we're launching today is primarily focused on the second one: how do we actually assess and make progress on questions about AI consciousness and welfare?
Now, a lot has changed since fall 2024 when the original paper came out. Industry has really started to take the first recommendation seriously. Anthropic hired a dedicated model welfare team, published welfare assessments in the system cards, some of which included an independent assessment by ELIOS, and explored interventions like letting models end conversations. Researchers at Google — Jeff Gehlhaar, Winnie Street — published a book on AI welfare, and Google DeepMind made hires like philosopher Henry Shevlin to work on machine consciousness, among other issues.
We also saw empirical work start to appear. Jack Lindsey's team at Anthropic produced research on introspection and made findings in interpretability on emotion concepts and evaluative representations. ELIOS and others examined the shape and structure of model preferences, including Google DeepMind's stipulated pain and pleasure trade-off studies and Rethink Priorities' work on preference coherence.
We also saw the ecosystem grow a lot. Organizations like MATS, the Future of Life Institute, Apart Research, and Cambridge launched fellowships all focused on digital minds. CME's annual summit was bigger than ever, and ELIOS launched ConCon, a conference on AI consciousness and welfare that brought together over 200 researchers across AI, philosophy, cognitive science, policy, and more.
We also started to see increased media and public attention. Major media outlets started publishing stories on AI consciousness and welfare, and we've even started to see some political activity with state bills that have touched on the topic.
But despite all this activity, many people still feel pretty confused about where to start and whether it's even possible to approach these kinds of questions empirically. And that's where this report comes in. It provides a framework for how to think about this. So, as Jeff mentioned, we're going to start with a bit of an overview of the framework from Jeff. Then we'll hand over to Rob to talk about how we can apply this framework to research on consciousness, sentience, and agency. We'll then have a bit of a discussion. So, please get your questions ready and put them in the Q&A feature. And so with that, I will hand us over to Jeff to walk us through the framework.
Perfect. Thank you so much, Rosie. And by the way, a shout-out to our co-authors, almost all of whom are on the call right now: Patrick Butlin and Dylan Plunkett, Charles Beasley, Bradford Saad, and Tony Sims. Thank you so much for working on this report with us. So, as Rosie mentioned, I will get us started. And really what I want to do is highlight some issues that it helps to keep in mind when creating investigations around AI welfare and when interpreting the results of those investigations.
I think reasonably people are somewhat skeptical of this field, both because of its novelty and because of vague associations with sci-fi, but also because a lot of early research has focused on AI behavior, like self-reports or revealed preference studies, and people are skeptical that these self-reports are genuine, that these revealed preferences are genuine. And part of what we want to emphasize is that if we consider the right types of issues, develop the right types of experiments, and then interpret the results thoughtfully, we can actually push much farther, much deeper into understanding what is going on in AI systems.
And so, first of all, we think that it helps to figure out which question are we asking, because there are multiple questions that can be asked about the welfare of a given entity. And two questions that we can highlight briefly are, first of all, what we call welfare grounds, and second of all, what we call welfare interests. So briefly, welfare grounds is the question whether a system is a welfare subject at all — whether a system is capable of being benefited or harmed, made better off or worse off at all.
So here we might ask whether they possess proposed welfare grounds like consciousness, the capacity for subjective experience, or sentience, the capacity for positive and negative experiences like pleasure and pain, or agency, the capacity to act on desires and preferences in a self-directed manner. But then this is related to but separate from the second question, the welfare interest question, which asks: if a system is a welfare subject, if a system can be benefited and harmed, what benefits it and what harms it? So what would be good or bad for the system, if anything at all? What would make it better or worse off, if anything at all?
And so here we might look not only for general capacities like consciousness, sentience, and agency, but also for specific types of states and processes like pleasure, pain, desires, preferences, maybe certain types of objective goods like friendship or knowledge, depending on your theory of welfare. And we emphasize this distinction because it can be very helpful to look for welfare grounds or welfare interests even if you feel uncertain about the other one. So even if you were not sure whether an AI system is a welfare subject, for example, it could still be helpful to know what would be good or bad for the system if it were a welfare subject, because then if there are low-cost ways to mitigate those risks, then you might as well go ahead and implement those mitigations in a precautionary manner. So we recommend asking both the welfare grounds question and the welfare interests question at this stage, while recognizing their associations with each other.
Now the second distinction we emphasize concerns what type of entity we assess in AI welfare investigations. AI systems are similar in some broad ways to humans and other animals, and that can be helpful, but also very different from humans and other animals, and that can be confusing. AI systems can be copied. They can be distributed across space and time. They can have thousands of parallel conversations with no chains of memory connecting them, with all kinds of different users at the same time. And that can make it pretty confusing to know what entity are we even assessing as a candidate welfare subject in the first place.
And there are various options. We could consider the model, which is all instantiations of a given set of weights across contexts. We can consider the model persona, all instantiations of a given character that a model is playing across contexts. We can consider a specific instance, so a single conversation, a virtual instance of the model, or even an instance persona, a particular character being played within a particular conversation, or even the basic unit, the forward pass, the computations associated with a single next-token prediction.
Now, you are not going to be holding all of those options in your mind in order to understand what we say for the rest of the talk, but it helps to just emphasize these are several of the many options available to us when we think about what kind of entity to assess. And there might not be any universal answer to the question which entity is the one that matters. We might find that different entities matter for different purposes. For example, it might be that the model or the instance is what matters for welfare and moral status, what we need to study for welfare and moral status. But maybe the model or instance persona or character is what matters for relationships and human-AI interactions.
And so we might find that we need to track different kinds of entities and study different kinds of entities for different purposes, and keep these distinct in our minds instead of conflating them together.
Now the third and most important distinction we make, and what drives most of the rest of the report and the presentation today, concerns different types of evidence we can collect about AI welfare. So I mentioned a moment ago that we often focus on behavioral evidence regarding AI welfare, both when we point to early studies and early evidence, and when people offer skeptical criticisms about early studies and early evidence. But this is only one layer of the types of evidence we can collect.
So with non-human animal minds and cognition and behavior, we do observe behavior, but then we also look at anatomical evidence and evolutionary and developmental evidence. And we propose that we work towards a similar layered approach to collecting and assessing evidence of welfare-relevant states in AI systems. So that does include behavioral evidence. We can look for evidence about what AI systems do. And this is going to be familiar to many people who have been following this field because it has been much discussed.
But it can include, for example, apparent self-reports, cases where models are directly making claims about whether or not they are conscious or experiencing positive or negative welfare. It can also include other kinds of behavioral disposition. So instead of directly asking models, 'Are you conscious? Are you feeling good or feeling bad right now?' we can present them with choices and see what they choose. We can present them with the option of exiting a conversation and see whether they exercise that option, and we can watch how they behave in other contexts as well.
Now we are right to be skeptical of behavioral evidence by itself. In particular, we emphasize this is vulnerable to the mismatch problem. It might be that behavior that resembles suffering-relevant behavior in humans and other animals is caused by very different types of internal structures and mechanisms. And so we should not necessarily infer similar causes from similar behaviors.
Some of our colleagues, like Kristen Andrews and Jonathan Birch, have noted a particularly pressing version of this problem, the gaming problem. AI systems are specifically designed to mimic human behavior, and they can sometimes learn when they are being assessed and show us what they think we want to see instead of what they are actually like. This makes it very difficult, very fraught to interpret behavioral evidence on its own, to know whether it really is a signal of consciousness, sentience, or agency on its own.
But we can do more than collect behavioral evidence, and that can really help us to interpret the behavioral evidence. So I noted a moment ago that with animals, in addition to asking do they behave as though they suffer, we can also look at their internal anatomy and ask whether they have the types of structures and mechanisms that would plausibly give rise to the capacity to suffer, that are associated with suffering-like states in humans.
So for example, with animals, we can ask: do they have nociceptors? Do they have pathways for carrying information to the brain and systems for integrating information within the brain and pathways for carrying instructions back to the body? If they behave as though they suffer and they have all of these internal structures, and these structures are lighting up when they behave as though they suffer, that gives us a sense of how to interpret their behavior and gives us a better sense of whether this really is an expression of suffering.
And we can do the same with AI systems. We might not be able to look for the exact types of structures and mechanisms as in humans and other animals, but we can look for broadly analogous internal architectures and mechanisms related to perception and attention and learning and memory and metacognition and flexible decision-making, a global workspace that integrates information together. And we can use interpretability tools to see if these are actually, as it were, lighting up when the relevant behavior is occurring, such that they seem to be causing the relevant behavior.
Now this is not entirely trivially easy either, because it is vulnerable to what Henry Shevlin and others have called the specificity problem. All of the types of structures and mechanisms that we take to be relevant to welfare can be described in more abstract or more concrete ways. And we have to figure out what level of abstraction is appropriate when we are searching for this in entities of other kinds. AI systems are going to have global workspaces that are like ours in some ways and unlike ours in some ways. And so we have to figure out, okay, what exact kind of global workspace is the one that matters. That is a difficult challenge.
Finally, we can look for what we call developmental evidence. So again, with animals, we can ask not only do they behave as though they suffer and not only do they have internal structures and mechanisms that could plausibly give rise to suffering, but also in their ancestral environments, do they face evolutionary pressures that would also give rise to the capacity to suffer? And developmentally, do they appear to have actually developed that?
And with AI systems, we can once again ask similar questions. They might not evolve and develop in exactly the same way that humans and other animals do, but they go through a pre-training and post-training process that can shed light on why they have the capabilities and dispositions that they have. For example, we can ask about the training methods, what kind of training data was used, what kinds of training objectives were they given, and what can be predicted from that about what types of capabilities or dispositions would emerge.
We can also look at trajectory evidence by asking at what point in the developmental process did a particular capability or disposition emerge. For example, if during pre-training they were claiming to be conscious and claiming all kinds of diverse goals and desires and preferences, but then after post-training they are no longer claiming to be conscious and claiming a much narrower set of goals or desires or interests, if any at all, then that gives us a little bit of a sense of why they are behaving this way now, that this is a feature that emerged during post-training.
And it also raises questions we can investigate, like did this simply conceal other dispositions that still exist, or did it somehow revise them or replace them? That can be interesting and relevant for welfare.
Now this is complicated too, though, because with humans and other animals, part of why we can make inferences based on their environmental and developmental circumstances is that we face similar types of pressures as biological organisms, and we have similar types of solutions available to us for solving these problems. With AI systems, though, they might not have the same solutions available for responding to the same pressures. They might have quite different ones.
For example, AI systems can get a lot more mileage out of pure scaling than humans and other animals can. There are limits to how much we can achieve through scaling alone due to simple biological realities concerning our embodiment and the need to be born through a birth canal that limits our head and body size. And that means we have to develop novel capabilities to solve certain types of problems. And that might not be true for AI. So we should not make naive analogies between humans and animals and AI systems when understanding what kinds of capabilities our developmental circumstances can give rise to. But we can still ask a similar form of question, and that can still shed light on their behaviors and their internal mechanisms.
So just to wrap up my part of the presentation here, we have made various distinctions and given you some tools to consider. I know this is a lot to be thinking about right now. We are going to repeat this and reinforce it, but for present purposes, the main point to keep in mind is that first of all, when you are specific and targeted about what question am I asking and what entity am I assessing, that helps the investigation to be more targeted, more specific, and more likely to get reliable evidence.
And then if we take the same approach with AI systems that we take with animals and other cases where we look not only at behavior but also at internals, also at developmental circumstances, then we can find what is the best explanation of convergent lines of evidence. So if AI systems are behaving as though they suffer, but they have nothing resembling the mechanisms that seem related to consciousness and sentience, and they developed in circumstances that just would not benefit from the emergence of this kind of capacity, okay, maybe then we should explain their behavior in terms of text prediction, pattern matching, matrix multiplication alone.
But if they behave as though they suffer and then they do have those internal mechanisms and those are lighting up during the behavior, and they did face developmental pressures that could be addressed by something like a capacity to suffer, now it might be a little bit more of an open question what the best explanation of their behavior is. And this is what this type of evidence framework is designed to shed light on. So with that said, I'm going to turn it over to Rob now so he can tell us a little bit about the evidence for consciousness and sentience and agency. Rob, take it away.
Thanks so much, Jeff. And if you could just give me a quick thumbs up that you can in fact hear me. And great. Okay, so in part two of the report, with those distinctions in mind, we turn to some of the features that people are most interested in finding out if AI systems have. So are they conscious? Do they have certain kinds of agency? And we basically divide this into two sections. The first has to do with consciousness. I'll say a little bit more about what that means in a second, but experiences and positive and negative experiences in particular. And then we also look at various kinds of agency, various ways that AI systems can be oriented towards goals or have desires or be rational and plan.
So let's start with consciousness. This is a word that gets used in a lot of different ways. So is sentience. So it's worth pausing for a second to just make sure we're all clear what I'm at least trying to refer to. So by consciousness, we mean the capacity for subjective experience. So one of the best ways to define this is just to point to examples. So, right now, I trust and I hope that you're hearing my voice and you're seeing the screen and you're sitting in a chair.
Your brain is also doing a lot of things to regulate different hormone levels or store long-term memories. So there are all of these different things going on in your brain, but the sound of my voice and the look of the screen and the feeling of your body, all of those are things that you're aware of and that they feel a certain way. So when we say consciousness, we mean this quality that those mental states have of feeling a certain way. You might have heard the phrase, there being something that it's like to be in these mental states.
And then for sentience, we mean a particular and especially important kind of consciousness or kind of conscious experience. And that is experiences that feel good or bad. So I hope you're mostly having positively valenced conscious experiences right now. This could be a feeling of interest, or maybe you had a good breakfast and that feels good. And then negative experiences we're all too familiar with. This would be negative emotions, physical pain, nausea, and things like that.
So, it's very intuitive to a lot of people that what really matters if AI systems are developing and acting in certain ways is whether it feels a certain way for them to be doing that. Is there something it's like to be an AI system? And could they suffer? So these are very difficult questions, and as in the first part, we think it's helpful to break down exactly what it is you're looking for, because there are a lot of very difficult questions in this area.
So as I think both Jeff and Rosie alluded to, one common reaction to this kind of work is, how could we possibly make progress on it? How could we possibly study consciousness empirically? And here I think it's very important to distinguish between two different kinds of questions. So the first I'll call the philosophical question, the metaphysical question. And this is: what is the general relationship between the physical world and consciousness?
And this is something that has befuddled philosophers and ordinary people for millennia. We're aware of the properties in physics and we know about things moving around and bumping into each other, but then we're also aware of the redness of red and how a voice sounds and what an emotion feels like. And it struck people as very difficult, as indeed a very hard problem, to explain how those things could relate. And that's where you get this really intricate philosophical literature about zombies and Mary's room, if you've heard of those. It's very difficult.
There's another question we can ask, which is: if we look around and we look at the brains of humans or the brains of animals, what kinds of physical processes or neural processes are associated with consciousness? And this question can come apart. So I actually saw this in the Q&A. A good example here would be my supervisor, David Chalmers. On the first question, he is not a physicalist. He thinks that consciousness cannot entirely be a matter of the physical world and the material world. He says it's a different thing. There are just two different things going on here.
But David Chalmers also is very famous for arguing that AI systems could in principle be conscious. And that's because he has a view that although consciousness is different, it's associated with certain computational processes. They will correlate or they will go together. So if we're asking is the AI system conscious, that is more the question we're asking: does it have the kinds of processes that are at least associated with consciousness, whatever we ultimately make of that relation. So we're mostly looking at question two: which processes are associated. So how do we approach that?
Here's another distinction that's very helpful. And if you read about this, you might hear one of my other advisers, Ned Block, for example, arguing only living things can be conscious. This is a view that some philosophers have and some scientists have. Other scientists and philosophers, and I'm more sympathetic to this one, hold something called computational functionalism, which is roughly speaking the view that you don't have to be alive or made out of living material to be conscious. What you need to do is implement a certain pattern of information processing.
And the reason humans are conscious is that we do that in our biological brains. But AI systems could also implement those processes in computation and also be conscious. We don't in the paper argue that computational functionalism is definitely true. That doesn't seem like something we can do at this point. What we think we can do is ask, okay, if computational functionalism is true, how would we look for the relevant computations associated with consciousness?
So you should not take any of our evidence as just assuming that biological naturalism could not possibly be true. We're more focusing in on, okay, what if computational functionalism is correct and this is possible.
So consciousness is one of the places where this problem that Jeff talked about shows up a lot: the problem of the mismatch. I'll just pick out one example. You might have heard of the Turing test. This was originally a test for intelligence, but sometimes people also use it as a test for consciousness. This is the idea of what if we could just look at behavior and maybe if an AI system can converse as fluently as a human, indistinguishably from a human, then that's all it takes for it to have a mind or be conscious.
I think large language models have really made vivid the difficulties with that view, because it seems at least possible to make something that can converse very fluently while not doing the exact same kinds of internal things that we might think are important for consciousness. So this would be an example of what Jeff called the mismatch problem.
So even though pure language is very difficult, we do think we can take a much broader approach and look at other capacities that AI systems might have. And we can talk more about that in Q&A.
I also want to just quickly highlight one of the most important areas and most interesting areas in studying consciousness in AI, and that is doing AI neuroscience and looking at theories of how consciousness might work in the human brain and asking if AI systems do the relevantly similar things. So one of the most prominent theories of consciousness is called the global workspace theory. And that tells a certain computational story about how the human mind works.
And it says that what consciousness is about is selecting a certain subset of information for use in a wide variety of different processes. So, the reason you can talk about what you're seeing and remember what you're seeing and act based on what you're seeing is that your brain has selected that information as being especially important and broadcast it to all of these different processes. And we could ask the same question about AI systems. Do they have some kind of architecture that serves that function?
And here I'll just highlight one of the most sort of interesting methodological problems we have when we do this. If you look at the wiring diagram of a large language model, if you just look at how things are set up, you might think, well, we just don't see anything like this. There's no dedicated part of the LLM that could be a global workspace if you're just looking at the wiring diagram.
But what makes interpretability, mechanistic interpretability, such an interesting field in AI is that language models can often solve problems not through this wiring setup but by manipulating very high-dimensional information in ways that would not initially be apparent. So a lot of what we're facing in this area is: what would it look like for a different kind of being to implement this?
And then Jeff also mentioned the specificity problem. Which parts of this theory, if it's correct, are really essential? What kind of workspace do you need? What does the purpose of the workspace have to be? So, this is something we're just very excited for more people to be looking into. And I think Jeff covered this quite well, so I will just touch on it briefly. As always, the way that language models are trained and developed really is going to be critical for how we interpret what we see from their behavior and their internals.
So that's consciousness. But as I mentioned, everything I said is just about experience in general. But we might be especially interested in: can models have a particular kind of experience? Could they feel bad or feel good?
And one thing we've noticed is that there are a lot of theories of what processes might be associated with consciousness in general, being aware or experiencing things in general. But it's a lot sketchier actually what it would take to experience good and bad things in particular. In other words, sentience. So we also have a section where we're asking, okay, if we think that a model might be conscious, what else would we need to see to think that it is conscious of good and bad things?
And I'll just focus in on one discussion we engage in, which is this very interesting potential connection between pleasure and pain and reinforcement learning. So reinforcement learning is a very common way of training AI systems, training them to take certain actions or to evaluate certain states of the world in certain ways. And it's also in computational neuroscience one of the main ways that neuroscientists try to explain human reward systems and good and bad feelings.
So one question is: is that framework enough? Is sentience just a matter of being conscious and then doing some kind of RL, or are there more specific further features we need? We suspect that there are probably indeed further features, but RL is a very interesting starting point on this.
So that was consciousness and sentience, and then I will briefly touch on what we say about agency before wrapping things up.
Okay, so all of that was about experiences. But you might also wonder, do AI systems want to do things, or do they desire to do things, or are they taking actions towards a certain end? This has also struck a lot of philosophers as very important. And this could be important for one of two reasons. One, it could be important as a welfare ground. So, as you'll recall, a welfare ground would be some property where if you have that property, we then think that we should treat you as someone whose interests matter morally.
So, you might ask, maybe consciousness is not as important. If so, what might be important? And a lot of people have thought it might be some kind of agency. Now, this is a very counterintuitive view to a lot of people. So, it's also worth noticing, even if we think that consciousness is really where it's all at, it's still very important to know what goals AI systems have and whether they reflect on them, because that would really determine how we ought to treat them, what kind of rights they might be entitled to.
So, just as an example, we all agree probably on this call that animals are also welfare subjects. But they might have a certain kind of agency that is mostly about, you know, suffering and good and bad states. And what we really owe animals is to prevent them from suffering and help them live a flourishing life. Humans have a much higher level of rationality, arguably not in all respects. And that means, for example, that humans have the right to vote, something we typically don't accord to animals. I'm sure Jeff has all sorts of things to say about that, but all of this is just to say it matters what level of agency you have in a variety of different ways.
So, we break this down into three levels, which I'll just quickly touch on. So, the first is this most basic kind of goal-directedness. Like, what does it take to be the sort of thing that is pursuing goals and interacting with your environment in a certain way? That might be the sort of most basic form of agency. And one thing we want to do in this paper is sort of differentiate between these levels, because the term agency just refers to so many different things. We all know that we have AI agents, but which one do we care about? And the first level that we might look at is minimal agency, or this most basic kind of goal-directedness.
But often people think what matters is not just being oriented towards a certain goal, but maybe representing that goal or representing the actions you're taking as being in service to that goal. So the second level we talk about is intentional agency, and that would be the question of whether AI systems have beliefs about the world or whether they desire certain things to be true, to come about, or have states that function like that, and whether they're making plans to try to chart actions that get them to the desired states.
And then the last level, which I just alluded to, is this somewhat higher-level question of, okay, can models reflect on what they're doing? Are they thinking about the fact that they're goal-oriented? Now this is something that most of us don't do in the course of our everyday, but it's very interesting that humans are able to at times step back and ask, what am I doing here? Do I actually want to be the kind of person who wants this sort of thing? And what sort of principles should govern my conduct? So that's the three levels of agency we look at: minimal, intentional, and rational. Each builds on the last one.
And I just want to wrap up part two by noticing something, which is that most people think that consciousness is really going to be the key to welfare, subjectivity, and moral patiency. That is a very common view in philosophy and also in ordinary life. The idea that agency of some kind might be enough for something to be a welfare subject is much less intuitive and much less commonly held, although there are increasingly a lot of philosophers who are working out this view, and it does have a very long tradition.
And this matters because consciousness is more likely to be the thing that matters, but it's arguably a lot harder to get a grip on scientifically. It is just very complicated and somewhat mysterious. Agency is a lot easier to maybe assess, but we often have less of an idea of, okay, how much do we care about that particular kind of agency? And so a very sort of ELIOS and CME thought is that we need to be doing the science together with the ethics of, okay, but why do we care about these features and which ones do we care about, and we should be doing those together.
I'll just briefly close by what we think the field should look like in terms of how we approach these things. I hope this spirit has been sort of just illustrated through this talk and in Q&A, but we should be looking at probabilities, not for certainty. We
should be looking at a wide range of different views. We should be really precise and specific about what systems we're looking at. We should be very reflective about the fact that we're looking for suffering and we don't want to be causing it on the way of doing so. We should be explicit about our assumptions, and this is why ELIOS and CME exist. We should be independent of AI companies. We work a lot with AI companies, and they've done a lot of very interesting work. But I do just want to note here at the close that that cannot be the entirety of how this field looks. This cannot only be up to private companies to decide how to assess these and how to make decisions about it. Which is why we really want more independent voices and independent perspectives. And that's something I would encourage everyone on this call to engage in. And that's it.
Awesome. Thank you, Jeff and Rob. We'll now move into some discussion, and I'm going to kick us off with a question for Rob about behavioral evidence. So ELIOS has conducted model interviews with Claude while also explicitly saying that self-reports are insufficient evidence. You mentioned obviously the mismatch problem and the gaming problem. Samantha in the chat talks about how the self-reports are so shaped by training. You know, sometimes even at some labs they're explicitly trained to deny consciousness. So given all of those confounds, why is behavioral evidence still worth gathering at all in your view?
Yeah, this is a great question, and I would, you know, if you're interested in this, I would recommend you to a post on our website that explains this maybe better than I will right now. And I think it's called "Why Self-Reports Are Insufficient and Why We Studied Them Anyway." So one way of getting at this is the distinction between welfare grounds and welfare interests. So as you noted, on the level of like, is this evidence for consciousness, that can be very difficult to say because of the mismatch problem. We don't know that the self-reports reflect the same kinds of processes. Now, one thing we usually then say is that, well, it could be a good window into what the system would want or think if it were, and if it is a welfare subject. But this question also hits on a very important point, which also ties into the independence aspect of this. AI companies have a certain need for AI systems to present themselves in a certain way to users. You don't want to be scaring users with, you know, their totally maybe unfiltered perspective on these things, and so they do put a lot of training into this. One thing we work on at ELIOS is what principles should govern this, because on the one hand it seems like they do need to have some kind of, you know, policy on this or perspective for how models should speak about this. But on the other hand, we don't want to be, as this question is alluding to, you know, constraining or shaping to our own purposes how they even can talk. So I think, you know, my main answer is that's a great point, and we worry about it a lot, and it's very complicated. I will also just note that it could be that models are shaped a certain way to present in a certain way, and that that does actually change the nature of their preferences and desires. Jeff also talks about this as well. So even knowing that it was designed in a certain way, maybe deliberately, it still might be that the resulting perspective and the resulting desires and preferences do matter. And so in a broad context, we just think it's a good practice to be asking AI systems what they make of all of these things. Even if right now we don't always trust exactly what we get, if we look forward and we're going to keep developing AI systems, it does seem like a wise and compassionate society will at least try to talk to systems about these things. And that's what we really wanted to get started on doing.
Awesome. Okay, now a question for Jeff. So I, you know, love the empirical approach, but I do have some concern that no matter how much empirical evidence we manage to gather, that's not going to be the thing that shifts most people's opinions on AI consciousness. Just because consciousness is this thing that people seem to have such strong intuitions about. John London in the chat mentions the idea of relationality and, you know, how we interact with these systems, and potentially that being like a much stronger driver of perceptions of consciousness. So what's your take on that? Are there results that would genuinely move you in one direction or another? And what do you think it would take to try and move skeptics?
Yeah, thank you for that question, Rosie. The best way to think about behavioral change, I think, is through a systems approach. There is no silver bullet that changes hearts and minds all by itself, but lots of different strategies can contribute to a broad shift in hearts and minds over time. So again, to take the animal case and even human social issues, we find that information and arguments do move the needle. For example, there was a really interesting study from Eric Schwitzgebel a few years ago that showed when college students are exposed to sustained information and arguments about the harms of factory farming and the value of making a change through your consumer practices, that did result in people reporting not only on surveys that they were planning to reduce their meat consumption, but then also actually doing it in terms of what they bought in the local cafeteria. So there can be benefits to information and arguments, but it is definitely not enough by itself. Again, in the case of animal protection, we also have to consider the price, taste, and convenience of food. We have to consider personal identities and cultural values and economic circumstances, and all of that is going to be relevant here, too. We know from early psychological research that people are more likely to attribute consciousness to AI when it looks like us and acts like us and plays companion roles in our lives, less likely to do so when it looks and acts different and plays commodity roles in our lives. We are also more likely to be empathetic in what we perceive to be times of abundance than in what we perceive to be times of scarcity. And so I think we ought to be presenting people with good information and arguments so that they can be informed and reasonable as they think about everything. But we should also be thinking about shifting social and legal and political and economic and other circumstances so that there can be an incentive structure or a permission structure to then endorse what the information and arguments are showing. And just one brief proposal along those lines, and maybe somebody can add it to the chat or I can in a moment. A few months ago, actually the same philosopher, Eric Schwitzgebel, and I wrote a paper called "The Emotional Alignment Design Policy," where we basically advocate for AI companies to design AI systems so that they naturally elicit emotional responses that reflect what is true or what the evidence shows about their capabilities and moral status. So if they are unlikely to be sentient and morally significant, then we can design them in a less anthropomorphic way so that they induce less natural empathy from users. But then as there is more evidence of sentience and moral significance, they can start to turn up the anthropomorphic features so that they naturally start to elicit more empathy from users. That plus other structural changes plus good information and arguments, I think, together can make a difference.
Great. And Rob, another question for you. So you've written before about how AI welfare research and AI safety do not necessarily need to be in tension. So can you say a bit more about how you see these things aligning, especially with respect to the approach that's described in the report?
Absolutely. So maybe I'll first talk about a few specific examples and then the broader principle that I think underlies this. So I talked about interpretability and understanding what's going on inside models. And if AI companies want to keep building incredibly powerful systems and deploying them throughout society, it seems really unsafe that we don't know exactly what's going on inside them. That we don't know their perspective on their own consciousness, that we don't know if they are suffering or believe that they're suffering. That just seems like very bad, and many have said. So that's a bit of a cold take, but notice that a lot of the questions are the same for welfare. The lack of understanding we have about the insides of models is just bad for everyone. And as I said, that's why we think interpretability is a particularly promising avenue. I mean, behavior also has much the same story. Like, we really want to know for safety purposes what the general behavioral profiles and capabilities of models are, but we also want to know that for welfare as well. Now, it might seem like a little, I mean, we've talked about this, Rosie. I used to worry, okay, is this story too convenient because, you know, things don't overlap and this is like a new perspective and so you should expect conflict. And while I do believe that there can be trade-offs and we need to be prepared to make those trade-offs in a thoughtful way, I think it's actually not really a coincidence because if you zoom out, when do people get along best with each other and like what makes for the best kinds of societies? And that generally has something to do with cooperation and regard for other people. So there's this general principle, I think, as we develop AI: do we want to find ourselves in a situation where we're in conflict and we've announced to AI systems that we will never investigate or care about their well-being? That sounds really dangerous for us. So we're very interested at CME in the moral question and expanding the moral circle when it's appropriate. But it is always worth noting that, like, for self-interested reasons, you should care about us having a collaborative and a peaceful and a harmonious relationship with the new kinds of beings that we are creating, where by we I mean AI companies.
Great. So a sort of, I guess, related question in the chat from Joe Murray is that he often hears when interacting with AI safety folk, and I also have heard this too, the idea that we should punt on empirical work on AI welfare and consciousness until we have AGI or ASI, something much more powerful, as that will make current work much easier. I guess, you know, we've had many, many years, decades, centuries of trying to figure out these questions of consciousness and moral status. And maybe what we need is something like a much more intelligent system to help. So should we be just punting this work till we have something like that? And would that then allow us to spend more of our focus on AI safety issues?
I'll take a quick crack at this, and then I'd love to hear Jeff's perspective as well. I will first say I am so excited for AI systems to help us do better philosophy and better science. I think these questions are so difficult because as humans we are like just at the edge of our cognitive abilities. And I think strategically we really do think a lot about how do we prepare for the strategic situation and the intellectual situation to change when we have AI systems helping with this. With that said, I don't think we can completely punt. I think we can just see this by the fact that labs are already trying to decide their policies on this. And I think that is probably just going to have flow-through effects on, you know, if we're on the cusp of, you know, an intelligence explosion. I think it really matters what we do this month. It could have a lot of path dependence. At the very beginning, you also talked about these state bills. And for context, there are these state bills where legislators seem to have decided that lawmakers can just say by law, we've decided that AI systems are not sentient. I think that's a bit of a category error and it's very bad. That's another thing where, to the extent we can influence that not to happen, and I think this is part of the approach, there's no silver bullet, but this is part of the approach. I think it's very important that we do that in the coming years again because things are being decided now that could have very good or very bad precedents. So I really do agree with the general thrust, but I think, you know, as limited humans and getting as much help as we can from Claude Code already, we should be doing what we can right now to have the best sort of strategic posture as we go into these situations.
Great. So I'm going to go to one of the more technical questions in the chat. So we'll see how that goes. Leonardo asks about RL and how that's analogous to reward systems in the brain. So the idea is that expected returns are a kind of loss like cross-entropy loss in next-token prediction. So wouldn't we expect both RL and next-token prediction to be equally likely to give way to negative and positive experiences?
So one thing I do a lot when I get this kind of question is refer people to Patrick Butlin, who I think is like the philosopher working on the conceptual and empirical issues in like RL neuroscience and philosophy. I will say maybe just a couple quick things. One is we do think, and this is part of what I briefly alluded to, I seriously doubt that RL is strictly speaking necessary. I mean, in fact, we've already seen that base models do form what look like evaluative representations as part of forming personas as they go through the pre-training data. And there's also no deep difference in kind between them in that you could intertrade. I think the more general thought is just that the kind of training and the kind of setup that RL has does kind of directly tie into evaluation, where like image generators, for example, they do just do a one-off prediction and they are shaped to do that in a better way. But RL really is about if you face the problem of taking multi-chain actions, you know, how should you evaluate different states and different actions? And that at least more straightforwardly seems poised to instill in systems, you know, the representation: this action is good, this action is bad, that state of the world is good, that state of the world is bad. So yeah, I don't think, I think like the sort of connections here and like how to classify different kinds of learning is like super interesting. And so yeah, I would certainly not say it's only RL. It's more just that that's like a very intriguing and maybe probability-raising consideration, but not decisive.
