Welcome to McGill University's philosophy, technology, and policy lab podcast. I'm Marina Nekrasova, and today with me is our guest, Professor Jeff Sebo from New York University, who will be giving a lecture to us later today on why we should be taking AI welfare seriously and what the next steps are in AI welfare research and policy. So, hello Professor Sebo. Thank you very much for joining us today.
Hello. Great to be here. Thanks for having me.
Before we start, I just wanted to introduce you with your official titles. So, Professor Jeff Sebo is associate professor of environmental studies and affiliate professor of bioethics, medical ethics, philosophy, and law. He is also the director of two centers at NYU, which is the Center for Environmental and Animal Protection and also the Center for Mind, Ethics, and Policy. And he's also the author of the book The Moral Circle, which I believe you recently published in 2025, so last year. Congratulations on that.
Okay, so Professor Sebo, can you please give us a preview on what you will be lecturing on later today?
Absolutely. So, this talk is going to be about next steps for AI welfare research and policy. And basically, what AI welfare research and policy means is people are starting to appreciate that AI systems are very complex, and this is raising questions about might AI systems ever become in their own way conscious or sentient or agentic. In other words, might they in their own way have feelings and emotions and experiences like happiness and suffering and desires and preferences. And if so, might we actually have moral responsibilities to them and might we need to build moral community with them. Now, up until recently this has felt like science fiction or a question only for the far future, but part of what my colleagues and I have been arguing over the past couple of years is that this is not only science fiction or a question for the far future. This is at least potentially a question for the near future. We have a lot of disagreement and uncertainty about the nature of consciousness and sentience and agency, the future of AI. And so, we are not really able to rule out a realistic possibility of morally significant interests in AI systems in the near future. And so, we argue that we should be doing research on this topic and we should be developing policies on this topic. So, my talk today is going to summarize the basic case for taking AI welfare seriously in that spirit and then is going to talk about next steps, thinking about questions like what is the public going to think about this topic? Are people going to over attribute sentience and moral status to AI systems and treat them with more concern than they deserve? Are people going to under attribute these features to AI systems and treat them with less concern than they deserve or might it be one in some cases and the other in other cases? And what follows for how AI companies should design, develop, deploy AI systems and how users should interact with them. How can we calibrate so that we can take an appropriate level of moral concern to our interactions with them. So, that is what the talk today is going to be about.
That's very ambitious and it sounds very interesting. In your paper taking AI welfare seriously that you have written with multiple co-authors, you begin by making a distinction between two very important concepts. So, one is moral patienthood and the other one is being a welfare subject. Can you define those concepts and could you discuss how they're interrelated with each other?
Absolutely. And this is a really important question because these concepts are often conflated and tangled together. So, being a welfare subject means being capable of being benefited or harmed. That means being made better off or being made worse off, having things go better for you for your sake, or having things go worse for you for your sake. And this matters for ethics because when you are a welfare subject, when you can be benefited and harmed, that means that you have morally significant interests and we have a responsibility to consider those interests when making decisions that affect you and at the very least to avoid harming you unnecessarily when making decisions that affect you. Now, being a moral patient, this is very closely related. This means being owed moral responsibilities. So, in philosophy we often distinguish being a moral agent and being a moral patient, where to be a moral agent means you owe others duties and to be a moral patient means others owe you duties. So, the question here is whether AI systems can be welfare subjects, that means capable of being benefited or harmed, and whether they can be moral patients, capable of being owed duties. The reason why these concepts are connected is because obviously, if you are a welfare subject, then at least in a certain sense you are also a moral patient because once you can be harmed, I ought to consider your interests and not harm you unnecessarily. But, these terms are not exactly the same. These concepts are not exactly the same because there might be other ways of being a moral patient, too. I might have duties to you for reasons that have nothing to do with your welfare. Maybe I have some duty to respect you or to engage in fair relations with you independently of how it affects your welfare. So, this is why it helps to understand the distinction between these terms. But, I think for purposes of our conversation today, for purposes of a lot of conversations about this topic, a big question that matters a lot is do they have a capacity for welfare? Do they have a capacity to be benefited or harmed?
I'm just wondering what would be an example of a moral patient without that capacity to be benefited or harmed? Because it almost seems like any example I can think of, the two concepts end up going together.
Well, that might be a sign that you accept a welfarist theory of moral patienthood, which a lot of people, including me, find very plausible. But, not everybody does. So, for example, some people accept social contract theories that say the source of morality is not that we have a duty to avoid harming each other, the source of morality is that we as rational agents make a contract with each other, and then we have an obligation to each other to abide by the terms of that contract, not because we would harm each other if we violate the terms of the contract, but rather simply because we have this relational duty to each other, or we have this duty to keep the promises that we have made to each other. And so, that would be an example of an account of why we have moral obligations to each other that has less to do with avoiding harm and more to do with something like keeping promises or having fair and just relations and for that reason some people think a certain type of moral agency and a certain type of moral patienthood can exist without welfare.
Okay, that's very interesting. I find it interesting that a lot of these things are so relational because it almost seems like to me there's a difference between our relational attitudes towards things and how we see things as opposed to the intrinsic capacity of the object itself.
Yeah, and to be honest I find myself having the same kind of intuition that it seems you do. I definitely think that relations can be morally significant including social contracts and other types of relations like familial relations, communal relations. At the same time I think the capacities like sentience and agency and the welfare capacity they bring is very important too and for me a lot of morally significant relationships and the duties and rights that come along with them it requires both the capacities and the relationships. This is not an either or phenomenon from my perspective and so I actually do think welfare is an essential ingredient for genuine moral patienthood as opposed to kind of instrumental or pragmatic or strategic reasons we might have for entering into certain kinds of contracts with each other and abiding by the terms of those contracts. So if you find yourself thinking, "You know, I get the conceptual distinction between welfare and moral patienthood, but I am not convinced that they can be thoroughly untangled cuz I just do think that welfare is essential for moral patienthood." Well, you would be in very good company because a lot of people agree with you about that.
I wanted to touch on one point that you recently just brought up in our discussion which is kind of these institutional linkages to this relational attitude we have towards moral patients or welfare subjects. So, I'm wondering whether the framework, I guess you propose, or I guess more broadly, I guess in the framework, just the way that we treat welfare subjects and moral patients in that article, do you think that that could be separated away from discussions about personhood, legal rights, political representation, or does it bleed into that? Like, is it possible to talk about moral personhood and welfare subjects without getting into those questions, especially for AI, or do they just immediately link over?
Yeah, that is a great question. And again, I think it depends on your substantive views about these issues, but certainly there are connections between the issues. So, one question is about what sort of moral status you should have, how we should treat each other independently of legal and political regimes. Then another question is what kind of legal status you should have. Then another question is what kind of political status you should have within a particular community. These are all, at least to an extent, importantly different. So, in the legal case, we typically think legal personhood is our foundational kind of legal status. This is what allows us to be recognized as rights and or duties holders under the law. This is what allows us to have our claims be heard by legal officials. The fact that we are recognized as persons instead of mere property. And then, similarly, political citizenship is the foundational status that we have in particular political communities. And this is what happens when we have actual membership rights that go beyond our basic legal rights to not be harmed or confined unnecessarily. When you have political citizenship, you may also have a right to reside in a particular territory, return to a particular territory if you leave, to have your interests represented by the political process, to if you have the appropriate capacities and interests, participate in those political processes. And so, these types of status are all linked because if you are sentient, agentic, morally significant, capable of being benefited and harmed, then you have a good claim to being recognized as a legal person and or a political citizen. But, you still have to have further conversations. We still have to have further conversations about what are the criteria for an entity being recognized as a legal person within a particular legal regime. What are the criteria for someone to be recognized as a political citizen within a particular political community? And that might require answering questions that go beyond merely can you suffer, right? There might be further conditions that need to be met beyond simply can you suffer, again, depending on your views about these issues.
Now, I just wanted to switch over to this discussion about two potential routes towards moral patienthood. So, in your article, you discuss that there are these two routes towards moral patienthood, and those two routes are the first one is consciousness, and the second one is robust agency. So, could you explain the distinction between those? I mean, intuitively, it seems like we all know what consciousness is. It's that sort of what it is like feeling of being yourself, of being something, of seeing red, of experiencing an emotion, I guess. The definition given by Thomas Nagel in philosophy. Do you see consciousness as that or something slightly distinct?
Yes, that is a great question. And to start with the reason for this distinction is we still have a lot of disagreement and uncertainty both in ethics and in science. So in ethics we have a lot of disagreement and uncertainty about what the basis, what the conditions for welfare and moral patienthood are. And then in science we have a lot of disagreement and uncertainty about what types of beings, what types of entities can actually have those capacities, have those relations, satisfy those conditions. And so the reason we distinguish two routes is we recognize that in the philosophical debate about the basis, the conditions for welfare and moral patienthood there are several leading views that ought to be taken seriously but two of the main contenders at least in the current literature are roughly consciousness and sentience on the one hand and a particular kind of robust agency on the other hand. So you start by asking about consciousness and sentience and you know Thomas Nagel famously defines consciousness as what it is like to be someone. What is it like to be a bat? What is it like to be a bot if anything at all? And that still is how we think of consciousness in the morally significant sense. So there are many senses of the word consciousness in everyday life. We might use it to distinguish being awake versus being asleep or being self-aware versus not being self-aware. But in this context when we think about consciousness as a possible basis for welfare and moral patienthood what we mean is does it feel like something to be you? Can you have your own subjective experiences, your own feelings, your own emotions? And then we use the term sentience to refer to consciousness with a positive and negative valence for your feelings. So can you have subjective experiences that feel good or that feel bad, that you want more of or that you want fewer of, like pleasure, pain, happiness, suffering, satisfaction, frustration, hope, fear. So, normally when people talk about consciousness or sentience as the basis of welfare and moral status, they have that full package known as sentience in mind. Because when you can have experiences that are positive or negative, that feel good or bad, that you want more of or less of, then you really have a stake in what happens to you. The more you have good experiences, the more you benefit. The more you have bad experiences, the more you are harmed. And it feels natural to think that at that point we have moral obligations to you.
What do people say about consciousness without sentience? Like does that also have moral status or are a lot of people quick to dismiss that?
Yes, that is a great question and an ongoing debate reinforcing how much disagreement and uncertainty there is on this topic. Some people say, "No, you really do need the full package. You need not only consciousness, but also valence and therefore sentience. Because you need feelings to be positive or negative, to feel good or bad in order for it to matter for you what happens to you. Because if you had nothing but purely neutral conscious experiences, if all you ever experienced was a completely neutral color perception or sound perception, and there was no even subtle pleasant or unpleasant valence. It was pure neutral color or sound experience, it is a little bit harder to summon the intuition that it matters to you whether you keep having this experience or stop having it, whether you have it versus a different one. But other people disagree. They think, "No, with consciousness comes being a subject instead of being a mere object. With consciousness comes having your own perspective on the world as opposed to not having any kind of perspective at all. And a lot of people have a strong intuition that even if your experiences lack positive and negative valence, the mere fact that you have experience, the mere fact that you have a perspective, the mere fact that you count as a subject at all is a morally significant difference between you and a table and a chair, a mere object with nothing resembling that. Now, what I owe you might be different from what I owe a sentient being who can experience happiness or suffering, but I might still have some kind of obligation to you as a conscious being capable of experience. So, this is an ongoing debate. Now, I myself am a little bit more sympathetic with sentience is required and a little bit less sympathetic with consciousness without valence is enough, but I recognize that smart people disagree with me. I think it would be premature for me to be 100% confident that sentience is required and this is part of why when I talk with people at AI companies about what sorts of values should inform their work, I try to be appropriately pluralistic. I try to represent a range of reasonable views rather than just bet everything on my own favorite view.
