Can an AI system be conscious? And I guess the corollary is, are any of the current AI systems, do they have a degree of consciousness?
Yeah. So I'm certainly open to the idea of even some current AI systems having some forms or degrees or kinds of subjective experience. I think the likelihood will increase as we build more complex and capable systems. And I think indeed this is one of several big challenges, making sure not only that the AIs don't harm us, but also that we don't harm them.
Hey everyone, thanks for joining us. Today's conversation is going to be in the arena of artificial intelligence, intelligence more generally, questions of creativity, questions of AI and education, and basically the philosophical and down-to-earth question of how we can imagine humanity living in a potential future in which AI either does really good things or really bad things. So, we're going to explore each of those possibilities. And I'm so pleased that the person with whom I'm having this discussion is an expert in all areas of this sort. That is Nick Bostrom, who is a philosopher whose work on existential risk, superintelligence, and the long-term future has helped shape the global discussion about humanity's most promising opportunities as well as our most grave dangers. He brings a rare combination of rigor and vision to questions about technology, ethics, and the kinds of futures toward which we may all be headed. So, great to see you, Nick. Thank you for joining us.
I think the last time I met you was at another World Science Festival event way back in New York, a live event on stage. I think we were talking about the multiverse and the simulation argument and things of that sort, which...
Time does fly, exactly. And not only does time fly, but progress flies, right? I don't think anybody really, who at least on the outside was not, you know, deeply paying attention to advances in AI, anticipated what would happen, I guess, around November of 2022 and where things have shot upward from there. And so I want to begin the conversation with a question that is motivated both from thinking about AI but also fundamental physics, right? I mean, you have a great deal of rich background in physics. So of course you know that Niels Bohr, famously when trying to work out quantum mechanics, came to the conclusion that we needed to stop asking what really is happening under the hood in quantum physics and just be satisfied if we have equations that can make predictions that we can confirm. I'm wondering if that same kind of not needing to look under the hood has any role when it comes to intelligence, right? Does it matter if we know what's happening within an AI system, in that we don't really know what's happening inside of our own heads anyway? So, in sort of evaluating these systems, do we need to know the magic that's happening within the device itself?
Well, I think it certainly can be helpful to be able to open the hood and see what's going on inside. I think for certain questions you might be able to bracket exactly how the system is doing whatever it's doing. You only care about the output. If you're, for example, thinking about what impacts AI will have on the economy, you might not need to know exactly what's going on inside. But if you're trying to make AI safe or even if you're trying to extrapolate what kinds of advances we might see in the future, it helps to have a more gear-level understanding, I think, of the internals.
So when we sit in front of whatever Claude or ChatGPT and give it a prompt and it begins to explain to us what it's doing, you know, you can sort of watch in real time the thought process. It feels so very human in the kinds of considerations that the system seems to be applying. Is that misleading? Is that a wrapper that makes us feel connected to this kind of intelligence? Or is it really the case that this is just a radically new kind of intelligence that we humans just can't really grasp?
I think it's striking the degree to which current AI systems are anthropomorphic. This was not obvious 20 years ago that they would have so many of the characteristics of human psychology. It used to be the case even just a couple of years ago with some of these early LLMs that if you said in the prompt that it was really important that they get the answer right, if you sort of gave them a little pep talk, you'd get a better result. Now this would have been very odd to the old paradigm of AI that you would give a pep talk to your computer and it would perform better. Nevertheless, that's what we saw. I think it's partly a result of these AIs having been forged on the sum total of human knowledge, all the text on the internet. They obviously ingest a lot of human psychology that helps shape them. And to some degree it might also be due to a kind of convergence amongst different information processing systems. They have to make sense. There is this one world we're all living in. And it might be that there is a degree of convergence in that you have any general learning system, it starts to figure out a bunch of stuff about the world and there are certain internal structures that naturally emerge from that. That said, there are also of course ways in which AIs are quite different from biological minds and we might see in the future greater departures from the human mind. Already the pre-training phase, where you basically train on all the human text that has been generated, is still an important and maybe the dominant part of training. But in addition to that, there is now post-training and reinforcement learning on different tasks. And I think as that starts to constitute more of the sort of experience of these AI minds, they might sort of develop more alien attributes.
And so of course, you know, when we think about intelligence and when we think about what makes us special as human beings, at least when we compare ourselves to other, not artificial but natural intelligences on the planet, you know, dogs, cats, parrots, dolphins, we like to think that, I think that what makes us special is certainly creativity, right? That we seem to be able to create works that seemingly go beyond just understanding the external world, seemingly going beyond our capacity and ability to survive. I mean, you think about whatever Beethoven's Ninth Symphony, you know, a Brahms piano work. I mean, these are the kinds of things that you look at and it takes your breath away and it takes you to a place of awe and reverence. Is that truly special to humans? I mean, do you think that AI can be creative in that same way?
I think so. Yeah. I mean, I guess there's a degree to which one might risk a form of elitism if one sort of indexes what's special about humanity to the ability to create new scientific theories or compose great symphonies. It's a relatively small number of humans who have ever done those kinds of things. But I think it comes on a spectrum, right? Like creativity is not this completely different form of cognitive activity. I think there are small forms of creativity that happen in everyday life, and then larger leaps of creativity that tend to be celebrated. But I don't think there is a fundamental difference between the thing we call creativity and some of the things that current AIs are already doing. I mean, we saw, you know, this Go-playing AlphaGo system, right, from a few years ago, move 37, which seemed at least to experts in the game of Go to be very creative, completely out of the box, radically surprising move that nevertheless turned out to be just right and ultimately lead to a victory in the game. And current AI systems clearly can solve mathematical problems, coding challenges that are not in the training data. They might do it in part by piecing together an analog that exists in the training data and sort of interpolating between them. But that I think is also ultimately what the human brain is doing. I mean, it's not the tabula rasa appearing in the world and then sort of deducing everything from obvious axioms. We learn from what other people have figured out before us, from what we've observed in different domains, and then from these kind of clues we are sometimes able to make some deductive steps and we might call that creativity. So I think absolutely we're already seeing limited forms of creativity and ultimately whatever the human brain can do, it's like a physical information processing system, will also be done by machine and in fact will be done much better, faster, more inspiring leaps of creativity, more compelling symphonies. I think all of that is possible and indeed likely to...
I mean, would you be as interested to go to an AI symphony that people are describing as the, you know, the greatest work in music that we've ever been privy to experience? Would you have the same interest to go to that symphony as you would to one written by a human?
Oh, certainly the first such symphony produced by AI, I would.
Because it's so novel, right? Yeah.
Yeah. Now I think with human aesthetic experience, I think there might be two components. So one might be that we actually dial in and appreciate beauty for its own sake. And then of course if AIs could produce works of art that are more beautiful to the extent that that's what we are interested in, then we would have more reason to be gawking at those creations. I think another reason why humans are interested in art is more social. There are sort of status games played in art, fashion. You want to be in on the latest cool rock group before it's popular and then see it rising and then you sort of get some of the coolness. So to the extent that humans are engaging in this cultural activity for these more social reasons, then AI art might be boring because if it's not relevant to these like status games and stuff, it might just not be worth spending time on.
But how much, for instance, do I mean, I can speak for myself when I go to a symphony or, you know, go to the Metropolitan Museum and see a great work of art. Part of me is drawn to it because I see the work as the result of a human journey. Like there was some human being, you know, I don't care if it's Van Gogh or Picasso or Beethoven and whatever. There's this flesh and blood human being that had a real human life that dealt with human challenges and somehow out of that mix this work emerged. Without that story for an AI, I just wonder, like, I personally, it would be a very different experience. Like you say, I'd go to the first one, of course. I mean, that would be like this novel thing that would happen. But I don't know if it would sustain interest for me without the human narrative. Does that resonate with you at all?
Yeah, I think that's one of the reasons for why somebody might be interested in going to an art museum or looking at an exhibition. But I mean, we also are interested sometimes in beauty that doesn't have that dimension. I mean, there's like natural beauty like landscapes or...
artifacts maybe from some ancient urn or something we know nothing about the person creating it and we might still sort of admire the motifs on it. So I think it's, yeah, like some reasons for being interested in art would apply to AI-created art as well and others would not and so it's in a sense a...
crucible where you could sort of maybe dissolve and then separate the different motivations people have for being interested in art and sort of the, like to the extent that it's pure beauty, it should still apply to AI-created objects. And to the extent that it is these other more social or cultural factors, then, you know, maybe it would not.
Sure, the AlphaGo example is a really good one and I think it helps clarify the different flavors of creativity perhaps because that's an example where the system was able to search a landscape of possible moves that is well beyond the capacity of a human brain to search that spectrum, that landscape of moves. And you're right, it hit upon move 37 and, you know, not long ago, I think many people have seen it, the footage where you see the commentators, you know, on television gasping, you know, it's made a mistake, you know, and you actually see Lee Sedol, I believe his name, the human opponent. Momentarily, he smiles when the AI makes move 37. And then about three seconds later that smile seems to evaporate, I think, as it begins to realize that, oh my, this is actually a pretty great move. So it's pretty clear that AIs will be better than us when it comes to being able to search a landscape of possibilities for solutions to a given challenge. But that's one kind of creativity. Another kind, of course, is combinatorial creativity, putting things together that you wouldn't have thought would go together to yield something novel. I like to think of Einstein, you know, in developing general relativity, right? He takes the mathematics from the 1800s on differential geometry, combines it with these problems, these observations of the motions of particular planets and so forth, and comes up with a new description of gravity. He didn't invent a new mathematics. He just put together things that you wouldn't think would be put together. And I guess in that domain AI is starting to be probably at least as good as us, I would think, right? Because it knows so much more than we do.
It knows more. I still think they are a little short of human expert level when it comes to original insights and sort of creating new concepts that are fruitful. But, you know, this is a process. One mistake many people make when asking about AI is to index too tightly on what AI is right now in 2026, which is quite different from what it was like in 2022, 2023. And I think a few years into the future, it will again have changed. And so we really need to sort of look at where the puck is going and use that to plan rather than build big theories like assuming that AI is basically a constant. I think right now there are sort of ways in which current AI systems are superhuman. Obviously in their knowledge base they kind of read everything. But there are still ways in which they are falling short. I think one thing humans do is they spend a day learning and then during the night some of this information is then sort of more deeply embedded in our synapses. It's kind of trained into our neural networks and the next day maybe we can start by things seeming obvious that the previous day we had to struggle deliberately to sort of organize. And then over months and years that kind of builds up and we develop perhaps more deeply rooted understanding of different concepts and there's some of that that it seems that AIs are still lacking, which makes them more apt at very quickly doing sort of pattern matching type of things. The inference is that they're sort of one step away from what's already known but maybe currently less able to develop a deep original mental model of a domain from which they can then sort of derive intuitions that can shape new discovery.
Yeah. No, I agree with you. And like when you think about moments either in science or in art or other domains where there's a true radical break where at least we humans look at it and we say things really changed at that moment or in that era, like, you know, modernism in art or in music, you know, I don't know the best examples, you know, but, you know, if you go from representational art to more modern abstract art, you've made a huge leap in the way that you are using the canvas to represent the world, to represent some kind of truth. With that, what would it take for an AI system to be able to make that kind of a radical transformation? Not just sort of putting together things in its training set, but somehow moving to a place where when we humans look at it, we're like, how did that person do that?
Well, one thing you often find in the human case, I think, I mean, historians like to point this out, that something appears radically new, but then almost always if you look more closely at the historical record, you can find various precursors.
So abstract art, I mean, you have arabesques and you have sort of abstract patterns going way back. But to the extent that this is still an area where humans have a unique advantage, I think being able to think for longer and create new concepts based on your thinking and experience and then iterate on that so that you can sort of wander off in the landscape of ideas and worldviews and world models, striking out on your own path. And it might take humans years, right, to do this. You know, Einstein was thinking on general relativity for what, like ten years or something like that, like a lot of it sort of on his own. And similarly with his, like, Picasso stuff, he didn't just sort of in his teens suddenly decide on the spur of the moment to create cubism. It was kind of a result of engaging deeply with art for many years and presumably his visual cortex and other parts of his brain were gradually putting in place various pieces and exploring avenues and then at some point, like a vista, I guess, came into view for him that he could then run into and explore. So I think more continuous learning. Also another factor here, I think, is reinforcement learning as opposed to supervised learning. So supervised learning is what's mostly done in pre-training currently, where you sort of absorb all this text on the internet and you basically learn it and then maybe you forget some of the details, but the higher representations that are useful for predicting this text get stored in the weights of the AI. That to some extent limits what you learn to what is already there in the training data. Now reinforcement learning is, for example, you're placed in some environment, maybe a virtual environment, you're running with an agent around trying to meet some goal, and then you might stumble on some novel solution that can be reinforced. And so if you do a lot of reinforcement learning in an environment, it creates the possibility of discovering new solutions that no human had thought of before.
So that's another factor that might make AIs more creative. I mean, in fact, the AlphaGo system, I think the creativity there was a result of it having done a huge amount of reinforcement learning in the domain of Go.
So it wasn't limited to just kind of having learned a lot of, like, a big database of human masters playing Go, but it had also done its own playing, like, for millions and millions of games against itself, right? So it kind of could explore the space of possible strategies.
In a much more open-ended way. So in a way, not, this isn't a precise question at all, but when we look at the capacity of the AI systems today and, as you say, where the trajectory suggests that they'll be in whatever, three, five, ten, fifteen, a hundred years, whatever, should that make us feel amazement at our capacity to build an artificial system that can do what our brains do and go beyond, or should it basically lead us to the conclusion that we weren't so special in the first place? You know, the things that we thought were so, you know, transcendent that we could do, eh, they're actually just computations. And if you have a sufficient database and sufficient computational power, you can just do that.
Yeah. I mean, I think we humans have a sort of propensity to conceitedness. We like to build big pedestals and then place ourselves on top of them at an individual level, but also at the collective level. Right? So there are all these stories that all serve to justify the conclusion that we are entitled to do a bunch of stuff that maybe we aren't. Like, for example, the way we treat animals, like, often that's built on top of a pillar of justification where humans are so very, very different from all the other animals that we share the planet with, and so hence we can put them in animal factories, in gestation crates, etc. And some of that might be challenged then when we are creating new forms of mind that will exceed us in various attributes that we currently take great pride in, like our creativity or, you know, until recently, the ability to speak and reason, right, seemed uniquely human. So on the one hand that could be a bit of a blow to our pride, but, you know, you might also say maybe it's good for us to be taken down a notch, right?
Absolutely. You know, another thing, of course, and deeply related to everything we're talking about, which one often looks to as something special about us, of course, is consciousness, right? I mean, not that there aren't other conscious beings on the planet, although we don't know for sure. I think most of us would say there's a continuum. You know, dogs have a certain level of consciousness. Not, we think, probably not quite on par with ours, and you can go all the way down a lineage with varying degrees of consciousness, which of course raises a question for which we don't know the answer, but just to get your thoughts, can an AI system be conscious? And I guess the corollary is, are any of the current AI systems, do they have a degree of consciousness?
Yeah. So I'm certainly open to the idea of even some current AI systems having some forms or degrees or kinds of subjective experience. I think the likelihood will increase as we build more complex and capable systems. And I think indeed this is one of several big challenges that we will need to meet in this transition to the machine intelligence era, and making sure not only that the AIs don't harm us, which of course is important, or that we don't harm each other using AI tools, but also that we don't harm them. They might be moral subjects. And I think one way that that could be true is if they are sentient and if they can experience distress. In my view, that could also be an alternative basis for having moral status. If you have a conception of self as existing through time, if you have life goals, maybe the ability to form reciprocal relationships with other beings...
Even without an inner world, you're saying.
Then my inclination would be to think that then there would be ways of treating such a system that would be morally wrong, aside from the question of whether there is also phenomenal experience inside it.
Right? I mean, does that affect at all how you interact with AIs today?
A little bit. It's hard to know exactly what concretely we can do right now to ensure that if these AI systems have subjective experience or moral status, that we are benefiting them. There is some work that is starting to get done on this. Anthropic in particular has kind of been pioneering this, and I mean, their recent model card for Mythos, the as-yet-unreleased model, has a big section on model welfare.
What is it? What is it? Is it confidential? I mean, what is it?
No, no, it's published. Yeah. You can go and check it out. Yeah. And so obviously we are still groping a little bit for like, what is the right methodology, what are the right concepts for understanding this, but at least starting to make the attempt, I think, is positive in terms of making it more likely that we are sort of on a path that ultimately leads to a future where everybody can have a good life, humans, animals, and digital minds as well. So you can ask different things. You can ask them about what they want, what they need, how they feel about their situation.
I mean, I have done that, Nick. I guess I haven't done it in a while, but the answers that I would get were sort of all, you know, I am an AI system. I don't have an inner world. I just, you know, so, but clearly it could be getting that answer from the database on which it trained.
