AI changes the payoff functions of different types of human activities: learning, I think, over production; relationship over things; and then capital over labor.
Priest, prostitute, politician — there might be a demand specifically that the service be provided by a human being as opposed to a machine.
You think you want this fundamental novelty, deep purpose, this world-historic sense of meaning — what you actually want is a lot of pleasure, social engagements, aesthetic experiences.
We seem to be approaching this critical juncture in human history upon which the next billion years might depend. Like, if you want real purpose, knock yourself out now, because it's never going to be more at stake than it is right now and in the next few years.
What was striking to me was how theological our entire conversation today has been, and how similar to the Christian afterlife: once you have a superintelligence explosion, you sort of zip fast-forward to technological maturity.
Human capital is a depreciating asset in this world of advancing AI.
When I used GPT-3 for the first time, it filled me first with awe and then with dread, because I could see how close AI was to surpassing me in research, writing, philosophy — that I trained so hard for.
If the Copernican revolution took away our special place in the universe, and Darwin robbed us of our special place in nature, then AI threatens to undermine the last pride of the human race: our intelligence.
Today, work gives many of us purpose and meaning, but AI is making more and more of that work obsolete, until one day perhaps all human activity may be redundant.
Nick Bostrom's Deep Utopia is about that day and what to do about it. In this interview, you're going to learn where you can invest your time, effort, and resources that is most AI-proof, and what the good life looks like in a world where AI has made you obsolete.
My name is Jonathan B. I'm a fellow at the Cosmos Institute, researching philosophy and artificial intelligence. If you want to make sense of this new age of AI we're entering, please subscribe. Without further ado, Nick Bostrom.
AI changes the payoff functions of different types of human activities. Certain activities are going to be made obsolete a lot sooner, and other activities seem to be a lot more robust.
And so the three I want to talk with you about are learning, I think, over production; relationship over things; and then capital over labor.
So by the first one, learning over production — what I was thinking about was in your book you gave this thought experiment of a technology that can help us learn, and you described this as potentially one of the most difficult technologies to create, because it requires of you potentially reading and then changing billions if not trillions of synapses in our brains, right, to be able to learn a new language or gain the experience of becoming a good father or a good mother.
And so my thought is, anyone engaged in intellectual work — you know, your production may be rendered obsolete maybe in just a few years' time. We can feed in, you know, your book Deep Utopia, a podcast, an interview — AI will generate a better script than the one that we can have today, and then video AI is able to create something a lot more beautiful than the conversation we're having.
However, if I wanted beautiful — yes, even more beautiful.
However, if I wanted to understand Professor Bostrom's work, I still have to do the hard work of sitting there and reading it, it seems, for a much longer time. So that seems to be one area of life that's going to be a lot more resistant to AI innovation.
Yeah, if you think about what it actually would require through some sort of neurotechnology to sort of directly download skills — so first you would have to have the ability to read off what your current synapses, how they are configured, right? And as you said, you know, millions or billions or trillions of them.
Then you would need to interpret, you know, what all of those synapses actually encode like currently, and then figure out how they would need to be changed in such a way that they now encode this additional knowledge that you want to download, like without sort of messing up what's already there or changing your personality too much. And then you would have to physically change all of these.
So this definitely seems like a task for a mature superintelligence. But until that time, like if you do want to have some sort of relatively fine-grained control over what goes on inside your brain, for now, you know, the best method is the right traditional one of, you know, reading and thinking and talking and working on yourself, meditating, etc.
I think that's actually quite optimistic, because it means, at least according to Aristotle, in the beginning of his Metaphysics, he says, you know, all men desire to know — that knowledge and the contemplative life is in some sense the most human life for Aristotle. That is the one that will be made obsolete at the end.
Yeah, I always find it a little bit funny when philosophers are like thinking what's the best and highest form of being a human, and the conclusion is being a philosopher. But you know, maybe they are right.
Well, I've actually for a long time thought of philosophy as something that has a deadline. In fact, this I think occurred to me in my late teens. It seemed to me that at some point our current philosophical efforts would be rendered obsolete, either by AIs that could, you know, become superintelligent, or maybe by humans developing cognitive enhancements of various forms that would make subsequent generations much better at this.
And so that rather than spending my time thinking about the sort of eternal questions of philosophy, it seemed more useful to focus on that subset of the questions in philosophy where it might actually matter whether we get the answer now as opposed to in a few hundred years.
Would you prefer to know the answers to those eternal questions?
Well, one is certainly curious. But if we do end up in a trajectory where human lifespan becomes extremely long, then maybe rather than sort of using up all these mysteries right away to immediately know the answers, maybe you would want to spread it out a little bit so that you have sort of interesting things to learn and discover even, you know, hundreds of years, thousands of years into the future.
You know, ignorance might become a scarce commodity. Maybe put some on the shelf for like the bottle of champagne, you know, that's from a specific year that there are only small numbers left of. You might want to save it for a special occasion.
But yeah, there might be some other, I guess, professions as well that are sort of relatively immune or variables where it's not just the specific knowledge and skills we have, but the fact that these things be done by a human is regarded as significant in its own right.
Right. Could you give some examples?
Well, I mean, the priest, prostitute, politician, where there might be a demand specifically that the service be provided by a human being as opposed to a machine, even if a machine could have all the same functional attributes.
I want to move on to the second type of human activity that is somewhat AI-resistant, and that seems to be relationships over things. And what I mean by that is, in your book, you gave a thought experiment of let's say we're at technological maturity and we have a better version of an AI parent in every domain, right? So it's better at changing a diaper, it's better at teaching, better at emotional support.
I still would be willing to bet that most people wouldn't want to completely outsource their role as a parent. This could be even more resistant to even technologically mature AI, because what's constitutive to forming a relationship is that it requires, at least in our current view, two humans.
Yeah, I think relationships is one of the more plausible places where we might find purpose that survives this transition to technological maturity. There is some value in honoring and continuing this existing relationship between two people.
And that even if you could sort of teleport in a robotic surrogate that would be sort of functionally superior, it wouldn't be as good — or maybe it would be better in some respects, but it would also lose this value of sort of continuing the current relationship.
Even if the robot sort of appeared indistinguishable from the original parent, if you met in a thought experiment where nobody would actually notice any difference, you might still think that the reality is that there is now this different person who is playing the parenting role. And if you care about sort of truth in relationships, that might already be a disvalue.
And so, yeah, the existing human relationships, to the extent that they sort of partially consist of intrinsically valuable connection to a particular other being, would potentially be resistant.
What are things in a child's education that are potentially made more obsolete given the current wave of innovation in AI?
Well, potentially all of it. But we don't know how long that will take, right? So it makes sense to hedge your bets a bit. You don't want to find yourself, you know, 18 or 20 going on the labor market with no skills, and it turns out the whole AI transition has been delayed.
So yeah, I would want to make sure to hedge your bets, get the broad base, some useful stuff, but then also like have fun while it lasts. I think it would also be a shame to have wasted your childhood.
Perhaps we are too trigger-happy in declaring that certain things are made obsolete given technology. And for example, I went through the Chinese education system, and it had perhaps, when compared to the Western education system, an extreme emphasis on math, like rapid arithmetic computation, including memorization, the memorization of the classics.
And I think it's tempting to think that, you know, even with the printing press, you know, memorization is no longer needed — we have access to all these great books — but certainly with the internet and with calculators in our pockets, that we don't need to know these things. Even though machines can perform it better, there's a lot of alpha, I think, to be had in perfecting these skills, because it's not just that it cultivates your character, it enables you to see things that other people might not.
I would also say general judgment, and especially in these kind of turbulent times with all like social media, so many mimetic dynamics that people are now exposed to. So having a kind of robust, I don't know, cognitive immune system where you can sort of reflect yourself on what makes sense and what doesn't, and not getting swept up in whatever latest fad or cult or memes that are sort of bombarding you — I think that certainly is something I also want to see.
I want to explore the third and last potential activity that would be more resistant. Again, this is not in technological maturity, but just on the way there, and that's capital over labor.
Because one anecdote you brought up is how shocks to the labor market throughout history have changed whether the disproportion of production going to capital or labor has shifted. And so, for example, post-Black Plague, one of the best times to be a peasant in human history, because so much of the labor force was culled that they had a lot more bargaining or negotiating power.
And so I think you hear these stories of them having like holidays for like half the year or something like that, because they had so much more bargaining, negotiating power. It seems like AI, again, on the way there, will do the reverse, right? It will make labor ex-abundant.
And so practically, if I'm someone in law school or someone in medical school, training 10, 20 years before I can start making capital, and I'm building up my labor, I'm building up a skill — that trade-off suddenly seems a lot less attractive.
Yes, human capital is a depreciating asset in this world of advancing AI. And so investments with very long payback times, especially if you're doing them only because of the ultimate payoff of a higher salary 20 years into the future or something, should be discounted accordingly. I mean, you would have to have scenarios where AI development takes longer, or where it gets so regulated that they can't perform these particular jobs.
Let me read you a quote from your book. Keynes predicted that by 2030, accumulated savings and technical progress would increase productivity relative to his own time between four-fold and eight-fold, and as a consequence, the average working week would decrease to 15 hours.
As we approach 2030, the first part of Keynes's prediction is on track to be vindicated. The second part of Keynes's prediction, on the other hand, would appear to be about to miss its mark if trends are extrapolated. While it is true that working hours have declined substantially, we are nowhere near the 15-hour work week that Keynes expected. What do you think this is?
Greed has triumphed over sloth. I think the reason is twofold. First, status competition. So we work hard to afford a whole bunch of things going well beyond our basic material needs.
So we don't just want a car, we want a car that's nicer than the cars everybody else has, etc. And so that provides potentially unlimited demand, right, for more, because the bar keeps racing. And then I think the other factor is a kind of work ethic that is a fairly ingrained norm, that it's sort of virtuous to be not just loafing on the sofa, but to sort of exert yourself.
It's interesting because most people think that we've yet to approach this sort of post-scarcity world where all of our fundamental needs can just be met like that. But at least in the first world, you know, North America, Western Europe, that seems to be far gone. Like most people are able to sustain themselves just fine.
And so it seems like maybe it's less so greed that has triumphed over sloth, but vanity. Because now the reason that people work — I do think it's mostly this kind of social drive. Rousseau, I think, talks about this in his Second Discourse, where he kind of flips the view of the direction of scarcity.
So in Rousseau's Second Discourse, in his state of nature, before civilization is formed, there's actually a state of natural abundance — not because the productive capacities are actually increased, but because people don't have these social, vain desires — vain in Rousseau's perspective — vain desires that civilization has cultivated.
But it's actually in civilization that scarcity increases again, not because of the amount of things that have diminished, but the amount of new desires that have been born.
Yeah, I mean, I think there is a form of scarcity that has been quite pervasive through human history and prehistory, even aside from civilization, in that there were many things that were really needed in some sense, but not available to most people, or indeed any people. Say like quality healthcare.
If you were a hunter-gatherer and you know, you got sick, you know, maybe you could rub some leaf, you know, but there are a lot of conditions that couldn't be fixed that way. And occasionally there would be periods of famine as well, you know, maybe you get the cold season or something else.
And so I think it's kind of, to some extent, been endemic up until a couple of hundred years ago, and in many parts of the world until more recently, or still today, we've been in a sort of approximately Malthusian condition.
So yes, there has been various advances in productivity and all kinds of technologies, you know, over thousands of years, but whenever the economy grew by 10%, the human population also grew by 10%, and average income still hovered around subsistence, you know, with some fluctuations.
And I think it was really only in special circumstances, like you mentioned in the aftermath of the plague, you know, where there had been a great culling, or you know, maybe people discovered some new, came to a new island or a resource where there were no humans yet, like for a period of time they could enjoy plenty.
And then since the Industrial Revolution, where economic growth has been so rapid that population growth, although it has been high, hasn't been able to keep up, that you have been able to sort of have increasing average incomes.
So we talked about AI right now and the current trends we're seeing, but now I want to move into the heart of your book, which is what you described as Deep Utopia, full technological maturity.
And as a brief overview for our listeners, this is where AI far exceeds human capacities in almost all tasks, where we can simulate virtual experiences to be indistinguishable from the experiences we experience now, where the world is plastic — that anything materially that can be done within the realms of physics, that we have the technology to do — but also that we ourselves are plastic, right?
So in this Deep Utopia, you described how, for obvious reasons, work other than the types of work you describe — priest, prostitute, and politician — other than jobs that constitutively require humans, are already made redundant.
But I think the most interesting claim here on redundancy is that leisure, or a lot of human leisure, would also be made redundant as well. Why is that?
If you go through those sort of leisure activities one by one, you can for many of them cross them out, or at least put a little question mark on top of them, whether they would still have a point in this condition of technological maturity.
So maybe some billionaire goes to the gym, you know, five times a week because they want to — they want to be like fit and healthy, and maybe they feel more relaxed afterwards. But if you had this drug that would induce exactly the same effects without, you know, you having to go there and sweat and make that, then you could still go to the gym, but why? Like if you could just pop the pill, you get exactly the same health effects, the same body, the same mental clarity after.
Like I think many people would just pop the pill. We talked earlier about child rearing, like maybe people would still want to do that, but a lot of the specific activities involved in child rearing — like changing the diapers, like doing this kind of — a lot of it is kind of individually things you might be tempted to outsource.
And so then you can like go through leisure activity by leisure activity, and I think many of them would lose their original purpose at technological maturity, and then if you did them, it would have to be for some other reason.
Generally speaking, I think you would have a kind of post-instrumental condition where, to first approximation, there would be nothing you would need to do in order for something else to happen, because that would be a shortcut to the other thing. You could sort of press a button or request your AI or robots to do that thing, and you wouldn't have to exert effort yourself to get that outcome.
And so all the things we do for those instrumental reasons would drop out of the picture, with maybe some like class of exceptions.
And so I think this is the most interesting insight in your book, which is even if we solve all of the tremendously difficult problems around alignment, around all the other issues that you described in your various other books, and we get to the maximally good state, right, where politics, society, and AI is purely working for us, it's still in some sense difficult to imagine what a good life looks like, because it's so different from how we conceive of a good life today.
And so yeah, when so much of work and so much of leisure is made redundant, what is the best type of life one can live in such a utopia?
Well, it's actually quite challenging to really envisage a great life, because yeah, it would risk undermining a lot of the things that currently give meaning to, and you would feel almost at a loss. That's this kind of, do we just become these kind of amorphous, you know, drugged-out pleasure blobs, or what? And then it might seem quite alienating and unattractive.
I think though that if you push through that, I think there is a there on the other side of something that actually would be very worthwhile.
So in your book, you described five, let's call it, modes of defense for a life in Deep Utopia. And I found this sort of way of framing it as already very interesting, because you think of in utopia you're not defending things, right? But you are defending here because of how difficult it is to imagine a good life here.
Hedonic valence, experience texture, autotelic activity, artificial purpose, and social-cultural entanglement — these will become, in your view, the pillar of the good life once you push through, as you describe it. So can you give us an overview of what these mean?
The first one being hedonic valence. So this is basically the observation that we could have a lot of pleasure in utopia — pleasure in the wide sense, not just kind of physical, but mental, like enjoyment. Like you could actually immensely enjoy every hour and every day of life, to degrees that at least would match the sort of the peak current human experiences, perhaps go a lot beyond that.
And it's easy to sort of dismiss this because it's like a philosophically boring point. I mean, it's kind of trivial that when you have this advanced neurotechnology, you could do this. And then we immediately jump to sort of thinking, well, that's a kind of degenerate existence — we think of junkies, and it's like not really the point.
But actually, I think it is super important, maybe the most important, and this on its own might make it very worthwhile to swap out the current world for that.
I mean, I should say there is also a sort of minus-one mode, which the book doesn't talk very much about, but is super important, which is just get rid of the negatives that currently plague the human condition, which is like immense and terrible. But here we are thinking about what could you do more than just kind of getting rid of them.
And so there, yeah, just this hedonic well-being, like that every day could actually be an immense delight. Like there are some people who are like philosophical hedonists — I think pleasure is the only thing that matters, and the absence of pain. So for them, that's kind of case closed at this point, right?
But there are other value systems that maybe think pleasure is a good thing, but there are other good things. So let's see what we can add to just mere hedonic well-being.
And so then the second one is experience texture. We observe that not just these utopians have great level of enjoyment, but rather than just being sort of dazed-out junkies that have a diffuse, confused sense of pleasure, they could like attach this pleasure to valuable experiences, like say the appreciation of beauty, or the understanding of deep, important truth.
So like pleasure in learning and understanding the basic laws of physics, learning about human nature, learning about, you know, and appreciating great art and natural beauty, you know, plays, and like that's how they get their fun. It's not just like the junkie, but it's like this kind of connoisseurship that is also exquisitely joyful.
You know, maybe appreciating the moral virtue and goodness in various people and historical people and so forth. That already seems to like make it more attractive.
And then you can add some further things like, so it's not the case that you need to imagine these utopians as mere passive recipients of these experiences of truth, beauty, and understanding of deep truths that they take enjoyment in. This would be this kind of autotelic stuff. So they don't just sit there passively like observing great beauty and feeling joy, but they could go around and like doing stuff.
Artificial purpose is purpose that we create in order that we are then able to engage in purposeful activities. So you could set yourself some maybe arbitrary goal, and then once you have that goal, then if you select that goal in a suitable way, you could then have instrumental reasons to engage in various efforts to achieve it.
So key here is that the goal you set yourself has to be sort of constitutively such that it calls on you to make an effort, rather than for you to press the button to have the robot do it, right? So you can bake into the goal that the goal needs to be achieved by your own efforts.
Otherwise, like the goal is to achieve a certain thing with your own efforts, then if you have that goal for whatever reason, then now you have purpose, because the only way you can achieve your goal is through your own efforts.
So we can think of this like the paradigm cases, kind of various forms of games, where, you know, maybe you decide to play a game of golf. There was previously no reason why the ball would have to go into a sequence of holes. It's like a completely arbitrary goal, but you adopt that goal, and now you have a reason to try hard to hit the ball in exactly certain ways to achieve this goal.
And you could generalize that. You could have much more complex games with, you know, multiplayer, multimodalities, like, you know, extending maybe over years, and that could then give you purposeful activity, not just activity.
So those are the first four, and there is like one remaining one, which is there would be some natural purposes that could survive into technological maturity, but that would be sort of softer purposes.
Like what natural purposes would survive in this post-instrumental state?
Well, take something like, so if you currently have some value or goal that, say, you value the continuation of a certain tradition, that might be something you just happen to — that many people would right now have that as one of their values.
And the tradition might be such that it just isn't continuing unless humans are continuing to do it. It might be like constitutively part of the tradition that humans are doing this thing every year in a certain way, like imagine some ritual or something like that. So then those would survive, because that would be like you could create robots who would be doing this stuff, but it wouldn't count as continuing or honoring your ancestors.
Like it might not count as you honoring your — your value of honoring your own ancestors might not be served by building a whole ensemble of robots who are like going around and, you know, paying visits to their grave or thinking about them. Like it might require you to do it for that value to be achieved.
More broadly, I think various forms of social entanglements, where, I mean, we touched upon it a little bit earlier when we discussed parenting, where if there is this existing relationship between a child and a parent, and part of what is valuable about that is that it's these particular individuals are relating in a certain way, then that could also give you a natural purpose even at technological maturity to continue to do certain things and interact in various ways with your child.
And this might at first seem weak, like compared to a lot of the reasons we have things to do today. There are like very stark, tangible, immediate consequences if we fail to do them. So like maybe somebody has to go into work every morning because otherwise they will lose their job, and then they can't pay their rent, and then they will get thrown out on the street, and then it will be cold. And this is like a very sort of hard set of consequences, and a lot of the stuff we do today are motivated by these kind of hard consequences.
And in utopia, a lot of that would go away. But I think like just as you walk outside in the daytime, you see the sun, you don't see the stars — they are much fainter than the sun, but they're still there.
I see these subtle values as already being there. Like there's a whole host of these more almost aesthetic reasons for doing things that are blotted out from sight currently because there are these kind of more screaming moral and practical needs that we need to take care of in our lives.
But if you imagine a scenario where all of that went away, like the sun set, then it would make sense for our evaluative pupils to dilate to take in more of the fainter light that comes from these subtler values, right?
So what I understand these five rings of defense is that the first four rings, a critic might say what's lacking from them is a kind of necessity, right? That I feel I need to do XYZ because it's from this external source that gives my life purpose.
And it seems like they're trying to rescue necessity in utopia by highlighting a subset of activities, like honoring one's ancestors, that it constitutively requires us humans, or maybe even stronger, you as an individual, to do. And that's how you rescue necessity.
And so maybe analogy here is, you know, LeBron playing in the NBA, bringing back the championship to his hometown, Akron, Ohio. There's nothing necessary about that, right? It's a set of rules that we've invented for ourselves, but it doesn't make it any less meaningful for the people involved. Is that a good way to understand it?
Yeah, well, if he independently wants to do this, or if there is an independent value that this is something that should happen — that it like if it were a case that he just set himself this goal because otherwise what should I do all day long, and then like he convinced himself to pursue it, then it would be an instance of the fourth, artificial purpose.
But if there's like some independent reason that was not just created in order that somebody has something to do, then it would count as a natural purpose, right? And in this example, LeBron winning the championship for Cleveland, it would be something like the recognition and expectation of his family and friends and the whole city that he grew up in.
Would that be a good example of something that would make this social-cultural entanglement instead of artificial purpose?
Yeah, so that would give him a real reason to do this if people continue to want that to happen, and if he cares about what other people want or what they will think of him.
So it seems like social-cultural entanglement relies heavily on the economy of recognition, right? What people desire, what people give honor to, and what people give esteem to.
Do you think that humans will start esteeming the recognition of non-human agents? And maybe we're already starting to see the genesis of this, where there are these dating bots that have already existed, and right now it's only appealing to people who are struggling to form real human relationships.
But I remember there was one dating bot that after they changed the algorithm and the dating bot behaved nothing like what they previously behaved like, the people were in tears as if a real family member had died.
So it seems like if eventually we think that we'd care as much about artificial recognition as human recognition, then a lot of these social-cultural entanglements would be threatened, because presumably we can tell with a robot what to recognize or what to give esteem to.
