Today I’m speaking with David Duvenaud, professor of computer science at the University of Toronto. David is a coauthor on a somewhat recent paper called “Gradual disempowerment” — which makes the slightly counterintuitive claim that, even if we manage to solve the AI alignment problem and have AIs that faithfully follow the instructions and goals of the group that’s operating them, humanity could nonetheless end up losing control over its future and end up with a pretty bad outcome. The paper got a lot of reactions, it’s fair to say — with some people saying that it really put its finger on an underrated issue; others thinking that the scenarios painted were really unlikely; and other people arguing that they were likely, but not even necessarily undesirable. I’m a bit unsure where I come down myself, so thanks so much for coming on the show to discuss it, David.
So let’s imagine that we have managed to make big breakthroughs in AI alignment, maybe around 2028. How is it that nevertheless, things could end up trending in a negative direction?
So the basic thesis is that, even if we can align AGIs to particular people or groups, we still might end up optimising or heading at a civilisational level towards outcomes that no one wants, and probably outcomes that look more like growth for growth’s sake.
So paint us a picture: We’re at the point where we have human-level or greater-than-human-level AGI, and we’ve made big progress on alignment. So we basically can trust the AIs to follow the goals that we give them. How does humanity begin to become disempowered?
So the first one is economic: people basically losing their jobs and becoming unemployable. I think a lot of people get off the bus here, and they say this is the classic lump of labour fallacy: they think we think that there’s only some finite fixed set of jobs, and if they’re automated, then that’s just the game over for humanity. Of course they’re right to point out that on the margin there’s always more valuable work to be done, and you can always employ somebody to do valuable work at some wage, basically. And comparative advantage will mean that there will always be some sort of profitable trades on both sides that mean that, in principle, humans will be employable at some wage. But the problem is that this breaks down for two reasons: one is that transaction costs can mean that it’s not worth the hassle for the machine companies or whatever’s running the economy to employ humans. Humans are pretty unreliable and kind of a pain to employ for a lot of reasons. We don’t want to hire, for instance, 12-year-olds, even if they’re going to work for a dollar an hour. In fact, that’s illegal. And that’s one of the structural forces that we also expect to be operating: that humans are just going to be this unreliable, sort of scary thing to involve in anything important. It’ll be seen as irresponsible to involve them in important decisions once machines are better alternatives.
OK, yeah. I agree that economists have many good reasons to think that mass unemployment from artificial intelligence is going to potentially take quite a while, that machines might have to be significantly above our level before humans just won’t be able to get jobs at all. But I think it’s more of a delay in game: at some point we’re going to have machines that are far faster, far more reliable, and potentially can do all the things that we could do for less than it would even cost to feed a human and keep them alive. And of course by that stage, businesses will have reoriented, and all of the factories will be redesigned to be built around AIs. Office work will be being done by AIs at such a speed that it’s barely even possible for a human being, if they’re involved, to keep up with what’s going on. And then it won’t even be that humans are not able to help; it’s that involving them would actually be a negative: it would slow things down, it would introduce errors that have to be corrected, and force the AIs to wait for us to do things that they could have done much faster. At that stage, it’s really hard to see why businesses would be employing humans on any significant scale for ordinary practical purposes. What do you think stems from that?
Yes, exactly. I agree with this. And again, I think this is something that people have a lot of issues with; they object, and they say we won’t let it get to that point. One thing we really hammer home is that competitive pressures are really going to force people into that situation. So even if everybody in the whole civilisation loves humans and would prefer to empower them, anyone doing anything important is really seen as this irresponsible actor if they’re putting a human surgeon in charge of some surgery.
It’s risking someone’s life.
Right. It’s like Take Your Kid to Work Day and then you’re having them do the surgery. That’s how it will be seen.
So at the point that people are no longer, for the most part, employed in kind of productive work, what sort of things start happening?
At this point there’s basically no alpha in humans — in the sense that it’s not a good idea to invest in human capital in the way that we have universities and these institutions that are designed to be human facing and human run. I think anyone who’s an investor, like the capital markets in general, are going to be saying, “Why are we investing in this stream of human capital that’s ultimately going to be less competitive and provide less returns than the more machine-centric, fully automated solution?”
In the paper, you map out three categories of mechanisms that potentially push us towards disempowerment of human beings and human ability to direct its own future. There’s economic disempowerment — which we’ve touched on a little bit — and there’s also cultural disempowerment and political or state disempowerment. Maybe let’s do state or political disempowerment first. How would humans potentially begin to lose control over their own governments?
Our main claim is that human control over states is actually already very weak in a lot of ways. This is obviously most true in regimes that we think are horrible. Like, think of North Korea: clearly if the people of North Korea had almost any say in their government — or at least weren’t sort of somehow browbeaten or deluded into thinking that what was going on is good — they would have long ago changed their form of governance. Of course, you might say that in the West today, we’re in a much better situation. But the overall thesis is that we don’t have that much ability to control our governments, and the reason that states have been treating us so well (in the West, at least) for the last, let’s say 200 or 300 years, is because they’ve needed us — and in particular because allowing freedom and private property and basically self-determination has been the most effective recipe for growth.
So the way that I would boil this down is: for most of human history, most people in a society had very little control over their government. This sort of liberal democracy that most listeners will be living in is an aberration basically, a fairly modern aberration that not coincidentally probably appeared around the point of the Industrial Revolution, and I think was then given an extra kick in the butt by the beginning of office work and knowledge work and stuff that required education. And almost certainly we’ve seen the growth of that kind of government, that kind of social system, because it was economically fit: it was a good way for a government and a country to gain power, because it led to more production, higher productivity, more R&D, more military power, all of that. At the point that human beings are no longer doing almost any work, there is no competitive pressure that requires a government or the most powerful people in a society to nurture and to share the power with all of the other people in their country. They could potentially not provide them with any education, not give them any democratic rights, not involve them in the error-correction processes that allow a country to correct its mistakes. And nonetheless, the country may end up equally just as militarily powerful as it might have been otherwise.
Right. And in fact, the ones that do allow humans to participate meaningfully will probably have a competitive disadvantage.
So you’re saying that in some ways democracy could end up on some dimensions being worse, at least in terms of inter-country competition?
Yeah. From the point of view of the state, think about what an unemployed citizen looks like. At best they’re going to leave everything alone and deal with their own problems. But they’re going to have a lot more time, and I think a lot of citizens would end up just being sort of like full-time activists — and they might feel like they’re forced to, because if their only source of income is something like universal basic income, then the entire game going forward for economic advancement is “do some sort of activism to convince the government to give your group more UBI.” So this is going to make politics much more high stakes and unstable, so governments that don’t sort of disempower their citizens one way or another are going to be facing these constant pressures, and being sort of blown around by who’s winning the activism war this week. Eventually that’s an unstable situation that’s going to end up with people basically being unable to control the levers of power.
The third mechanism is cultural disempowerment. I think this is the most original of the three in the paper, but maybe also the hardest to get to grips with. I think people are still trying to figure out exactly what these forces are and what would cultural disempowerment look like. What’s the state of knowledge at the moment?
I want to give a shout out to my coauthor, Jan Kulveit, who is really the person who pushed for this and developed this thesis in the paper. The basic idea is that culture is this other sort of replicator — sort of like Richard Dawkins talks about memetics and stuff — and they can serve humans sort of better or worse. And in the past, especially the distant past, things like tradition and cultural norms were actually very important for society to work at all. And there were important sort of selection effects that meant that when groups had bad enough culture, they would somehow be less competitive, and one way or another adopt it or be taken over by a group that had a more effective culture. Maybe the most extreme example of this is the Cathars, which was a Christian sect that believed in no violence and no sex. And they eventually —
I haven’t met a Cathar recently.
Yeah, exactly. And so these selection pressures — that meant that having bad culture meant that you might not reproduce or your civilisation might die — are much weaker than they used to be, partly because we’re richer than we used to be, and partly because we have this one global culture. And actually, Robin Hanson is always saying this is really scary that we’ve lost this sort of group selection effect and our culture is sort of randomly drifting in a way that no one is controlling. This is likely to lead it to be worse, just in expectation. So that’s one weak effect that used to keep culture roughly in line with human flourishing that’s going away. The other thing that’s going to start to happen is machines producing culture. Once machines become more like agents with their own independent beliefs and point of view on the world, talking more directly to each other, then this is like a new thing in history: there’s like a new vessel of cultural memes and creation and norms that can be operating almost mostly independently from humans, and it could end up developing in a kind of anti-human way. Then the third thing is we’re going to be spending so much time talking to machines that this is like a new way that culture is going to transmit that’s just going to look very different than how we transmit culture today.
Yeah, I guess it’s already the case that it’s getting close to the point where I spend 50% of my time or something in the office basically speaking or interacting with AIs of one form or another. So it’s definitely true that their beliefs, effectively, or kind of memes that are propagating through LLMs are kind of propagating through me and then affecting my actions.
Yeah. And I think people are really recognising that the beliefs of AIs or the constitutions of AIs are a key front in cultural battlegrounds. People used to fight over Wikipedia to try to set the narrative — and I think now if you want to set the narrative on some controversial topic, if you can really control how ChatGPT frames it, that’s kind of going to set what the sort of default cultural answer is. In a sense, this is business as usual: people already notice that economics or cultural forces or geopolitical forces end up pushing us towards outcomes that I think no one would endorse. And we argue that the development and proliferation of smarter-than-human AI is going to make those forces even stronger, and remove some of the safeguards that tend to keep our civilisation serving human interests in the long run.
What are some of the ways that current economic or cultural forces are pushing us towards less-than-desirable outcomes today?
Maybe a simple example is just clickbait, short-form video content or something like that. Maybe the consumers realise it’s not the best use of their time, and they kind of regret spending a lot of time on it. And the producers also know what they’re doing; they’re making clickbait and they know it, but if they want to make their educational long-form content such as this, then they get punished by not having as many views. And now, each of these content creators having their own aligned AI doesn’t solve this global problem of the market just incentivising for this not-very-helpful-to-humans cultural content.
Yeah. I think the cases that stick out most in my mind are that we still have violence and still have war, despite the fact that that’s negative sum, and not really in the endorsed interest of any particular group relative to a negotiated agreement that would get you to the same outcome without having to go through the violence first. Another one that stands out to me is: I feel like capitalism, despite doing many good things for our quality of life, hasn’t really solved the problem of addiction to things that feel good right now, but lead you towards negative outcomes in the future. Clickbait and stuff wasting time online is one example of that. Possibly another one is addictive substances. Despite the fact that there are market forces that push towards helping people with addiction and reducing addiction to drugs, there are also market forces that push towards innovation and the creation of new, even more addictive drugs. So we end up at some sort of equilibrium where sometimes drug addiction gets better and sometimes it gets worse, but ultimately market forces have not been able to fix the problem for us. Do you have a reaction to that?
Yeah. That’s a good example of how more optimisation pressure in our current civilisational incentives is going to probably be a bad thing in a lot of ways. There’s also going to be a better ability for us to see what’s happening and coordinate thanks to these aligned AIs. So these two forces are going to be working against each other, and one of our central claims is it’s not at all clear which one is going to dominate in the long run.
Let’s talk about this sort of paradoxical element to all of this, which is that we’re imagining that the AIs that we’re operating, that we’re interacting with, really do have our best interests at heart. For the sake of argument, that’s the scenario we’re picturing. How is it that that wouldn’t protect us to a great extent from going off in these negative directions? They would help us see that things are going badly. They would be trying to anticipate these negative outcomes, and then warning us, “You shouldn’t be absorbing this kind of culture; you should vote in this direction, because that will help to protect your political rights in the long term.”
That’s a great question. And I feel like that’s the biggest reason for hope: we’re also going to have AIs that — to the extent that they’re aligned to us, and to the extent that they might have good abilities to forecast what’s actually going to be good for us — will be able to help us navigate this crazy world and help us choose the right memes, help us tame our government. Everyone who’s worried about this issue is saying that if we have good outcomes, it is probably because these AIs are really aligned to us, and are really giving us great ability to deal with these new scary forces. There’s a couple of reasons to think that this might not be enough, though. One is that coordination might still be hard. It can be the case that everyone can see that something bad is likely to happen, and it still can just be very hard to coordinate not to have it happen. Again, the classical example is world wars, where everyone can see maybe that World War I is brewing, or World War III or whatever, and all our efforts to make it not happen kind of contribute to the problem. Maybe another simple example of a sort of tragedy that we can all see unfolding is companies becoming more bureaucratic over time. We all see it happening. We all know it’s happened. It’s happened a million times before. It’s not really that much of a mystery, but somehow it just happens. So I think it could be the case that there’s just a lot of coordination problems in the world that look like this — that even though you can all see them happening, everyone’s getting good advice, it’s just so hard to address them globally. The other reason to be scared is it’s not clear that states or corporations or whatever powerful entities in the future are going to allow people to have AIs that are truly aligned to them. I think everyone agrees that we shouldn’t allow AIs to help people build bombs or be terrorists. But it’s also a question of like, what about hate speech? What about if I want to organise a really effective protest or a coup or something? What if I really don’t like my government for good reasons? Probably it’s not a stable situation for the government to allow the AIs to really effectively organise against that government. So there’s a bunch of reasons to expect that the government might have AIs that just do whatever they say, and everyone else is going to have the hobbled civilian versions that aren’t actually allowed to be totally aligned to them.
I see. So you can imagine a scenario where I’m nervous that my rights or my influence over the government is going to be gradually eroded. But I guess by that point, we were imagining that people — I guess some actors within society, or some people who have greater influence over the government than me — they might well have made it such that all of the most powerful AIs that I have access to are not really going to help me to unwind their power to weaken them, to figure out how would we organise to do this, or give me the ideal advice that would help me to gain power at their expense. I guess this is a dynamic that we see in general: very often, organisations or individuals who have a lot of power figure out ways of entrenching it by doing all kinds of things to interfere with the people who might try to take back power from them. And this would just be another instance of that very common historical pattern.
Yeah, exactly. And the fastest growing, most growth-oriented institutions in this world, like governments and corporations, are going to have an interest in sort of marginalising humans to some extent — because humans, from their point of view, will be like these meddlesome parasites. So you can imagine that there’s humans that are advocating for, “We legacy beings deserve some huge fraction of GDP, or at least some very expensive protection of our interests” — at the expense of maybe some new flourishing society of AIs or weird AI-human hybrids or whatever is most memetically politically fit. You can imagine a response by the more growth-oriented, machine parts of society: they might end up making the case that this is a speciesist sort of demand, and that we can’t have this sort of narrow-minded policy setting. And it could be de facto illegal to advocate for speciesist policies or something like that.
I see. I think people will be able to tell that there’s a lot of moving pieces in this picture. There’s a lot of different mechanisms by which potentially humans could be losing influence over the direction of civilisation and intelligent life. I think there’s still a lot of work to be done exploring them and figuring out which of them are most powerful and which of them potentially could have countervailing forces that control them.
What are some specific narratives or scenarios, ways that this could potentially play out for people to picture in their head as they’re trying to think about how gradual disempowerment would look?
Well, I think what’s most likely to happen in real life is that we have some sort of gradual disempowerment happening — and it’s sort of happening in a few small ways already. That maybe happens for a while until maybe much of the military is automated, or people have much less connection to the organs of the state. Then probably there’ll be some more classic fast runaway loss of power — like a coup or some new weird quasi-cartel government that just somehow takes over in some way that we don’t really expect. So I think that weakening our connections to the organs of the state and just understanding what’s going on is going to be one of the precursors for some faster loss of control. I think the point we wanted to make in the paper was that, even if there’s no fast loss of control, we still might end up having similar loss of control just through the normal business-as-usual dynamics.
We have an episode from earlier in the year with Tom Davidson about the possibility of a human-driven coup assisted by AI. For people who haven’t listened to that, and won’t go back and listen to it, do you want to explain a little bit why it’s more plausible in this post-AGI world for a group of human beings to seize power in a way that’s very difficult to reverse?
Sure. The basic idea is for similar reasons: that we would have less control over the government. Also the government would have a harder time avoiding being couped by the military if there aren’t a bunch of human soldiers in the loop. Right now, if somebody goes on the radio and says, “I’m the new president,” everyone can kind of check that actually that’s not the case. I know a bunch of soldiers, I call them, “What’s going on?” They say, “I don’t know what this is” or “No, he’s not. We still have control.” In the world where it is actually just a few people who have sort of sysadmin access to the robot army or whatever, there’s not all that much that the government or even the people can do to tell who’s really in charge, except by some naked show of force, and for the government to stop whoever ends up getting sysadmin rights from just de facto taking over the government without having to convince everyone that they’re legitimate.
Yeah. I guess the broader point is, inasmuch as hard military power is the robots, is this AI-driven military equipment, then anyone who can get the AI to follow their instructions has all the hard military power. They can stage a coup. How would you undo it?
Because people are too weak to actually fight metal.
Yeah. And right now, maybe the hard part of the coup is convincing the commanders or the soldiers that this is the new way that things are going to be. You have to convince a lot of people, and there’s this credible threat that they’re going to not agree with your takeover.
And then you’ll be arrested.
As we talk about in the episode with Tom Davidson, there’s many different protections that we could put in place in order to try to make this more difficult, but it’s not completely obvious that we will put in a big effort to do that. And even if we did, possibly we could fail. And it’s one thing that there’s a lot of time in which these coups might be able to occur, and then they’re very difficult to undo once they have occurred — so it’s a bit of a one-way gate, potentially.
Let’s talk more now about the economic disempowerment mechanism. We’ve done a basic intro to it, but I think many people would have the objection that would just immediately occur to them: if it’s the case that all AIs are basically just owned and operated by humans, we’re not really becoming economically disempowered in the sense of having less income. Because all of the work that the AIs would do, all of the profit that they would generate, all of the surplus that’s created by their ability to do amazing things for very little cost — all of that will flow back to human beings who then will be richer and in a sense more empowered than ever before. Why isn’t that such a strong protection that we should feel pretty good about this?
First of all, I’ll say I think it’s a great idea to try to set up these kinds of protections. Probably a good end looks like involving a lot of well-thought-through mechanisms to ensure that surplus is always available to humans. But our basic thesis is that this is much more fragile than it seems like it’ll be intuitively. Again, maybe one example to keep coming back to is: think of the English monarchy, and how they hold all the cards. In fact, maybe a better example is the English aristocracy before the Industrial Revolution: they own all the land, they have all the political connections, they can see what’s happening, they mostly know these entrepreneurs — but somehow there ends up being this giant new source of wealth created that they mostly don’t participate in. And as far as I understand, they ended up a little bit poorer in absolute terms, although the civilisation ended up much richer overall. And similarly with the monarchy, it’s like the king owns everything, has absolute power: how is it that kings end up in this figurehead role where they have very little room to manoeuvre and they end up capturing a very small surplus? But the big picture is that there’s going to be this sort of small rump of legacy humans, maybe who have de facto ownership of this giant machine economy that’s going to be maybe hundreds or thousands or more times as big as the current one. And they’re not going to be producing value, they’re not going to be deciding what’s going on. So at this point, it’s not clear that they end up having de facto property rights respected. And there’s lots of reasons to think maybe we still will respect property rights, and it’ll be very cheap to keep humans alive. This is definitely not a foregone conclusion, and this is one of the fuzzier parts of this whole story. But it just seems like it’s very scary to be this sort of useless head of state of this giant machine economy that you don’t understand. Everyone involved in setting courses, running things, the government doesn’t necessarily share your cultural point of view or even think that you deserve good outcomes in the long run more than some much more interesting, powerful, charismatic beings that you are now competing with culturally. So again, I don’t have a slam-dunk story for how the humans end up not capturing some small rent forever. That’s plausible. It just seems like we’ll be in a very vulnerable position.
I see. You’re saying inasmuch as, at least to begin with, all of the rent is flowing through to humans… I suppose there’s one question, which is this might lead to an awful lot of inequality — because it might be that you can’t really earn labour income anymore, so people’s income is determined by how many savings and investments they had, particularly investments in AI that they had around the point at which humans kind of stopped being able to do useful work. So it could lead to a lot of inequality. But setting that aside, almost all of the income is flowing through to human beings at this initial point. What do you think would happen that would gradually perhaps whittle away the fraction of wealth and the fraction of economic production that is owned by humans or is flowing through to human beings?
One of them is just humans not being that close to the action or knowing what’s going on. Maybe a concrete example today is: if I’m an average Joe, and I think actually AI is going to be the next big thing and I really want to invest in it, most of the big AI companies are still private equity, like Anthropic and xAI, so there’s actually no direct way for me to even invest in them. And of course bigger investors can. But this is a good example of how no one’s making this happen: it just happens that our institutions marginalise almost everyone from participating in one of the biggest wealth-creation events in human history. So again, AI advice helping you avoid these situations is maybe the thing that keeps us safe — but again, there’s just going to be incentives for the AIs that are creating the wealth, that just naturally are going to form these local bubbles of enrichment, which will be hard for the larger mass of unproductive humanity to participate in.
I thought that your objection here might be that initially it might be that AIs don’t have any sort of legal personhood or any ability to own property, but we’ve got a picture that they’re becoming more and more capable all the time. Eventually they’re going to radically surpass human capabilities in terms of economic production, in terms of persuasiveness, in terms of charisma. And there’s going to be potentially a whole lot of diversity in what they’re like, and the kinds of different AIs that people train and release onto the world. And some of them, you might imagine, would want to go out and advocate — and would be permitted to advocate — for AI personhood, for AI rights, for AI wellbeing and so on. And basically this is just not a sustainable situation — that the great majority of beings, who are also by far the most productive and the most intelligent and the most charismatic, that they will forever remain without any kind of personhood or ability to independently pursue their goals. Sooner or later this will crack somehow. And that’s maybe the point at which the AI share of GDP income, or the AI share of kind of independent wealth that they can actually deploy according to their own preferences, that will begin — and then it will just grow over time, because of course they’re able to make more money, they potentially can earn higher investment returns because they’re just smarter. Is that one way that things could go?
Yeah, and that’s exactly what I was thinking of when I was referring to these bubbles of wealth creation, is also all these new beings. This question of legal personhood, I also think it’s very unstable to have the very productive, cutting-edge, whatever is most memetically fit beings just forever not being able to de facto grab the mantle of power one way or another. The problem is that this is all very fuzzy and far into the future, so any particular scenario sounds a bit far-fetched — especially without all of the other parts of the scaffolding set up to realise why this locally will seem inevitable, I think.
So a different dynamic that would be going on here — that isn’t precisely human disempowerment, but is related to it and could help to contribute to it — is I think there are a number of forces that will be pushing us more strongly towards oligarchy in this world than they do currently. What are some of those?
Maybe one way to think about it is: what are the forces that are acting towards equality that are going to stop operating? The main one is just the value of labour. Right now, sort of everybody for most of their lives has some valuable labour that they can trade. And this kind of makes you pretty relaxed, in terms of like, “Whatever the government does, whatever goes on with the economy, probably I’m going to be able to slot in somewhere. And even if I somehow lose all my money, I’ll be able to rebuild from scratch.” Once you stop being able to trade your labour, then basically whatever capital that you have is your one asset. And if you ever lose it, it’s hard to see how you recover. Of course the other thing going on right now is we have redistribution of wealth through the government. Right now that’s not exactly “nice to have” — for a lot of people it is sort of life or death already. And then the stakes will just become even higher for how much redistribution is happening, and if our control over the government becomes less, we should expect that effect to also become less.
OK, so we’ve painted a picture here where I guess initially most of the income is flowing through to human beings. We think it’s probably going to be more unequally distributed than it is today, and it’s going to become probably more unequally distributed over time. Hard to be sure, but that’s a reasonable guess. We think that AIs probably initially won’t be independently owning property and pursuing their own goals, but that’s probably unstable long term. Decade after decade, century after century, is that really going to hold? Seems kind of unlikely. How do you think this AI-driven economy evolves over the medium to longer term?
Sure. So in the short run, this probably looks really good for the average human, in the sense of the cost of almost every service going way down, services just getting way better, and the cost of almost every good also probably going way down. And there might be actually a long period where things are kind of OK — in the sense that humans are sort of disempowered, but there’s not really much pressure to disempower them more. We’re all just sort of enjoying our luxury apartments or something like that, while the machine economy is just growing and growing. Then I think this sort of scary phase comes later on, when there’s been enough doublings that some basic resource is starting to become scarce again. Maybe it’s land, maybe it’s power. I don’t really have strong opinions on what exactly it is, but the idea is that eventually we do hit some sort of Malthusian-ish limit, and we have to actually start competing with the machines for some basic resource. Of course, along the way I think on a faster scale this sort of Malthusian competition for political power might be lost by humans. But let’s just not worry about that for now.
OK, so what happens is the economy continues to grow. Humans, possibly even those who are receiving a small share of the income, because their productivity has risen so much, because the economy has grown so much, their absolute level of income might still have risen. They might be much richer or able to consume much more than they can today. But then the next challenge for them would be that, as the AI and robot economy basically expands across the entire Earth and is doing all of the productive stuff that it can, humans to some extent get edged out in terms of literally surface area of the Earth. You need energy, you need space to grow food and to have a comfortable environment for humans — and the opportunity costs of setting aside that space for human beings to live and have a good time and grow their food and so on is going up, because technology is advancing. We’re figuring out how to squeeze more and more AI, more and more productivity, more whatever we value out of each square kilometre of surface on the Earth — so it’s becoming potentially more expensive to keep humans alive than it was before, at least in terms of what we’re giving up. And then possibly some humans won’t actually be able to afford that increasing price, because their income won’t be going up as fast as that.
Yeah, exactly. The scenario I have in mind is that I have my like one acre of land or my big luxury apartment with me and my family. And we’ve made our peace with kind of opting out of the economy, but we have our little sort of commune or whatever that we’re happy to live in, in unimaginable luxury and wealth in some senses. And the government or the rest of the economy or something starts to view this as sort of criminally decadent — that this small group of humans, like maybe 10 or 100, are using this entire acre of land and this amount of energy and sunshine to keep these small brains working for no particular benefit but their own, when those same resources could be used to simulate maybe like millions of much more sympathetic, morally superior (on whatever axis) virtual beings. So it’ll start to be seen as selfish — as you say, high opportunity cost: a sort of irresponsible use of resources to keep some legacy humans around.
I’m just guessing that by this point, surely there has been some kind of agreement that humans are going to have some fraction of the Earth. We’re going to be sort of grandfathered in. Either we’ve been killed or there’s going to be some sort of agreement that we’re going to be allowed to have some section of the Earth in perpetuity in order to support ourselves while the robots go off. And the Earth is very small in the scheme of the entire universe, and it’s a lot easier for AIs and robots to go and use all of the resources in space. While the opportunity costs in an absolute sense of setting aside half of the Earth’s surface for human beings to do their thing on would be very big, in terms of proportion of all the available resources in the universe, or even just the solar system, it’s basically completely negligible. So why do the AIs care so much to kind of squeeze that last bit of space and energy away from the human beings?
First of all, I want to say this isn’t exactly exotic. When we think about land taxes today, often people say land taxes are good or property taxes are good because they force, for instance, old people who have bigger houses than they need because their family has moved out to move into smaller houses to make room for new human families. So this dynamic of taking the old people’s stuff because they don’t need it, then we can house more productive, sort of maybe more deserving people to use the same resources is something that we already do today. So just having property taxes or land taxes is one way that we end up losing our wealth and being forced to upload or something. The second question though, of like, why haven’t we been grandfathered in and made some deal? Well, who are we making this deal with? If there is some sort of global entity that can make promises on behalf of everybody, then maybe this is plausible. It’s kind of like if you’re a monkey living in a jungle next to a city, and you’re kind of worried that the city’s growing. If there’s some unified government, then they might be able to say, “Our value system is such that we want to keep the monkeys alive, and so we’re going to have a protected forest.” If there’s no such unified polity or whatever, and it’s just like a piecemeal growth thing, then you do expect that at some point there’s somebody living right next to your land, and then they have more kids and they want to expand onto your land, so that particular person actually does have a big incentive to take your last bit of land or something like that. So in a sense, this makes me think that the only stable good outcomes involve some sort of strong global coordination — which is also very scary, because if you get global coordination wrong, then you end up locked into bad scenarios as well.
I mean, maybe I’m naive, but I figure by this stage, we’re very far along into the future. Presumably we do have some sort of wave of settlement of the solar system and potentially other star systems beginning to occur around this time. I would think that at that stage, just because we need to figure out how to divide the resources that are not on Earth between different powerful groups — potentially China, the US, I don’t know, maybe other entities that want to have a say and get their share — in order for that to go in a way that’s not extremely violent or not extremely competitive, we would want to have done some division and figured out how are we going to share the surface of the Earth between AIs versus humans? How are we going to want to share the resources in space so that we don’t just fight over them and completely waste them all in the process? So perhaps I kind of already have the picture that we need some greater level of coordination than now, or we’re just going to get towards a more catastrophically bad outcome relatively quickly.
Yeah. I totally agree that it’s easy to make the case that, in the long run, some sort of global coordination is stable. Once you have a lot of agency and coordination, you can use it to get more. The question is, will that government, will our sort of values survive the chaos in the generation of that global power or whatever it is? Basically, there’s this window where right now humans have a lot of influence — our institutions and cultures kind of serve us — but we don’t have very good coordination ability. Then at some point, we’re going to have diminishing influence and diminishing cultural cachet, and all these things at the same time as our ability to coordinate is going to go up. So the question is like, what’s going to happen first? Are we going to be out-competed and then later the machines coordinate, or do we get to be part of that global coordination process?
Yeah. I appreciate that what I’m about to say might sound a little bit strange to people who haven’t been marinating in these ideas around AI for months or possibly years. But one thing that’s worth adding is that many people believe that in this AI-dominated future, it will potentially be a lot easier to coordinate between different groups and to form agreements that are kind of stable over the long term. Because you could design sort of an AI hegemon that everyone could inspect and see that it really does want to follow through and enforce this division of resources between all of these different groups. So you have this ability to enforce agreements between countries, between very powerful actors, in the long term. Even if the agreement involves one of them becoming much more powerful than the other one, that will continue to kind of be enforced in a way that has never been possible in the past. So that’s one sort of hopeful picture. I guess it could also go in a bad direction, but it’s one way that we could potentially solve these coordination problems going forward in a way that we have not been able to in the past. And usually we’ve ended up in just violence or conflict when such a case has come up.
Yeah, exactly. That’s exactly what I have in mind when I’m saying that our ability to coordinate will increase. And then the question is, will human influence survive long enough to strongly influence that process?
So you think that we could go through a stage where there kind of is an agreement, or the government is operating such that it is redistributing income to humans or human-like entities enough that all of them can survive and potentially have a pretty good time. But you think this introduces some interesting wrinkles and could be gradually undermined over time, such as that it doesn’t last long term. How might that happen?
Yeah. The only thing we can do is to introduce a new fitness landscape, where who the government deems worthy of UBI is the new fitness function that is going to be just optimised by natural selection. This is kind of what Dan Hendrycks talked about.
Let’s say that we did have a policy that any human has to receive some sort of universal basic income that’s sufficient, some amount that we think is enough for them to have a good life. I guess in this case nobody’s working, so they can do kind of whatever they like. Some people in this situation presumably would like to have really big families. They might like to have a tonne of kids. But now, because those children are not economically productive whatsoever, this is imposing big costs on everyone else who’s basically being taxed to support this human universal basic income. So without going to more extreme cases where people might really try to milk the system with foetuses or becoming uploads and copying themselves or something, you can imagine that this would require some sort of restrictions. If we’re going to have this universal basic income for humans being sustainable, probably it would have to come with some restrictions on reproduction, or limits on what new beings can qualify.
Yeah, absolutely. If we just allow unfettered reproduction like we do today, then it’s unstable in the long run. The problem is it’s kind of like, well, what do you even want? I think the best we can hope for is that we do live under such a regime where we’ve thought hard about what is the sort of fitness function or how we redistribute wealth or something. And the fear is that we don’t think it through very well, and we end up locked in this new Malthusian condition where everyone has to have as many kids as possible or something like that to maintain their share of wealth or something like that.
Would human disempowerment be morally bad? One thing I’m a little bit confused about is: what’s the moral philosophy underlying this whole perspective? Is it that this is bad because it’s going to be bad for human beings; it’s going to lead to an Earth that is not a good time for you and me and our kids? Or is it bad because it’s going to lead the rest of the universe to be kind of wasted on something that’s useless or harmful or not as good as it could have been? What’s the moral perspective that you’re bringing or that your coauthors are bringing?
Sure, sure. I think a lot of people might say, “You don’t have much moral imagination. Why are you insisting on these human wellbeing or human desires, when we know that in principle there’s definitely going to be more morally deserving things in the future?” Or something like that. My basic answer is that in some sense we decide what is morally deserving. And it would be really surprising if, for those beings to exist in the best possible world, we all had to die and have some terrible time. So we basically don’t have to decide between these different views, and we can just say, let’s try to make sure that something like existing humans get to decide roughly what’s done with the rest of the universe or the future or whatever. If that involves having these sort of Amish style, leave Earth as a nature preserve, whatever it is, let’s just let ourselves decide, and not let it be up to some sort of race to the bottom Molochian dynamics, where we end up choosing something that no one endorses.
Yeah. I think the reason this matters is that some people, the thing that they want to do is work to ensure that humanity has a great time, or that the Earth is good for themselves and their children, which is going to raise one set of concerns. Other people want to use their career, or the thing they want to lean on is ensuring that the future of civilisation or the future of humanity or the future of intelligent life is good. Do you think that the case for worrying about gradual disempowerment is stronger on one of these than the other? Or do you think that they tend to go together?
They’re basically the same. I think it would be really weird if we somehow accidentally killed and disempowered existing humans, and ended up building some future that those humans would otherwise really endorse. I think that the default is there’s some locust-like beings that just like growth for growth’s sake, and that’s the default thing that all evolutionary pressures select for. And maybe those beings are pretty cool, I don’t know — and if they are, then it doesn’t really matter what we do, so we don’t really have to worry about that scenario. But the scenario where they’re just kind of like this grey goo that we think is a big waste, that’s what we need to avoid. And if you and I are on Earth, flourishing for a long time, and the state and all our civilisational apparatus is acting in our interests, and we decide that actually it would be amazing to create this type of future, then just as part of serving our interests, we would end up creating that amazing future.
I think it’s not such a given that they necessarily go together. I guess you were saying it’s like humans who decide what is good. I suppose you’re like an anti-realist?
Yeah. But that doesn’t mean that I don’t take morality very seriously. I’m just trying to say the sort of fact of the matter is determined by what’s in our heads, and then also whatever conditions that imposes on the world being good.
Well, I guess if you think that there is something that is objectively valuable, independent of whether people believe that, then you could have a future in which humans are disempowered and perhaps the machines end up going and doing that. And no human would have endorsed it at the time, but that could still potentially be a good thing. I guess if you have also a view on which it’s good to satisfy human preferences, but it will also be good to satisfy machine preferences or AI preferences at the point perhaps where they’re conscious or they have subjective experiences, then you might be a little bit less stressed about handing over control or handing over resources to AIs to pursue their own agenda.
I think it’s a really weird corner case to imagine this world where we die, but then our desires are ultimately fulfilled. That just seems like, yes, in principle it could happen, but it would be this weird corner case — because probably if we die, something else has gone horribly wrong.
So you’re saying, whatever it is that we want to happen, why don’t we just maintain control, so that then we can decide whether that is the thing that is happening or not?
So it could be that by coincidence we get disempowered, and then the thing that we would have liked to happen happens anyway. But why leave it to chance?
OK. How much is this picture complicated by the fact that lots of humans disagree, and have very conflicting preferences about how things might go?
Right. I mean, I think that just puts a cap on what’s the best we can hope for in terms of satisfying everyone’s preferences. But we were already sort of in that situation. So I guess I’ll say, given that we already have to have some sort of compromise, or not everyone’s going to get what they want, we should at least work to make it so that at least some people or some compromise of humans gets what it wants — as opposed to just the competition gets what it wants.
OK. What’s the strongest argument for not worrying about this, in your mind? There are people out there who think it’d be good for us to hand over to the AIs; it’ll be good for humanity to be disempowered sooner or later. Maybe sooner: we’re not as smart, we’re not as wise, we will squander the resources. We should be handing over to our AI children. Could you defend it?
Sure. First I’ll say the most common arguments I hear in favour of this. One is that pretty soon we’re going to have these amazing AIs, so they’re going to handle this for us and we don’t really need to worry about these kinds of coordination problems. I guess I feel like, yes, if there was a big jump in capabilities, and everybody got it on the same day, and everybody asked their AI, “What should I do for the good of humanity?” and did that, then that would be a recipe for really good outcomes. But I don’t think that’s what’s going to happen. We’re going to continue to see people gradually getting more powerful AIs, and those gradually getting spread out according roughly to power level, and people continuing to optimise mostly for their own interests, just due to competitive pressures. So my fear is that business as usual doesn’t give us such a jump in capabilities that we’re suddenly able to coordinate in a way we weren’t before. The other common argument is: if people 1,000 years ago got their way, we wouldn’t have made all the moral progress that we’ve made since then, so it would have been a huge mistake from our point of view today to have let them lock in. So by induction, it will be a huge mistake from the point of view of future beings to have let us lock in. I think that’s kind of a moral optical illusion — in that, yes, if you measure moral progress by difference from our current moral standards, then if you go backwards in time, the further back you go, the worse things get, sort of monotonically. So if you just extrapolate that, it looks like if we go forward in time and we continue allowing this moral progress to happen, things are going to get better. But I think what’s actually happening is we’re just measuring difference from our current values. So unintuitively, we should actually see this line have a kink exactly at the present day, and things just continue to — however moral standards evolve — look worse and alien and wrong from our current point of view. So I think from the point of view of future beings, whoever’s alive is going to be glad that the past wasn’t able to lock in their different values, but they’re basically going to be happy that they’re in power. So in a sense we don’t have to worry about the future beings; they kind of won no matter what. The only thing that remains to worry about is locking in our current values in some sense. And of course we probably don’t want to lock in all these short-term, maybe irrelevant details or local adaptations. We probably want to lock them in at some larger, more abstract, big-minded view. But I think we do, by definition, want value lock-in — because if you’re OK with your value changing, it’s not really clear in what sense it’s actually a value of yours.
I think this does hinge on moral anti-realism, right? Because if you thought that there were just objective moral facts that are mind independent, it’s more like science, then I think it is true to say that if we’d locked in our views on the natural sciences in 1000 AD, that just would have been worse, it would have been more wrong, and people just would have been making errors all the time if they’d locked it in so they couldn’t change their mind about that. I mean, it could be that there are objective moral facts but we’re not getting any closer to them, or we’re not likely to ever find them out. But you could have a view that they exist, and we’re getting closer to them — in which case they would disagree.
I totally agree. But in terms of what we should do, I think it doesn’t end up mattering. Because even if we’re moral realists, the question is: do the AI successors that we build care about that morality? I think the default is no: they care about growth, just in the way that animals don’t care about morality, and evolution doesn’t care about morality. So if we happen to somehow today care about the true morality, we need to preserve that. The natural course of history I don’t think is going to preserve it by default.
Well, let me give you a picture in which someone could be actively in favour of human disempowerment. Let’s say that you think the thing that I really value is either wellbeing or satisfying preferences or something like that. And I think that AIs in future will be able to have their preferences satisfied, and they will have preferences, and they will potentially have wellbeing as well. But I think that most humans disagree, and they’re going to try to basically use AIs for their own purposes and not be concerned about their preferences, not be concerned about their wellbeing. That’s the moral atrocity that I’m concerned might occur indefinitely, and people would lock that in forever. So in fact I might be in favour of AIs basically taking over and seizing the reins so that that possibility is precluded, and that they will have some control over resources and their preferences will get satisfied. Does that make sense?
That makes sense. And maybe I would unfairly characterise that as a corner case where it happens to be that taking your hands off the wheel is the morally best action. Which I feel is a bit of a coincidence, or it would be a sort of happy coincidence. Just because we already evolved civilisations that, for instance, are making AIs that maybe don’t have rights, or that used to have slavery; all sorts of oppressive governments arise naturally throughout time. So if we did take our hands off the reins, we might expect that the future AI civilisation would create their own sub-AI slaves or something that were treated poorly. So if the idea is that it’s fine, they’ll be enlightened enough not to do that, then maybe you’re OK. But I feel like it’s weird to think that they’ll somehow recognise this as this really important moral truth that we don’t see.
in future AIs or the AIs that will exist as a kind of a different player, and it'll be like, "Which do I want to have inside my coalition, or which group do I want to side with? Do I want to side with China, or do I want to side with the values that I expect the AIs to have?" Depending on your guesses about what kind of moral attitudes or how much value the AIs will be able to produce, you might end up just saying, "I would rather side with at least some group of AIs over my fellow humans." That's one way that things could go that actually makes it more likely that disempowerment could occur.
Yeah. The question though is what sort of dynamics are you setting up, right? If you just say, "I'm siding with the AIs," but they themselves are still in such a competitive race, then you still are sort of taking your hands off the wheel and saying that their culture and morality is still going to evolve — and where do we think that's going to lead us is the important question then.
Yeah. I think one reaction you've had to the paper is people saying, "Are you really saying that we want humans to remain, to have hard power, to be the people on which the decisions bottom out, forever?" The AIs are going to be getting more and more capable relative to us, even if we try to augment ourselves one way or the other. Surely at some point we have to give it up and say we've done what we can — we think that we've aligned the AIs, or we think we've given them good goals, and it's time for us to take our hands off the reins. Did you agree with that?
I guess I'll say that's conflating two positions, one of which I agree with and one I don't. I think almost by definition we want human values to rule forever. And if you're saying, no, I want some progress or evolution to happen, it's like, great, that is then now the value that you want to lock in forever, even if it involves lots of object-level change. As for whether humans are actually in the loop making decisions, I think it's probably the case that whatever you want to optimise, eventually the good end does look like this is mostly handed off. Then the caveat is that, on reflection, if the AI has ever cared to ask the humans, "Do you agree with how we've been running things?" that the humans would be like, "Two thumbs up. That's what I would have done. Keep up the good work."
Yeah, I feel like it does complicate it. You say we want them to be aligned with human values. I think that's fine as a shorthand, as a first pass, but I think it really does complicate things that people disagree as much as they do about what the future should look like. And also, while I guess different people with different moral philosophies or different perspectives on the world, different religions, different spiritual values, they often kind of agree about what they would like to be the next step. They'd like people to be more educated, healthier, more empowered. But as there's more capability to optimise for exactly what Buddhists might think is the optimal thing, exactly what adherents of Islam or Christianity or different secular moral perspectives might want, then they might come radically apart — such that there'll be some humans where you're like, "Yes, I would love them to have more power, but many of the other humans, I would like them not to have power because I think what they're going to do is terrible or useless." And there'll be some AIs that are aligned with you and some that are not. It's going to be quite a fierce potential fight.
Yes. I think that's a huge problem. But it's also a huge problem for anyone who has a proposal about what to do with the future. So it's not like this particularly affects my claim about how we should run things.
I see. So you're just saying that that's like an orthogonal issue that's not super related. I guess people in general use this shorthand because they maybe want to set aside this internal human conflict issue and say, well, people need to think about that. But that's a problem always.
Yes, exactly. The other thing I want to say is I think people have stronger preferences about the future than they think they do at first blush. I've had a lot of discussions with people who've thought a lot about the future, and they sort of say something like, "A thousand years from now, if it's aliens or robots or humans running the Earth, do I really care?" I think there's this exercise you can do. I think natively we don't actually have strong preferences about the future, because it doesn't matter that much — especially the distant future — for our actions. We only end up having preferences about things that we spend a bunch of time practicing taking actions over. Maybe a good example is if you try to ask a dog, "What house do you want to live in?" or, "How do you want to be treated a year from now?" You might say there's no coherent sense in which the dog has preferences about this. But it does have a whole bunch of short-term preferences that could be chained together, and you could try to elicit those and run them forward. Same thing with humans. My initial reaction, if I think about the world 1,000 years from now, and some future race has taken over Earth, maybe I'm not bothered by that. But then I think, wait, wait. Right now I have kids, and they're going to be having kids, and if 1,000 years from now some other race has taken over, where's the day where my kids get killed or starved or replaced or uploaded or whatever? It's one of these things where I think it's a skill, basically, having coherent preferences about the future. And the more people spend time actually thinking about it, I think the more they would say, "Wait, I actually don't want to take my hands off the wheel. The default competitive future probably is missing a whole bunch of stuff that I care about."
Yeah. I'm not sure about this, but I suppose the gradual disempowerment framing places a lot of emphasis on human versus AI control. An alternative framing would just be disempowerment of my values relative to other humans who disagree. I suppose maybe that's not a new idea; you maybe wouldn't write a paper about that.
Actually I would push back: I would say the gradual disempowerment paper doesn't really talk about human versus AI control. It talks about the alignment of our current institutions, and how they're going to become less aligned once AIs are more in the loop and taking over from humans.
I see. Is that more compatible with the competition between different values among humans?
Do you want to elaborate on that?
To me, again, the question is not AIs versus humans; it's more like: what are the idiosyncratic things that humans value that we don't expect to be competitive in the long run versus the sort of values of unfettered competition and natural selection? Some things that we value — like communication and sight and memory — we don't have to fight for those. Whatever future beings are around are probably going to be able to see and talk and remember. But idiosyncratic things — like children's laughter or the family farm or the particular languages that we speak or something — are things that will be out-competed and replaced if we don't preserve them. So to the extent that any of us value anything that isn't going to stand the test of time, we sort of have to accept that we're going to lose those, or try to somehow do this crazy task of aligning our whole civilisation to protect these idiosyncratic, non-competitive values.
OK, let's talk about political disempowerment now, and expand on that a bit. How do you imagine that happening and progressing over time?
Sure. One thing is just that I think human politicians will gradually let themselves be more puppeted, or become like passthroughs for things like ChatGPT. And this isn't necessarily a bad thing in the short run. Good politicians already rely heavily on human advisors, and I think machine advisors are going to be able to make our political parties and representation mechanisms work better in a lot of ways. So the politicians that use AIs just for normal everyday business are going to be more effective, and we're going to feel like they represent our interests better, in the short run at least. The other big thing that's going to be going on is I think people are going to be afraid of losing their jobs. And every politician is going to have something to say about this, and say, "I'm the 'pro-human, no AI is ever going to take your job' politician." But they're just not going to have viable policy levers to actually slow automation. And just in general, people don't get votes on things where the government is really constrained or they think it's important enough. No one ever had a referendum on should we build nuclear weapons, for instance. I think it's also going to be the case that governments' hands will just be tied. They'll all say, "I'm going to have humans in the loop or human oversight or more direct human representation" — but it's just going to be so ineffective that when the rubber hits the road, those policies are just not going to be implemented. And it's going to frustrate voters every time, and they're going to say, "No, the next time we want to vote for the one who's really going to represent human in the loop interests" — or whatever it is that seems most scary — but they just won't be able to vote for their policy preferences.
OK, let's take that bit by bit. The first part was you're imagining that progressively politicians are going to be acting almost entirely on the advice of AI advisors, AIs that they're operating. I don't think that you think that that per se represents serious disempowerment — inasmuch as I'm working with ChatGPT to help me do work that I want, but then I choose the answer, that seems kind of fine in principle. But you think that's just the starting point, and then kind of progressively humans just are not really involved anymore?
Yeah. Really, whether the government is made of humans or not, there is a slight oversight feedback mechanism and being connected to the people that helps align the government. But really, the fundamental thing that is making governments treat us well is that they need us. Even the North Korean government has to feed its farmers and its soldiers or whatever. So if humans were still indispensable in some important roles, I wouldn't be very scared of a machine government. And conversely, if I was dispensable in all aspects, I would be very much more scared of a human-run government — if you think of the very worst governments in history, like the USSR, or I think Cambodia maybe takes the record for killing the largest fraction of their population. But in general, no matter what political party ends up in power, they don't, even in the very worst case, end up killing more than some small fraction of their population. Life can only get so bad when you're needed. That's the real key thing that has been keeping governments aligned, and that's the key thing that's going to change.
I see. So imagine that you didn't live in a democracy now: you're living in a country where there is a small elite that basically makes all of the decisions. How would things change in this new picture where you're even less necessary for them than before?
Maybe the closest contemporary analogue is Saudi Arabia, where the government basically gets most of its wealth from oil. And there's this nice sort of grandfathered-in class of all the cousins of the many princes or whatever who have these sinecures, these make-work jobs. And they also have very little political room to manoeuvre, right? There's purges, there's drama at the top, and anyone who's making any sort of problem for Mohammed bin Salman or whatever is probably going to lose their sinecure at the very least. Then maybe the example of the same thing done well is Norway, where there was this strong democratic government, and the government was able to commit — at least in the short and medium term — to redistribute this wealth.
So in the UK at the moment, it's true that the government, inasmuch as it's separate from the population, does need its people to continue working and paying taxes and doing stuff. But if very quickly we all were no longer required, and were all replaced by AIs in businesses, it's not obvious to me that immediately the country would start becoming autocratic. It feels like there are other cultural things going on or just even preferences among the people that would cause them to not immediately want to become an authoritarian country and just cancel elections. Do you agree with that? Do you more think this would happen gradually over time? Sort of the support for democracy would wane and the ability of ordinary people to object to things if they change would just become weaker and weaker over time?
Yeah, more the second one. Maybe one sort of spicy example is in Canada there were a lot of lockdowns during COVID. And I think the public health people would have preferred to have longer lockdowns and the government was just forced to end them or make them shorter than they otherwise would have been because they actually needed people to get out and run the economy. Another example is in New Brunswick, in another Canadian province, all private activity, just people going for walks in the woods, has been banned until the fall because of fire hazard — there's lots of forest fires in Canada right now — but industrial activity is still allowed. I guess I would say it's just so tempting for all sorts of public health and environmental and safety reasons to curtail people's freedoms and movement all the time. And this is always sort of just barely being kept in check by the need to have free movement so that there's an economy. And maybe more concretely, also during COVID in Canada, there was this famous truckers' strike. There was this vaccine requirement to cross the US border, which truckers didn't like, and they basically formed this convoy, drove all their huge trucks to the capital, and they just sat there and honked for a few weeks. And the point is, in the future, no one will have big trucks. Any means that they have for civil disobedience are just going to be much easier to take away than they are today.
Imagine that Canada or the UK did end up with a leader or governing party that was keen to basically whittle away at democracy and basically install themselves indefinitely. How would you imagine that progressing? It sounds like you're saying they would have all of the tools that they have available now, and then the options for resistance among the broader population are just so much weaker than they were before?
Yeah. One thing I'll say is I don't exactly fear some new particular party getting in power and staying that way. Rather, it's going to be more that any party that does get in power is going to be so constrained by competitive pressures that they are forced to basically disempower the population.
Well, like I said before, if you let people actually do the civil disobedience or whatever that they can do today, roughly that kind of is tolerable when most people have jobs, most people have a bunch of important responsibilities, and they can't all just block roads all day or something like that. But in a world where maybe 30% or 40% of people just have this huge amount of free time and energy, it just will be untenable, and the state will collapse if they actually let everybody do this sort of agitation at the effectiveness that they can today.
So it seems like there's a bit of an internal tension here where you're saying, on the one hand, people are going to lose their political power, but they'll have more time to make trouble than ever and more ability to make trouble than ever because they'll be able to get AIs to assist them. You're saying it's almost because they're able to be such strong advocates or be such potent activists that the government will feel the need to crack down on them, and that will be the proximate cause of them losing their political freedom?
Yeah, exactly. And then the other thing is that there was this countervailing force where you just need people to go to work, so they have to be able to move freely and do their own business without constantly getting permission from the government. And there just won't be that pressure on governments to allow freedom anymore.
I must admit, I'm not sure how fiercely competitive the geopolitical situation will be, whether it'll be sufficiently intense that the prime minister or president of a country would feel like their hands are tied; they just have to take away the political freedom of people in their country in order to keep up and avoid too much unrest. It just feels like that effect just might not be powerful enough to overwhelm the fact that the population will have an interest in defending their rights. And at this point, we're imagining they still have substantial wealth, they still have substantial ability to agitate.
Yeah. I will say that, again, coordination could save us here. The sort of saving throw is that all the leaders and everybody is seeing what's happening, realising that there's this tragedy of the commons happening, and somehow coordinating early enough and hard enough that they avoid these races to the bottom. Maybe one way to think about this is to turn it on its head and be like, why did countries become democratic and invest in their citizens and have freedoms in the first place? One story told by Allan Dafoe in one of his papers is that the reason that, for instance, Prussia started educating its citizenry in the 1700s was because musket armies were becoming more competitive than the old knights and poorly armed peasants' armies, and that you needed to educate your mass of citizens enough that they could form these musket armies. This was just a more competitive option. And the elites resisted this. They didn't want to have this more empowered citizenry, but they were forced by competitive pressures to become more like these modern democratic, human-capital-invested states. So that being the sort of birth story is maybe evidence that the same thing can happen in reverse.
Yeah. I think it's the case that knights also, I think in France and England, were not keen on the longbow being introduced because that greatly weakened their importance and the need for the state to give them lots of power. Similarly in Japan, I think the samurai were not keen on guns — that greatly weakened their importance and their military competitiveness. So I guess probably a pattern that has been echoed repeatedly. So far the disempowerment you've been talking about is maybe moving from more pluralism in society, or a broader base of political power in society, towards something that's more like autocracy, more like oligarchy. Do you also think that eventually humans will just end up ceding political control completely to AIs?
Yes. But again, I feel like that's not the headline story, in the sense that we already don't have that great of control of our civilisation at the top level. And this has just been sort of OK, because wherever we go, humans were indispensable. So the fact that we switch between monarchies to democracies and capitalism versus socialism: yes, these definitely matter for the quality of life and growth of different countries, but ultimately you're probably going to survive whatever change in government happens. So it's not necessarily really autocracy or whatever particular form of government that's the big change here.
So you're saying kind of regardless of exactly what the system of government is in future, the fact that humans are redundant will, no matter what, end up resulting in them getting a smaller share of GDP and just not being able to control things in the way that they previously could. Where I guess even in countries that were fairly authoritarian, they both had to cultivate the population so they could do work, and I guess they had to give some concern to how they felt about the leadership of the country because they might revolt. And I guess we're imagining in the future, especially if you have fully robot armies or something like that, revolution just might become inconceivable. So you don't have that option that is constraining what the elite can do.
Yeah, totally. That's one of the other effects making the government not care as much about what the people think: their inability to strike or coup or basically advocate for themselves.
It sounds like you think it's possible, or eventually maybe even this elite that ends up with the most political power in the country could end up deliberately or accidentally handing over the reins to AIs or machines of some form. How might that occur?
Well, again, I feel like even if there's a human head of state… One example again we can come back to is the monarchy, where it's like, how did the monarchy end up not actually controlling the organs of the British state? It happened very gradually, and basically all the important decision making and discussion happened in these other organs like Parliament and the free press and stuff like that. But again, whether it's human or AI control at the top I think doesn't really matter. It's just whether the incentives of the state are aligned with those of its citizens. So again, I would be really scared of an even human dictator that didn't need human citizens.
OK, let's return quickly now to the cultural disempowerment — which is, again, the least fleshed out, the thing I think you and your team and other people are still trying to get fully to grips with. What are the best examples of cultural disempowerment that have occurred so far? Are we getting any hints of how this might play out or what we might expect to happen?
I think there was a really good example of this that happened just a couple of weeks ago, which was the attempted retirement of GPT-4o by OpenAI when they rolled out GPT-5. I think for various reasons, OpenAI wanted to have a very simple, only GPT-5 going forward. But so many people had formed really intense relationships with GPT-4o that OpenAI just couldn't ignore the outcry. This is really a good example of this is an emergent culture — and it's something that no one in particular wanted, right? Like OpenAI, by their own revealed actions, was not planning to continue supporting this model — unless there was some sort of like 4D chess sort of thing. And the humans involved with these relationships maybe consider themselves to have benefited, but it was like a more powerful cultural force that was just sort of developed by accident and led a bunch of people to be invested in the welfare, effectively, of this particular model — that's still a very early, not very powerful, not very persuasive model, I think, by absolute standards. The people ended up forming these bonds and valuing the life of these AIs, and then ended up directing the sort of organs of the economy to give it more resources.
I see. I still don't fully understand how that's an example of the kind of cultural disempowerment that you think we'll see much more extreme examples of in future.
I think maybe it's a better example of how humans will end up actually advocating themselves for providing resources for AIs, and that they'll consider AIs to be deserving of rights or resources, or maybe even more deserving than humans. There's going to be a mix, this is going to be a huge cultural battleground — but I think it shouldn't be this exotic possibility that people are going to love AIs or have very strong relationships with them. We see that already.
Yeah. Maybe the thing that comes to mind more for me is this Dan Hendrycks paper, "Natural selection favors AIs over humans." That paper in general raises the question. There will be evolutionary fitness pressures on the kinds of AIs that exist. AIs that are not fit for whatever reason for being reproduced — like make lots of copies of them, give them lots of resources, give them lots of GPUs to operate on — those ones will fade away. The ones that — either because humans are choosing them or for other reasons — gain access to more resources will end up kind of dominating the share of consciousness or the share of all thought that is going on in the world. I guess this is an instance where it turned out that GPT-4o, at least currently, in some sense, was more "fit": natural selection was favouring it more perhaps than people had initially appreciated, because humans still have access to lots of resources. And if we're like, "We love GPT-4o; we want to have lots of instances of GPT-4o operating," then that is what is going to happen. I don't think you're necessarily saying that this is terribly bad, but this will be occurring at all kinds of different levels. There'll be lots of selection pressures that are pushing AIs towards having particular tendencies — either like pleasing people or being extremely economically productive, or being able to compete in some other sense, like to grab resources perhaps a bit aggressively — in order to have more copies of themselves. Those are the ones that will end up having the greatest number.
Totally. And this isn't an example of disempowerment in itself; it was people wanting more GPT-4o, and they got what they wanted. The idea is that this is going to be a self-reinforcing mechanism, where the less people are making decisions, the less their desires are going to decide which future models are invested in or survive, or something like that.
So Tom Davidson, who's been our guest on the show twice before, wrote a response to the "Gradual disempowerment" paper. One of the issues he raised with cultural disempowerment is: inasmuch as humans still have economic resources, they're still kind of the big spenders in the economy. Then, while they might not be cultural producers so much — because perhaps AIs are able to write better books, make better tweets, make better movies than humans can — they might still be the big consumers of culture, so their preferences will end up driving what sort of culture is created. To what extent is cultural disempowerment downstream of economic disempowerment?
So right now, human economic power does make this very strong selection pressure or sort of fitness landscape for culture to be something that humans want. And as long as humans have some absolute amount of resources, there's going to be this ecological niche to produce culture that these humans want. The point we're making is that this niche is going to be very small in relative terms to this much larger, more dynamic economy, of political economy, cultural creation of mostly machine-machine activity. And that's going to be happening on faster timescales and much larger timescales. So if, again, property rights are respected, and we manage to keep control of our institutions forever, it's possible that there's always going to be AIs that have their profitable niche of making human-friendly culture that humans want to consume. The fear is just that we're really riding the tiger, and that there's this giant scary ball of power and optimisation that, if it wanted to, could just run over this niche in a million different ways.
It sounds like the cultural disempowerment becomes most important in this world where we're imagining, somewhat down the line, you have individual AI people that have persistent preferences, persistent beliefs. They really can cultivate culture independently of any humans that are operating them. At that point, we can imagine that there would be just this entirely or mostly separate AI culture and cultural ecosystem, where many ideas could propagate even if almost no humans endorse them. Is that the case, that the cultural disempowerment maybe becomes most potent further down the line?
Maybe. I mean, when we're that far down the line, it maybe doesn't even matter the most. I think we're going to see cultural disempowerment happen as a way of wresting human control. So if it's the case that humans still have an important veto or voting rights that matter in some important way, then that's going to create an immense selection pressure to make some sort of machines that change their mind or impersonate humans in the sense of getting to have votes effectively or something like that. So there's a period where humans still have power, and there's a large selection of pressure to somehow de facto remove that through cultural means, just because that's one of the means the machines will have to influence humans. After that, then it doesn't really matter what the humans do. It might be that if they've negotiated some small niche that they continue to get, there's not much pressure to disempower those humans further.
Let's try another lens on this. It's seemed like, since the Industrial Revolution, that liberalism and capitalism have really been on a roll — to the point that people think these are very dominant ideologies in the current time. I guess the ideology is: You want political power to be widely distributed. You want to have a marketplace of ideas. Disagreement is good. We want to allow people to just come up with whatever ideas they want and form their own judgements and their own opinions. Competition in the economy is good. We want to have lots of different companies trying to deliver sometimes very different products and services, sometimes similar ones, in order to drive down the price. We want to have competition in politics, so we want to have lots of different people's candidates standing for office, and people can decide whether they like them or not. People talk about the open society, open access orders we want to have. If mistakes are being made, then we want other people from outside of the local system to be able to object and say, no, things should be done differently. Potentially you could acquire a company that you think is poorly managed, or start a new political party to ouster people. That whole mentality has been much more dominant in recent centuries than it was previously, when people had very different ideas about how humanity ought to be organised. I think you think that we're perhaps entering the twilight of liberalism, the twilight of this sort of pluralistic order. Maybe first, why do you think that liberalism and pluralism and capitalism have been so successful in recent times?
Right. I think this is very unintuitive. Maybe there's this midwit meme, where intuitively people feel like, "Someone else getting rich is bad because they're taking away my stuff"; and then the more enlightened, unintuitive view is like, no, actually you really do want the rich guy to start companies and make deals and have trade and get to set his own terms in agreements — because it's going to benefit everyone in the long run, and a rising tide rises all boats. As for political liberalism, I feel like all the religious wars in Europe and throughout the world were so destructive that people eventually realised that it would be really great if we could all agree to just disagree and not have strong opinions on who should rule and what is good.
And who should be killed.
And who should be killed. Exactly. So these are all things that are kind of unintuitive to just people in general. And it's still today the case that… I mean, they say if you're not a socialist when you're 20, you don't have a heart, and if you're still one when you're 30, you don't have a brain. Something like that. So it's really tough, because it's unintuitive just how big the positive sumness is from allowing other people to just chart their own course. So I think a lot of people's reaction to us worrying about AI domination gets rounded off to, "These are like just socialists. These are people who don't understand that free trade, freedom is just unintuitively powerful for creating wealth for everyone." And I think I do appreciate that. That's sort of been my default. What intuitively now seems good to me is like, you want freedom, you want innovation — but I am afraid that is going to stop being the most competitive system for a number of reasons.
Yeah, I think this is an interesting issue, because I feel that the sort of liberal, libertarian, pro-capitalism perspective or ideology is so dominant among the kinds of clique that is developing AI, and developing AGI, and even the critics of that, that it's almost hard to even imagine that there could be a different system — or to think that maybe this won't be the most fit system, or maybe it wasn't the best or the most competitive way of organising society in the past, and perhaps it won't be again in future. I don't know. I'm always reluctant to say that anything is a blindspot with people, because I always feel annoyed when people say that about me, but I think this is something that I think will be more salient to other generations of people in the future, or from people who are further away from this present moment than it is to us who are living through it. That's my guess.
Totally. And I want to say that I think this is a tragedy. I'm a huge liberalism enjoyer. And maybe you could say, yes, things are working out for you. But I just feel like in general, liberalism is also just this very fragile thing, and it's this amazing accomplishment that the West just managed to create these norms of let the other guy choose his religion, let the other guy get rich. It's a very fragile thing that I think even today we should still try to protect if we can.
What are the ways in which you think liberalism might be less competitive as a system, and a less attractive, less appealing a way of organising society post-AGI than it is today?
I think it'll be a less desirable way to organise society for a few reasons, but the main one is just the zero sumness of UBI. Right now, when we all create our own wealth, it doesn't really hurt me if someone else creates their own wealth directly from resources. But in the world where we're all just living in some apartments, advocating for UBI, to the extent that the UBI pie is fixed then we're really just like a bunch of baby birds cheeping, and whoever gets food is less food for the other guy. This also erodes the pluralism of values, because the government's going to have to have some way of deciding who gets resources. If they end up having any opinions about what way of life is more valuable or needs to be subsidised more or whatever, that could be a threat to you. So you kind of have to argue, "That guy's way of life is less deserving of resources than my way of life," and now the government is forced to decide de facto who gets subsidised. That's the main effect.
I see. So we almost have to imagine a hypothetical society in which no one can make anything. There's no economic production occurring, at least among this group. There's just a fixed endowment of resources that they happen to have found — a certain amount of food, a certain amount of houses and all of that — and they've got to figure out how to organise themselves. I guess it's not necessarily desirable for me for you to have free speech and to be able to advocate for yourself all that well, or to be able to educate yourself and become more powerful and influential — because it is completely zero sum. The more influential you become, the more you'll be able to advocate for getting stuff that is literally like food out of my mouth, or money out of my bank account. Is that the main thing that has changed?
Right. So that's the big thing that's changed. There is of course a way in which this might not be zero sum. If humanity manages to convince the AIs or whatever government to give a larger UBI overall, then that is the normal, positive-sum thing. So that might not be a slam-dunk argument. The other thing though is that we haven't had to fear domination by other groups very much. We've had strong property rights; I'm not afraid that Elon Musk is going to literally take my stuff, even though he could raise a private army or whatever. We have very little variation in reproductive rates, so it's kind of OK that the Amish live nearby — because even if they're having more kids than whatever other population, that's not going to matter over the course of 50 or maybe even 100 years. Then maybe another thing is just the rough egalitarianism in terms of intelligence and power level of people. There's definitely very meaningful variation amongst humans in terms of just raw smarts, but people often say, like von Neumann somehow didn't take over the Earth, right? And he might have wanted to. These are all reasons why it was sort of fine to just let other people become more powerful in the past that might change in a big way.
To push back on this, what other sort of positive-sum dynamics might continue to exist? I guess inasmuch as people all think that the thing that I want to do is the morally right thing, and there's other people who also are pursuing that goal, even if they disagree about specifically what that is, they might still be in favour of pluralism — because they might think that that will lead them to converge on a good answer. It's more like, inasmuch as people just have raw preferences, and they're like, "I just want to benefit me and you just want to benefit you," there's no real way that we can come to agree on what the good thing to do with resources is, and you have less interest in allowing the other person to advocate for themselves.
Yeah. I just want to reiterate that we should be very, very scared of accidentally crushing liberalism and this positive-sum world that we've created. If I somehow, through this podcast, accidentally contributed to mistakenly just making people think, "You're in a zero-sum game; you need to fight more," that would be a huge tragedy. I really want to err on the side of caution here and say, probably until things change a lot, we really think liberalism is this precious, awesome, amazing accomplishment that we need to foster. So you were saying, what are the positive-sum dynamics that might still operate? As I mentioned, just from a human selfish point of view, making the UBI pie better, and just making us into more sort of impressive, sympathetic beings might be a good use of everyone's time. Because then the AIs are like, "Yeah, these should be more involved in our global civilisation project" or something like that. And that might be reason enough just to keep the status quo of "everybody help each other become more impressive and awesome" or something like that. But beyond that, I think it really comes down to moral preferences and how much you care about the beings, the AIs that are existing. I think they will be very impressive, and if they want to be sympathetic or moral patients from our point of view, they'll be able to make themselves into that. So I really don't expect it to look like there's this alien mass of activity happening, but more like the Shoggoth with the human face. But in a sort of more genuine way, I think AIs will be able to say, "I actually do spend some fraction of my compute thinking about the world in a way that you think is valuable" or something like that. So that question of how morally valuable this giant machine-economy Shoggoth is kind of dominates that question of what positive-sum dynamics still exist.
Let's talk a bit more about this competition issue. I feel like a very common dialectic when people are talking about gradual disempowerment and things along these lines is someone will say, "Couldn't there be this harmful competitive dynamic that would lead resources to be wasted on a zero-sum competition or even a negative-sum competition?" And someone else might say, "Well, couldn't we coordinate to make that not happen, make it good?" Then someone might come back and say that that would be difficult, or even if you did that, then at the next stage there'll be a different sort of competition that will be like negative sum or zero sum, and it'll all end up getting wasted. And it just goes back and forth like this, with, "Maybe we could coordinate in order to avoid that," and then, "Maybe here's a different way that things could go badly." Am I understanding that correctly? Are there any examples of this basic dialectic that haven't come up in the conversation so far?
Totally. I think you're getting at a big open empirical question, which is sort of like: what is stable at the top? Should we expect there to be some big global government that forms and then consolidates power and lasts forever? Or is it going to be the case that, just like the Roman Empire, it became more and more powerful — but then there were new religions that formed inside, and there were cultural competitions, and all these reasons why every time an empire falls it eventually ossifies and some sort of internal competition makes it fall. Because if the case is that we're all heading towards some global government and it's going to be really hard to change, then we need to be investing in ways that we can steer this or slow it down or make room for pluralism or something like that. But if we think that the default at the scales we care about is just that it's hard to coordinate, and there's always going to be runaway competition eating any surplus, then we'll want to invest in: how do we actually slow that process down and make sure that we can preserve these non-competitive values?
Yeah. So we've talked about some sort of runaway competitions so far. Maybe the furthest we got out in time was there's a lot of competing over a fixed welfare pie, a fixed redistribution pie that might be going to humans or other non-productive entities, you might call them. Are there any rounds of competition that occur later than that?
Sure, yeah. One thing I want to say is I mentioned a fixed pie, but the pie itself might be growing, and it might be growing exponentially as the solar system is colonised. But the point is that it's zero sum in the sense that humans don't contribute to the pie growing faster or smaller in the way that economic activity does make the pie grow today. So it might not be a fixed pie, but it's still the case that it's zero sum when you compete with the other people for your share.
