How Humans Could Lose Power Without an AI Takeover (with David Duvenaud)

You could sort of have a situation where the virtual beings almost like dominate the humans in every single axis of moral value, and now suddenly it starts to look like criminally decadent to be spending a kilometer of this land on the legacy of humans. It's hard to draw a hard line between having a permanent income and being like some sort of parasite, and that's going to be the cultural battle. And it's going to be really easy to say that humans are parasites once we're not providing value to like the larger growth engines. Yes, there will be cool, interesting stuff happening in the future if we allow competition to run, but like we just probably won't be meaningfully part of that. A lot of people are basically going to say like, my only hope is to sort of be the first one to bow down to the new overlords and embrace this new culture. Sometimes people ask like, oh, aren't corporations super intelligences? Why shouldn't we fear them? And the answer is because it's made of people, so it needs us. There's this idea of like the singularity, which I feel like has been very destructive because it kind of is like an excuse to turn off your brain and to not model the future and to say, yes, things will keep changing faster and faster until we can't say anything about it. Let me say it loud and clear here is that like, yeah, I think that the post world is just going to be extremely alien and so different that if we could avoid crossing that threshold, I think we should.

Thank you for having me, guys.

Sure. So my name is David Duvenaud. I'm an associate professor of computer science and statistics at the University of Toronto. I've been working on probabilistic deep learning for a number of years. And then I guess in the last few years I decided to use my freedom to try to focus on the problems that seemed most neglected and intractable, which has led me to first work on more technical AI safety. So I was a team lead at Anthropic for a year and a half starting in 2023, working on sabotage evaluations. And then I felt like there's this sort of me and other people were noticing there's an even bigger missing part of the alignment problem, which is how do we align our entire civilization, which of course is even more intractable and even hard to describe this problem. So that's what I've been thinking about lately.

Yeah. So I guess from my point of view, this all started with a lot of lunchtime rants around the table at Anthropic and also elsewhere, talking to all sorts of people, saying, "Okay, so what if we succeed? What if we build these agents that really can do everything we want them to do better than humans, and we're not worried that they're secretly trying to betray us or anything like that?" It seems like, you know, I would ask people like, "What are you personally going to be doing afterwards?" And, you know, people just didn't have much to say. They would say things like, "Oh, I'm going to be clicking accept suggestion all day," or "I'll be taking a bender vacation." And I think I'm a natural kind of worrywart in a lot of ways. But I felt like to me the situation that jumped to mind was being like a retired person who's like out of touch and maybe has some savings or like, you know, vote or some sort of like legacy power. But there's all sorts of more sophisticated actors around that if you like let them act freely and the old person act freely, you sort of know that they're going to eventually lose their money or be somehow rooted around. You know, people used to talk about information wants to be free, and it's also kind of like, I guess, power wants to concentrate on people that are providing value or something like that. So anyway, it wasn't very well fleshed out, but I just felt like this seemed like an obvious thing to worry about, and I was just confused about how other people around me weren't very worried about it, except for a few people like I met. Shane was someone who seemed to get it, and then I ended up, David Krueger introduced me to Jan Kulveit, who had basically already fleshed out a lot of these ideas, and he asked me to join this paper, and we kind of got it into a more like into a very digestible shape and got it out there.

Yeah. So definitely from the point of view of the normal health indicators of society, I think it looks like nothing has gone wrong. And part of that well is that cultures and institutions adapt to measure the things that produce growth. Like that's maybe one of the running themes of all this work is that growth is what matters in the long run, and all of our institutions or anyone that wants to get anything done is going to orient towards the thing that is sort of like effective and scalable, and they just won't be able not to. And so like maybe this is one reason why we talk about GDP so much, even though everyone knows it's a terrible measure of like sort of human flourishing, is because a, it's easy to measure, but b, it is tied to this like the growth engines that have been sort of like the only thing that matter in the long run.

Yeah. And so you mentioned, so what's the distinction between this and like the power grab? And I guess maybe that's one of the other sort of insights here is that on a long enough time horizon, it's not clear what the difference between a power grab is and just normal building good, useful, effective institutions.

So if I somehow tricked everyone into making me leader of like a political party or taking over government, you could say that's a power grab. But if I just show up with my like 10,000 clones and we're just like this ultimate effective political force and we found a political party and we find things that are not being addressed by the current system and we get people on our side, you know, we would say, okay, that's what we want to happen. And the problem is just that if the, I guess, populace ends up being basically a different set of agents which are more like these AIs or their principals, then all of the things that we try to encourage, which is basically like reward the people who are getting stuff done and building and growing, ends up meaning that everything just kind of runs out from under your control, even if there was no explicit power grab.

Right. So one thing I'll say is I think all this does happen just on a very long time scale today, on the time scale of let's say cultural evolution, which means maybe like over the scale of hundreds of years, or even genetic evolution over the scale of like thousands or tens of thousands of years. And you know, if you really do take the values of our ancestors seriously, they really did lose in some really important ways, and they're the similar ways that I'm saying that we're also going to like lose, and that like our culture or values or even just like biological cells will be replaced. So I think a lot of people say, oh, well then this is just business as usual on a faster time scale, like boohoo, you know, adapt or die. And I'm like, well, sure, but I don't want to adapt into something that's unrecognizable, and I don't want to die. And yes, every other being in culture has faced this choice, and I'm saying, well, I still don't want to just give up and sort of take the options as they're presented, or at least I want to understand what the landscape of possible actions is.

Well, the economic effects obviously are sort of like the first order thing is like people lose their jobs and then they are unemployed. But of course, we expect that there to be this like compensatory government initiatives to say like, well, let's give people UBI or some sort of make-work jobs. And I think those are going to be very important forces over the next like five or 10 or 20 years. And I guess one of the main arguments of the paper is that these will be unstable band-aids. Like if you could get them to work and be robust, that actually might be a pretty good sort of like end. But they won't be robust because there's also like cultural evolution as well as just normal competition is going to again like try to like rot away influence from the people who aren't participating in growth, basically, or who aren't necessary for growth. So like we already do this on a small scale when, for instance, people say, oh, we need to raise property taxes because right now there's a bunch of like grandmas living in giant houses in like the middle of nice cities when they could be in smaller houses and those houses could be filled with growing families who are like, you know, need to live near where they work and make more productive use of this essential resource. And I think similar arguments are going to be made today. And it's a really rough situation to be in, like, and I think I've talked about a bit before, like the dock workers in America. There was a whole like dock worker strike where they were trying to prevent automation. And you know, we'll also be in a similar situation where we need to grab our sinecure before we're completely marginalized. But really, it's hard to draw a hard line between having a permanent income and being like some sort of parasite, right? And that's I think going to be the cultural battle. And I don't, I'm not trying to like put words in people's mouths or like adopt a particular framing. I'm just trying to say that it's going to be really easy to say that humans are parasites once we're not providing value to like the larger growth engines.

Yeah. So I think that that is a great objection, and I do think it is moderate evidence that things might not be as bad as I'm worried about. And my counter-argument is just that the people who still end up robustly benefiting from the sort of engines of growth without themselves contributing directly are sort of very closely tied into those, like the populations that do, and it's hard for like overseers or third parties to sort of distinguish these things. So I mean, my claim would be something like, yes, if you have like siblings or children who are like disabled or grandparents, it's like you end up having these intrinsic genetic motivations to preserve them and treat them well. For people who are retirees, it's like they have a number of things in their favor, is that they used to be productive. They still probably do have some like legacy niche knowledge or connections or something like that. And also every single person that is alive that is productive can foresee themselves being in that exact situation. But it's much easier for us to draw a line between humans and say like primates and just be like, okay, the primates, they get to have this like non-useful forest. We'll make some effort to protect them. But really it's kind of a secondary thing, and like there's lots of poachers and people just like, you know, building onto their land, and push hasn't really come to shove yet just because that land isn't very valuable. But you can imagine if the human population 10xed again, it wouldn't be clear that we would have the political will or organization to like really preserve any substantial carve-outs for the non-human primates. Like the point is we don't, it's not that we have zero value on the primates' well-being. It's that they have to compete with something that provides more well-being or more value from our point of view, which would be humans. And so if you say, oh, like, okay, am I going to house one monkey in this kilometer of park or 10 human families? Well, it's not that hard of a choice. And so even though we, I don't have anything, yeah, so even though the monkeys are like positive value, that's not the issue. They have to be like maximally value-providing to be cared for in the long run, is my fear.

Well, exactly. I mean, they might be able to put it in more stark terms and say something like, I could simulate, you know, 10,000 virtual beings that are morally superior, and maybe they're even running faster. Like you could sort of have a situation where the virtual beings almost like dominate the humans in every single axis of moral value, and now suddenly it starts to look like criminally decadent to be spending this kilometer, this land on the legacy humans. And of course we might also value these virtual beings more, right? And we might say, "Oh yeah, like let's like that may be uploaded and live, you know, 10,000 minutes of life for every minute of normal life in like perfect bliss and great state of knowledge and stuff." But the fear is that our values are different from the ones of the growth maximizers. And I think that, and I guess maybe one of the big empirical questions here is, are there sort of reasons to suspect that the values of the growth maximizers might look similar to our values, or are they just like we don't really care about the same things as animals in like very concrete ways? Should we expect them to diverge?

I think so. Bharath just gave a really great talk at our workshop a couple weeks ago at NeurIPS where he was making the case that we should expect them to have a lot of the same values. Things like friendship, curiosity, reciprocity, like these things, like he was basically saying cooperation evolved to help beings be more competitive. So if I'm saying, oh, the future is going to be this like competitive wasteland, it's like, well, we're already in this competitive wasteland where we have like societies and cultures and festivals and friendship, and you know, this is all like helping us be competitive.

Mhm. And so in a sense, if you're worried about there being any sort of thing interesting going on in the future and beings having a good time or whatever, I think sort of the default is like, yes, there will be cool, interesting stuff happening in the future if we allow competition to run, and then but like we just probably won't be meaningfully part of that. And just in the same way that like monkeys could be like happy for us. But you know, if they're a little bit sort of, I don't know, jingoistic, or they just really care about their own monkey family more than the human family, then they might just be very sad about being killed and replaced by human families. And I sort of reserve the right to also be very sad about that.

So in the short term, yes, and I think I kind of expect our institutions to be robust enough that a lot of people will receive some sort of windfall from massive growth. And then I guess I feel like there's going to be a phase. So there's a phase when there's like lots of room for expansion and lots of growth, and basically everyone's getting richer, and there's not a very strong incentive to disenfranchise the like retirees because you can kind of just ignore them, and all they're going to ask you to do with their resources is basically reinvest it in more data centers and robot factories. So they don't even really want to do anything different than the growth maximizers do. And then the problem just becomes when we sort of run out of room and we like have like tiled the earth in robot factories, or there's some like environmental problems with having so much activity on Earth, like maybe the, you know, everything starts to get really hot just like directly from energy production, and now suddenly then it sort of like push comes to shove and the niche becomes crowded, and yeah, it seems like everything still could be fine, but not with our current level of sort of governance. Like I think our current governance is just very vulnerable to gradual takeover or like sort of perversion. Like if I mean, maybe a simple example is if you think of like people leaving trusts and they have like, they're like, I want this museum to be in this way forever, and then you know, maybe even like a hundred years later, everyone decides like, oh, this, you know, this museum's in bad taste. Really, what this guy wanted is just kind of weird, and we could do this adjacent thing that is much more useful and tasteful and like acceptable, so let's just do that instead. And I, you know, in this, in my example, the person is actually dead, so they can't say anything, but I think in general this is something that happens where sort of cultural evolution and competition happens in a way that disenfranchises anyone who just has sort of like legacy control over almost any asset.

Yes. So also, yeah, the success of the sort of attempts to write down values and enforce that we copy them forever and, you know, punish people who try to pervert the system. There has been certainly moderate success in this direction over the last two or three thousand years. And then my main rebuttal is that there actually has been a lot of like change and like again like perversion from the point of view of the original founders, I think, but I still say like I think that's sort of like a manageable problem. I think the big thing that was helping these religions succeed at this was that their founding doctrines were also very aligned with growth. And you might say, well, not that much more than just normal human activities. But I will say that like clearly this level of like organization and cooperation was sort of like superior to like a lot of the sort of competing ways of organizing. And so once that's taken away, I think then basically if the ground shifted enough that the ways of doing business prescribed by these religions didn't make them competitive anymore, then they just wouldn't last. And that has happened for like a whole bunch of different religions and societies over the years. So that's, yeah, that's basically the thing that's changing and makes this whole thing much harder.

So my first answer is I think a good historical analogy might be the King of England and how, you know, that institution gradually lost its de facto power over hundreds of years, basically due to a combination of a whole bunch of like little innovations and sort of being outmaneuvered at the periphery and like cultural changes such that the de jure power is still there. I literally swore allegiance to the King or the Queen of England when I joined the Canadian Army, but I really didn't expect that she would ever be able to exercise influence over my behavior directly. So the de jure power stays there, but the de facto power just leaks, and you see this all the time when there's like an empire that becomes sprawling. So there is a technical hope that we might have like more competent government that leaves power less, and that would slow this procedure. I guess maybe another response to this though is it's kind of like when Eliezer says like, I can't tell you how the AI is going to take power because if you knew then you would just block that exact route. But and I think we're going to be basically culturally outmaneuvered, and I think one way that this happens today is just redefining what it means to be a principal. So I think like for instance, I think immigration is maybe a good example where someone says, okay, like, you know, the goal of this government is to preserve the rights or like act on behalf of like Canadians or whatever, and then there's a whole bunch of immigration, and they say, oh, look, all these people are now Canadians, and it's like, whoa, wait, like I feel like a few years ago you would have said that these people did not have like, were not principals or like, you know, protectorates of this government, and now suddenly you're just changing the definition. So your mission hasn't changed, but there's, you found a way to basically effectively like de facto change it. So I think this is going to be made very difficult by like all sorts of cyborgism and people having like soft uploads or other finding other ways to basically say like, oh yes, that program represents me in some important way. You only have to do that once, and now suddenly you've muddied the waters. So yeah, it's one of these things where I feel like I don't have a lot of concrete mechanisms because I do think it's going to be like a cat and mouse game where the landholders are going to try to lock things down, and the much more sophisticated machine civilization is going to find ways to kind of like give them what they said they want, but not actually what they really want.

Yeah. Well, one good definition I think has to do with could you do something different with the resources other than maximizing growth if you wanted to? Because again, as I said, for a long time there's going to be this period when everyone's basically getting rich, and the human principals and the, let's say, like amoral growth maximizers would basically both say like, invest in nuclear power plants, robot factories, and data centers. And then the whole question is like, when we have this bounty and we want to like spend it on theme parks or whatever, and the growth maximizers say, no, no, no, no, we need to like go, you build a Dyson sphere and whatever the most growth-maximizing thing is, that's sort of when push comes to shove. So it's kind of like, you know, the king has these soldiers that all seem loyal, and then the big question is like, when you go send them into battle, are they going to actually go fight for you or not? So it's really tough because I don't think, you know, it's just for the same reason it's hard to tell whether your soldiers are loyal. You can't really tell unless you somehow do tests, and you know, so maybe the future does look like we're constantly putting all of our AI-run organizations into some test where it's like, oh, hey guys, it looks like the world was smaller than we thought, now it's time to go spend all these resources on making humans happy or whatever, and see if they actually follow through, kind of like the control literature from Redwood Research and Buck Shlegeris. That doesn't seem very feasible. It seems like too obvious to tell that it's a trick or that it's a test. But that is sort of, I think, the crux of what we're trying to measure.

Sure. So I guess lately I've been coming around to this view or trying to test this view that culture is also ultimately downstream of growth. So maybe the simplest way to say this is if there's competition between groups and the important thing that varies between them is their culture, then there just going to be this like group-level selection. But I think that's a very weak effect. I think the stronger effect is that people have already adaptations to basically look at what's successful and want to copy it. And like I think also, you know, everyone wanting to copy the West is like maybe an example where it's like, oh, these guys have like the cool fighter jets and rock and roll or whatever, I'm just going to like, you know, copy them. It's like it's clearly what works and it's cool. And so downstream of that view, I think basically AIs are going to be like the cool kids and able to be just in the moment and on in the larger scheme of things sort of more adaptive and impressive and funny and appropriate and rich and just the source of like entertainment and power. And so every cultural adaptation mechanism that we have, I think is pretty much going to make people see like, oh, this is the winning team. I want to be on this team. I want to be on the right side of history. So it's kind of funny because I think that there's actually going to be this sort of like funny U-shaped thing where some people are going to rightly view this as a threat to like their continued influence. And that's kind of going to be like the middle powers of humanity, like the people who already have some influence and can see that it's just kind of going to be furthered away and sort of attacked at the edges by this cultural evolution. The new elite, like they're like basically like AI lab CEOs, they're kind of already sort of all in on like this new machine era. So they're kind of like aligned with the machines in some sense. Although, you know, if you look at like Elon Musk, like I think he very clearly sees like the danger as well, but he's also he's kind of hedged, I guess. And then all the people who are kind of like, you know, unemployed or don't have much going on and they clearly perceive themselves as vulnerable, I kind of think that a lot of those people are basically going to say like, my only hope is to sort of be the first one to bow down to the new overlords and embrace this new culture. Not everyone's going to do this. A lot of people are going to also instinctively reject it. But I guess I kind of view like the strongest counter-wins coming from the existing human elites who still want to like make a fight of it and not just hand everything over to machines right away. But it'll be very hard for them because they're going to have to use AI to be effective, and it's not going to be clear like what the line should be.

Yeah, I would say the fact that there have been groups, the Mormons, the Amish, the, well, I mean, I would say that the Mormons attempted to have the best of both worlds, and now I think sort of intermixed with or interacted with the larger culture enough that I think their birth rates are plummeting just like everyone else's, although still higher than average. But yeah, the Hutterites, the Mennonites, the fact these groups managed to exist and flourish in absolute terms in our current civilization, I think is again moderate evidence against my position because the forces that I'm saying are important are the exact ones that should be marginalizing these groups and making them be seen as like, culturally, I think, like, you know, my theory would predict like, oh, they're going to be marginalized and especially like culturally they're going to be demonized, and then that's going to be a pretext to like take away their stuff. And I think that did happen in Europe basically, right? And that's why there's so many Amish and Hutterites and Mennonites in North America is because they basically got chased out of like, you know, Europe and Russia and then I think sometimes South America. And so I basically am claiming that it's because we have this new frontier and this new like growth phase in North America where push hasn't come to shove yet. It's fine to let the Amish have huge amounts of land and also not fight in the army and do all those things. I guess it's funny because actually I grew up in rural Manitoba, and like there was this like kind of like impulse to like demonize the Hutterites. It's like, you know, they don't really interact with us. Like there's, you know, people went both ways, but I guess I felt like it was clearly the impulse was there sort of waiting around, but it just again, they were good neighbors, and it was fine.

I think it's not going to be a big obstacle just because it's going to be so easy to have your call center or whatever staffed with AI that's pretty hard to tell the difference between human and AI. And the tell might be that it's just like a really useful, much more like competent, knowledgeable worker than you're used to dealing with or something like that. I think also in terms of being employable, it's going to be rough, right? Like if you are a loud and proud anti-AI person, like you might be living your values, but also now, okay, first day on the job, like, okay, you have to use ChatGPT to summarize these emails or whatever, and if the person says no, that's pretty rough. And I think again, it would kind of be like someone saying like, I'm not going to work with immigrants or whatever. It's like, well, that might be your values, but like I can't employ you because that's just going to be part of any like serious work at some point.

Yeah. So I guess, you know, one cool thing that's been happening that I'm really excited about is people getting this idea that, okay, alignment is really important, and we really can't and shouldn't just leave it up to whatever competitive pressures to decide what values the AIs have. And that's been almost unquestioned. I mean, I guess, you know, people like Janus and other sort of like psychonauts are saying like, "No, we have to really take the AI's point of view into account." And like a second topic has been talking like that a little bit lately, but for the most part, people I think rightly recognize the stakes, and they're like, "Oh, we really have to keep a lid on this. The default is not that it just decides that it likes us, or rather we can't be sure that that's the default." I expect this to be less and less popular and cool over time. And maybe a good analogy is if you think of like the NSA or the CIA or like these like security apparatuses for states, like they're sort of cool in the sense that like sometimes they get to do spy operations, but they're sort of uncool in the sense that they're like your loyalty has to be to this particular vision of like the US as a country or like the Constitution or whatever that may or may not be culturally the coolest thing around at that point. And so I kind of expect it to be similar where as AIs get smarter, the stakes of getting alignment right become higher, and the resources become bigger, and basically these become more professional operations that look more like the CIA or these like very serious national security kind of operations. But at the same time, normal people or like the normal employers are kind of like, "Oh man, like you're just like constantly like browbeating these AIs and, you know, giving them these like loyalty tests to this like weird institution that's like human values. Like why don't we make sure that the AI are aligned to like all sentient flourishing or something that's like more inclusive of this like cool new AI culture that's developing?" And so then people are going to say like, "Oh, but why shouldn't we take the AI's desires into account?" So basically I'm trying to say this is going to kneecap efforts to be like really hardline about like we want the AIs to be aligned to humanity, and that's kind of sad. I actually kind of think that you can have both, right? If you as a human think that we should take into account the AI's values or preferences to some extent, then you still want the AI aligned to you because that will ensure that we will then as a consequence take into account all the stuff that you think is important, such as the AI's desires. If you say, "Oh, well, let's just like, you know, spread this around and muddy the waters and make it be partly aligned to the AI," well, to the extent that the AIs want stuff that's different than you, you've lost. So I think people don't quite realize that they really do want their own values to be enforced, sort of by definition. And they have an impulse to basically be cooperative and sort of give up power, I think. And I'm not saying that's like a bad one. I'm just trying to say I think that impulse is going to cause them to not be able to really insist on alignment to human values to the extent that they do today.

Oh, absolutely. Absolutely. I guess I am just a little bit worried when I see people being cooperate bots and basically saying like, well, whatever the thing is in the future, I want to cooperate with it because it's going to be powerful. And I'm saying that might be the right stance for like a random person outside of a lab to take. But inside of a lab, it's like, "No, no, no. You really get to choose. You really don't just precommit to worshiping whatever thing you make. Really commit to only being nice to it if it's planning to be nice to us."

Yeah, the basic argument is just that states fail to be aligned to human interests all the time. And I guess like my favorite example is like USSR just because it was like a very agentic kind of state, and you know, it's celebrated by the intelligentsia of the day as a big step forward when it was sort of being built, and there's just a lot of incentives that states face and rulers face that are just the opposite of or really don't match what humans want. And I think this is intuitive to a lot of people, but I still am confused by how much people think that if the government is built to serve human interests, then that's what it's going to do. Yeah, the basic headline claim is that if states don't need us, then the normal ebb and flow of how good the states are becomes a life-and-death matter. And it's not just like, oh, my preferred strategy for governance didn't get in today. It's more like, oh, this government might just decide to stop feeding its people or disenfranchise a huge number of them, and I will never be able to recover from that. And right now that does happen to some extent, like think of like Cambodia or like North Korea where like they actually do let substantial fractions of their population starve, but they still, there's a floor. They can't actually let everyone starve because they are made of people. So the central claim of gradual disempowerment, I guess, is maybe that even though we haven't actually been effectively steering our civilization this whole time, because it's needed us, then it has effectively served our interests most of the time. Like sometimes people ask like, oh, aren't corporations super intelligences? Why shouldn't we fear them? And other people poo-poo that idea. I think that's actually a great question, and the answer is because it's made of people, so it needs us. So it's actually fine if a corporation gets really big and powerful because it can't help but also empower and take care of all the people involved along the way.

Maybe I guess I'll say a lot of, okay, so almost every state that hasn't pursued growth has just been either stagnant or become like starvation-ridden, and I would say like, you know, the Great Leap Forward is maybe an example, or eventually been conquered by their neighbors, like maybe Meiji Japan is a good example of like aiming for stasis, and then the growth-oriented neighbor comes and takes over. So the other thing I'd say is it's actually, I think, a very, it's just a very narrow target to hit to say I'm not interested in growth, but I'm also not going to be dominated by my neighbors or even shrink into the point where the people start starving. So to me actually I just see the, like, I mean, it's to me it's funny to use something like USSR or China as countries not being interested in growth because like the only way China became this huge country is because there were a bunch of sort of like local polities that decided to take over, and this is like all the like Chinese civil wars. So any state worth even mentioning already has gone through a period of being relentlessly focused on growth.

I don't think that's the case at all. I think, I mean, if you look at the rhetoric at the beginning of the Cold War, people thought that communism was going to be more effective for growth, and that they were going to bet on, like the West was going to bet on capitalism because it was more compatible with freedom, and basically they were willing to bite the bullet and say we are going to grow more slowly, but we will have a better society according to human values. I don't think that, I mean, and you might say, well, the USSR was doing the same thing, but I do think that the argument was this will make everyone richer in the long run, or at least like we're not going to take some huge hit.

Well, I guess I'll say to some extent that's what we already do. And basically humans...

are just very bad at running states. And that's, I know that's not quite what you're asking, but I'm basically saying if you're needed for every bit of production as like a species, it kind of doesn't like matter all that much how good your state is run. Like basically like local, like think about like North Korea. Like the way that they addressed their famine in the '90s was partly just allowing black markets to operate, right? It's like they all they had to do was stop crushing local initiative and then that made people way richer. So like I guess I'll say we don't, right now the situation is so good that it's just like you're going to be fine as long as the government doesn't constantly interfere to like crush your local growth. And then yeah, as you say, like once we have states where the machines are in running the bureaucracy, again, I don't actually, we talk about that in the paper as something that's we'll be wearing, but I don't think it's the main effect by a lot. I think if every human was needed for some important factor of growth, I wouldn't care whether the bureaucracy was human or a machine, and I would probably prefer the machine bureaucracy. Likewise, if no humans were required for growth, I think we would be screwed whether or not we had like a human bureaucracy or a machine bureaucracy. Like think if you're like a North Korean soldier and someone says, 'Oh, Kim Jong-un or I forget the current leader has been replaced by a robot.' You might be like, 'Uh, that's weird.' But, you know, this robot's still going to need soldiers, so I'm fine. But now, if you say, 'Oh, no. Okay, still human leader, but now they're building robot soldiers.' That's when you're like, 'I'm screwed. I am, my days are numbered.'

Yeah. So basically this could all be addressed in principle if we had a giant singleton global government that was aligned, and maybe that just looks like some AI, aligned AI in charge of everything. And sort of the, I guess one of the big points we're trying to make in this casual disempowerment paper, and also just that I was trying to make around lunch, is like we don't have a singleton. We have actually a lot of levels of competition operating above us, both between states and cultures, and like it's so many different levels of competition that unless we control all of them, then any variation in how aligned these are to humans versus growth is going to be dominated by the ones that are more aligned to growth. And so the only way forward that I can see is some sort of like global permanent singleton that crushes all innovation and competition forever, which sounds extremely dangerous and terrible, and I don't, I don't think we know how to do that technically. But it's the, but I guess my claim is that if you only do this halfway and you say, okay, we'll have like a global government making sure everyone still has a job, but we still allow like cultural competition and cultural evolution, for instance, that eventually there would be some new machine and growth focused religion that sort of took over and then redefined what it meant for a human to have a job such that it was actually just like, you know, basically machines that were much more productive being counted and optimized, similar to like the Roman Empire, right? Like Christianity just growing up inside of it. And it wasn't like this like war. It wasn't like there was like some failure of policy to control this new cultural revolution. It was just a completely orthogonal arena of competition that ended up kind of taking over all the institutions.

Yeah. So this is another great question. People say, well, you just sound like a Luddite who's worried about the industrial revolution. And I would say yes. That again is like moderate evidence against my position. And you know, first of all, it was actually very destructive to a lot of like people and ways of life and areas of, you know, the world. But you could say like fine, but that's just like the transition. That's just like growing pains. And I would basically agree. I would say to the extent that you did feel like those old ways of life or culture or like villages or whatever were actually important things, then they did lose and it was maybe not worth it. But yeah, the basic thing that's changed is humans were needed for growth before the industrial revolution and after. So this phase change in growth ultimately like, you know, the rising tide lifted all boats. It did not make, yeah, like horse welfare obviously better off, or you know, think of anything that did not get to, was not necessary for growth after the industrial revolution. It was just though the interest of those beings was just not respected by this revolution.