David Duvenaud | Exploring the Cruxes and Possibilities of Post AGI Futures

Well, I would say that's probably a big part of the success of the West is this notion of like liberalism. Like it's so unintuitive how positive it is when a bunch of people just leave each other alone except to do business and trade and build like science together and institutions and civilization neutral rules. And so like in a sense that's like our most precious thing that we have in our civilization. But the like hypothesis that me and some other people are just starting to think about is like once humans aren't providing value then it's not so clear that there's any advantage to this liberalist framing of society where you like really want to not try to argue about what is the true religion or the right way to live or whatever. You want to just set some very basic parameters and like then give people their own resources to manage. And so when we're all living on UBI, if anyone is ever able to say like, "Hey, my way of life is more deserving of UBI than yours," then that is like a very zero-sum competition. We all just become full-time advocates for our like UBI growing fashion and trying to like reframe the narrative that our types of people deserve more resources than those guys. We'll probably have to try to reframe this a few different ways, but that's the basic fear.

Sure. Yeah. So I'm David Duvenaud. I'm an associate professor at the University of Toronto in computer science and statistics. I recently spent a year and a half doing an extended sabbatical at Anthropic where I was running a small team that was doing alignment evals which was basically trying to see if the models could sabotage our decision-making, like could they trick us into thinking that they weren't capable of something when they weren't or steer us secretly towards a bad decision something like that. I've been sort of following the AI, let's say like AI safety/AI doom case and arguments and space for a long time probably since about like 2005 or so. And yeah, but you know only recently was there all that much practical work to do. And then I guess what led me into this recent line of work that's more conceptual and more about the future is just going around and systematically talking to all the smart people I know who have been thinking about this and sort of saying, so what is the plan for what if we even succeed and solve all these tough technical alignment problems? It seems like we're going to be in a very vulnerable position forever. Basically, humans not really being needed for anything that's like could go wrong for a whole bunch of reasons and just not really getting any satisfactory answer and kind of being a little freaked out about that. So trying to do something productive about that and so I organized this workshop. I met Jan Kovačević and some other collaborators who we co-authored a paper called gradual disempowerment which tried to lay out the argument that well we probably can't rely on just like comparative advantage to save us or the government and in general we're probably used to a world that's a little like friendlier by default than it's going to be just because right now everyone needs us. It's like it's kind of like being the like hot girl. It's like everyone's just like happy you're there, right? And that's kind of like with humans and trying to say like okay but in the future it's going to be really tough but more like old people that it's like okay cool you might be fine as a person but we don't actually like need to involve you and in fact it's going to like slow things down to involve you anyway. We're getting a little into the object level but that's what I've been thinking about lately.

You know, amazing book. Like he, you know, I think we should all just give him another round of applause for thinking these things through like so early. And just making the basic the basic big blob of compute arguments, saying like it doesn't necessarily matter what I mean, we're going to figure out the algorithms. What matters is as a first order is how much compute there is. And in the future, there's going to be more digital compute than biological. Like we're going to be able to do the same things on digital computers as on biological computers. Just just very simple arguments. He was definitely overconfident in the way he framed things, but like he just was a total visionary. And then I guess online I found Overcoming Bias. This was like you know Robin Hanson, Eliezer, Nick Bostrom, all these people. It's kind of like the post-extropian lists. I found Shane Legg back when he was a PhD student on Live Journal. Yeah. So just like there was this like very embryonic community and a lot of it was like really about basic stuff like cyberism and just like could the future be different? It hadn't really focused entirely on like AI risk at that point I guess.

Yeah. Well, I guess I'll say so I mentioned me systematically going around to every smart person I know and sort of saying what's the plan and not really getting a satisfactory answer. And I've been sort of going around telling everyone, hey guys, like no one has a plan. You probably think that they do, but they don't. But people shouldn't take my word for it, right? And maybe I missed somebody or whatever. So part of the rationale from my point of view was I'll just do the same thing but in like a big room where everyone can like see. We invited all the plausible people who would like have something concrete to say about this and ask them concretely what you know the plan should be. And we learned a lot of fun cool stuff but nothing that was like oh yeah and this is how it's like here's our plan. So, and the workshop was concretely called post AGI civilizational equilibria colon are there any good ones? And the idea was to try to bait people into saying like obviously there's a good one look at mine and just to really make it clear that we were trying to ask the like sort of big important scary question of like what is life going to look like? So, as for updates, I mean like disappointingly I guess there what you know the best outcome would have been one of our speakers was like hey guys you've been thinking about this all along. Here's how we have this like thing where we can understand roughly what's going to happen and why.

Carl Shulman had an amazing keynote was basically can goodness compete sort of asking the even harder question. It's like how can you ever compete against the beings that are just optimized for being like growth and competition and sort of using resources and winning as fast as possible, right? Like if there's anything that you care about that doesn't support those goals, it's not really clear to how you're going to last in the long run, right? So he's kind of asking the like really hard version of the question. And then Richard Ngo had this nice keynote. It was flourishing in a very unequal world and basically making the case that a lot of the ways people talk about post AGI sometimes asks as if there's going to be these peers in the future like there's like the humans and then the AIs and maybe we'll like trade and we'll come to some sorts of agreements and stuff. And he was saying, well, probably, he didn't put too fine of a point on it, but he was like, there's going to be just like today very unequal ants versus humans and apes. Like, it's just doesn't really make sense to like sign a contract with an ape or whatever, right? Like on either side. And I think he was sort of trying to make the case that we should be thinking about if there is good relationships in the future, it'll look more like parent child or something like that. And the first step is just to like acknowledge differences. Yeah, it was interesting the asymmetrical relations or something like that he called like but official asymmetrical relationships or something like that.

Yeah. No I agree. I thought both of those keynotes were kind of like updates for me I think to some extent or like I think I updated positively towards the asymmetrical relationships as well or like that that's something that could potentially be one solution. Yeah. And what do you think about like well actually we could maybe dive into that because another thing that relates to this is another thing that Richard Ngo wrote and that I know you come down it's it's like a bit of a Less Wrong discussion basically where you talk about well Richard presents this case that he's been thinking more about I guess it relates to these asymmetrical relationships as well about how he's been thinking more about sort of right-wing political concepts and ideas that could actually be like useful and beneficial in a in a post AGI world. And I think you had some thoughts on that in relation to like why liberalism for example is maybe not a great solution in a post AGI world. Do you want to maybe just expand a bit on that?

Yeah. So the basic idea is that right now we have a bunch of institutions that are pretty great and they basically support us even if we can't directly contribute very well like welfare and pensions. It's actually been a big battle to have things like freedom of religion, freedom of speech, limited government. Like I think the default in human history is you get this sort of like very totalitarian state that just has opinions about everything you do. Everyone has to constantly run around and justify their existence from whatever point of view of like the new morality is that's in charge. They, you know, the king's instinct isn't generally to be like live and let live and like oh you guys live how you want and we'll we'll still do business or something. And that and like learning how to do that was such a step forward and that kind of is in some sense like this success. I would say that's probably a big part of the success of the west is this notion of like liberalism of like hey guys positive like it's so unintuitive how positive sum it is when a bunch of people just leave each other alone except to do business and trade and build like science together and institutions and civilization neutral rules. And so like in a sense that's like our most precious thing that we have in our civilization. But basically the fear the like hypothesis that me and some other people are just starting to think about is like well maybe that like once humans aren't providing value then it's not so clear that it's worth leaving us alone or that there's any advantage to this liberalist framing of society where you like really want to not try to argue about what is the true religion or the right way to live or whatever. You want to just set some very basic parameters and like then give people their own resources to manage. And so concretely, when we're all living on UBI, if anyone is ever able to say like, "Hey, my way of life is more deserving of UBI than yours," then that is like a very zero-sum competition. The like bad vision of the future maybe is that we all just become full-time activists or advocates for our like UBI growing faction and trying to like reframe the narrative that our types of people deserve more resources than those guys. Those guys are just like the locusts who want to have, you know, control everything for no reason other than power. We're the ones who have like true values and we really worth preserving the like flame of humanity or whatever. And so I'm I'm not sure like we'll probably have to try to reframe this a few different ways, but that's the basic fear.

Sure. Well, that was just a thought experiment where I was trying to think like, okay, what does life under UBI look like? And so I'm imagining right like right now the government has various like requirements to get welfare or whatever. And people of course like lie about whether they are eligible and stuff like that. But even if ignoring like the lying part, you can imagine someone saying like, "Oh wow, it turns out that like every human is entitled to like some tiny fraction of like global GDP or something. Maybe we've set up like a pretty good UBI." Then the like next day I think a bunch of entrepreneurial kind of people are going to say like let me just figure out what is like the minimum viable UBI recipient right the minimum viable human according to this government definition and maybe that involves something ghoulash like make a factory full of like fetuses that you like freeze or like somehow put in suspended animation in a way that like still meets the government's bar and now you're just like sort of welfare farming and of course the government's going to patch that right they're going to say okay your fetus has to be at least like this big and then it's like okay well then they make you know, if there's going to be this whole like back and forth race sort of forever of the government patching the UBI requirements and people at least like some people optimizing those. And of course, the bad thing is that in the long run, the people who optimize the hardest against whatever the like welfare rules are, are the ones that sort of inherit the future, at least to the extent that the humans in this scenario still matter like in the big picture.

Well it's kind of funny because like equilibrium also sounds kind of bad in a sense right it's like well that's like permanent stasis and like what about growth and adaptation and like I've made this case to a few people and they've I think rightly said well what do you want right like we already have a world where the world belongs to the people who reproduce the most in a sense like in the long run or maybe mathematically reproduce a whole bunch of levels of competition but it sort of can't not be that way unless you just like freeze everything and then then that's also like a very destructive thing that's maybe even worse. So maybe the way to think about it is we have to play some game and maybe in the future we'll have a strong enough state that we'll sort of be forced to and have the opportunity to set new rules and maybe then if we have the imagination we can think of what would be an actual utility function that it would be sort of like fun and worthwhile to fight over and optimize for the foreseeable future. Maybe being good at chess or dancing should be like decide who gets to have like the most kids or whatever, right? Like I have barely begun to think about this, but you can imagine that like there might be positive seeming versions of this and that's something that I think is like a big sort of like project that people should pick up.

That's horrible, right? Like that's like the very first trap is like what you never ever want to do is like pay people according to how much they suffer, right? Cuz that's the recipe for like unbounded suffering and then you have your like child torture farms or whatever to get the most welfare. So like absolutely no you should pay people for being happy.

Well exactly. So I I guess I would say like this is actually like maybe this is one of the things where we don't have to worry about it because it's such a good problem to have, right? It's like if we could perfectly design our future sort of fitness function, but I think it's actually a really good exercise to do and I think we should have more fleshed out ideas of like okay the obvious pitfalls of like yeah paying people to suffer or have them as babies or whatever try to like foresee at least how this conversation is going to go, right? Cuz we can probably look two or three steps into this like back and forth of like people Goodharting and then the government changes the rules and then they Goodhart the new rules and stuff like that.

Maybe. I mean, the thing is that I think liberalism will still work for like machines or whoever is productive. It's it's kind of like liberalism works for people who if you just leave them alone, they're going to make everyone richer through positive sum trade. You just like want to leave those people alone a little bit as long as there's like sort of like rule of law and a bunch of other things going on. And that they and they can't reproduce so fast that they just like tile the solar system before you get a chance to go out there yourself. So yeah. So it actually might not work for machines just because they might be able to reproduce so fast that they'll kind of just bring us all back to Malthusian because conditions really fast. I mean I guess I'll say it's very early days like I I think there's only a few people who have started thinking about this. Um I also will say that you know there's the I think one reason people haven't thought about this is they're like well if we're already like it might be a kind of narrow target to hit that machines are so much more powerful than us that they haven't just directly disempowered us or taken all our stuff or you know killed all humans or whatever but we still are around. It's kind of like a slow takeoff scenario. But I do think that that's so I guess I was trying to say I am also worried about the like acute loss of control risks and but I also now I'm trying to say like even if we solve those it's not clear what the long-term like good outcome looks like. So anyway, I I'm basically trying to say I don't have a good answer but also we just started thinking seriously about this recently. But I guess I'll say if we can redesign our civilization then that's already like probably a pretty good end. Like it's just seems so hard for us to achieve the necessary coordination to do such a thing. I mean it's I mean it's kind of like communism, right? It's like there's so many times everyone says why don't we all just blank, right? And why don't we all just like not have war? Why don't we all just like you know feed all the poor people and then it turns out like oh you to do that you have to create this like very powerful state and then everyone just fights over that power and it doesn't actually work out how you thought it would. I guess I would try to say almost anything we try to do globally today, I'd say like step one is massively upgrade our ability to coordinate and like forecast the future before we try anything ambitious to like determine what the new fitness function is for everybody.

Uh I definitely do. And so a lot of people reasonably say like oh if you had tried to plan the future like 200 years ago or 100 years ago or whenever it probably would have gone terribly. You would have not done a bad job of predicting. You would have had some like way too restrictive control that would have destroyed a lot of wealth and freedom and like probably wouldn't have solved the problem anyways. And I think that that's a very on point criticism. Um my reply is basically I think there's no way out but through. So maybe one way to say is there's a big easy win which is that if we can get better at forecasting and providing and making common knowledge than we have then I think that is almost like a strict win. And so in particular, I'm really excited about like the usual like prediction markets, maybe using AI aided forecasts, right? Like you know, human super forecasters is actually like a really amazing invention, but I think it's just sort of scratching the surface. Better coordination mechanisms like more like Kickstarter style things. So that that's sort of like the easy wins and I think it'll just be in general, it's a matter of degree how well you can forecast and coordinate. And while humans are in like have power and our our civilization is incentivized to keep us happy and healthy, the better we can plan and coordinate like that's just a huge win. And I think we can and should do that. But as for like like this argument like no, let's stick with the tried and true method of like just set up a state but don't really give it too clear of an objective or like you know utility function to optimize. Just tell it to like give people freedom, let them innovate. I'm like, "Yes, that has obviously worked really well. I really don't want to throw that away if we don't have to." But I guess I would say like at this point we should be really seriously entertaining the idea that that predictably will fail. Again, I'm like super scared because I feel like I spend all my time like most of the time feeling like everyone just wants to introduce some like socialist thing that's actually going to be disaster in practice. What I'm proposing does sound scarily similar to that. So I don't want to take any steps in that direction till we thought it through more. At this stage, I'm just sort of calling for more thought.

Well, I I'm pretty active on Twitter and like my papers I try to publicize them. Yeah, but I mean I I guess like I do have a side project on using LLMs trained on historical data to forecast like the past like from the like from their point of view, it's the future. So we can try to validate how well we can get LLM to do some of the like long-term forecasting that I think that they would be able to help us with. But I mean overall this is like a project that's much larger than me and like other people are barking up the same tree. Um, I worked a little bit with the societal impact team at Anthropic and it's funny because I think they have actually fairly similar view, but it's it's pretty rough because like they're doing a lot of really good work on cataloging what's happening and saying like, oh, this is the people who are using AI for their jobs and these are the ones who are being replaced versus augmented and stuff, but it's very like Cassandra sort of like they're doomed to just watch it happen and even though Anthropic and the other companies have like robust policy teams like and all this stuff is beyond the scope of any one company. So it's like it's it's it's pretty rough position for them to be in because their job is to watch something that this company's scope is probably too small to address.

So the inevitable part pretty much save some sort of like global don't build the AGI Amish totalitarian thing and soon I guess I'll say like I've thought about this to some extent but I haven't done any like detailed like forecasting like the AI 2027 folks. I mean I think those guys are also more like AI 2028 or 2029 now. But yeah let's say like I don't really expect my kids to have normal jobs maybe concretely and my kids are like my oldest is like six. So like somewhere in the next like 15 years, it's like stuff changes enough that it's like all of our current scripts about how do you provide value in the world and like navigate as a person mostly don't make that much sense anymore.

I mean the first one is just a very big one of like the brain is a biological computer and it's almost certainly not the most efficient thing. And even if we build machines that are not quite like energy efficient, the fact that you can copy digital memory is just like such a huge like overpowered ability, it's always just going to be worth investing in like the very best machine like lawyer or engineer or mind and then making a million copies of it than it is going to be to train a million human engineers. It's just Yeah. So I think it's not going to be very long before there's no alpha in humans basically. And it's always just like, oh man, I just got to like get my one amazing AGI mind to work better. So that's that to me that's like the big thing that changes. It's not like the day that everyone actually loses their jobs or it's like actually completely obsolete. Everything changes. It's the day that it's like common knowledge. It's like oh like like school is kind of like pointless, right? Like it's one of these things like gradually people kind of think it like it's true in some cases some of the time and but I do think there's going to be sort of like a not preference cascade but like the thing when something becomes common knowledge and that's going to be the like big like depressing day.

Yeah. Well, here's maybe I can bring up like a moral crux that I think I've identified with some other people, particularly Robin Hanson, which is like if you are worried about the future containing all sorts of like basically are you worried about yourself particularly surviving and flourishing in the future or just like life doing okay and surviving and flourishing? Because if you're okay with the second thing and you're not too worried about like your particular bloodline or even like species surviving, then like you kind of don't have to worry about anything, right? It's like probably whatever ends up taking over in the future is going to be having fun like high-fiving each other, taking over the solar system and building all sorts of stuff. So, you know, just like us, it's like, oh man, think of all the like horrible wars and extinctions and all the stuff that led to humans, but now we're like, "Oh, awesome. We have this amazing world and we can play in, right?" And it's like, you know, whoever's in the future is probably going to be pretty happy that they're there, more or less. So, in that sense, it's like, you know, don't worry about it. Um, and then you're kind of, yeah, it's like 95% chance everything's going to be basically fine. And then but if you're like I want my particular kids to have like rich and fulfilling lives in the same way that I have had and humans not to go extinct or whatever like uh then then you have a much harder like narrower um target to hit and then it's like and I think also again it's like what are you happy with? So I I think people tend to be a little annoyed by me having a very sort of high standard for what a good world counts as. And like something about as good life as I've had for like almost everybody like over you know not everybody all the time obviously but like you know a similar fraction of people as today probably like increasing and in particular like my family getting to like basically not be somehow completely marginalized then then it's like well that just seems like that was already always going to be hard because in the very long run like even biological evolution was going to be cause or even just cultural evolution was going to be causing the same kind of obsolescence and marginalization that you're worried about which I totally agree with. So, I guess I'm like, "No, no, no." The same thing happened that happened to all of our ancestors where they got out competed by like more adaptive descendants of theirs. That to me is like kind of crummy. And I agree. I really I agree it's better than nothing. But if you're okay with that, then again, it's kind of like whatever. You're probably going to be fine no matter what.

Sure. So I think the main one is that we can't all just relax and assume that UBI will be fine because basically we will be the like bitter clingers that are like probably like culturally demonized for wanting our ridiculously expensive sort of retirements and that there's basically like all the different parts of our civilization are going to probably work to marginalize us because that's sort of what serves growth. I mean so the reason the sort of like reason that paper would need to be written from my point of view was just talking to so many people who are like but surely this institution that has always served us well will continue to serve us well and like the government is there to serve humans or like even sometimes people say like but I don't understand how who are these machines trading with if the humans all go broke then won't all the machines go broke because then there's like no more economy and it's like no like this is like monkeys saying like who are the humans going to trade bananas with like how are they going to survive with their own economy that doesn't involve any like trading with monkeys it's like they don't need us. So I the point is that like I don't know I'm already afraid of the government. I already think like governments are these things that we sort of accidentally make and they end up like doing horrible things to us on a regular basis. But I think a lot of people have this intuition. It's like why would like sometimes the government goes a little bit wrong but ultimately it's serves us on the long run. And I think both things are kind of true because the governments that really don't serve their people collapse or like don't grow very fast and get out competed like USSR. Like the the point is that like our intuitions about how well they're going to treat us have been formed in this world where the governments like needed us and then it's like well even if they hated us it's like some governments really kind of act as if they hate their people but it's like but they still keep them alive and they still can't imprison everybody or liquidate everybody and that's not going to be the case anymore.

Sort of. I mean uploading is kind of like the old person moving from a big house into a small apartment. It's like it reduces the tension because they're not like wasting all these resources. And I I I expect kind of like one of my like visions of the future is like I'm like, "Okay, the world is covered in robot factories, but I have my like little like 1-acre plot of land where I'm just going to like live with my extended family or friends or whatever. Um or a little human city or like and we don't really need that much resources. Just leave us alone. You guys are also rich with your like amazing robot civilization." But then some of the robots saying or like machine mind saying like you guys are using so many resources to house like only a few humans with those if you were uploaded or if we just replaced your house with like another data center we could house like millions of minds including your own also a bunch of like much more deserving and sort of like morally superior versions of yourselves or humanity or whatever. So it's like sort of ridiculous it's like criminally decadent of you to use these resources that aren't on the like most morally deserving beings, right? It would be like paving over an entire country to make like a wildlife preserve for like a single gorilla or something. It's like no like look at all the suffering orphans and refugees you've created because of this like wasteful thing you've done. So uploading is like I think something that is one of the ways I think we'll be marginalized in the sense that it's just like we'll be forced to use less resources either by just like market forces or it being seen as this like moral outrage that we're wasting so much. And then that helps a lot. And then I guess depending on your point of view though, it is like just one more step to being completely disempowered. Like I really love that there's like no like landlord of like physics that can like kick me out, right? But as soon as you're like hosted on a data center, it's like oh man, like someone can cut off my power or just like you know schedule me for later but never actually get around to like running me again or something like that.

Well, so the basically a think about it is that we've been designing our governments and institutions on easy mode. Um we will have to learn this art of designing even more robustly aligned institutions which I think is going to be a very hard art to do. Um, I mean, I really hate it when people say like, "Let's just ask the AGI how to solve alignment or what to do." But I guess like I think in general there's like if we as a civilization become more competent, depending on the timing of things, it might sort of be okay because we can outpace the worsening incentives by upgrading our institutions as we go to make sure that they continue to respect things like human property rights and avoid all the other like fast and slow kind of loss of control. I mean another way of thinking about it is like everyone I think has been

Okay, we all have this instinct, I think, as like sort of kids, like the naive thing is like when you rule, you care about everything. And then we've all sort of had to have this counterintuitive lesson beaten into us of like, no, just live and let live, don't worry about where things are going. Like make pointed, like inflammatory, like predictions about like, oh, this group is having too many kids or whatever, like or this religion is growing or whatever. Like people rightly are like, hey, you're really playing with fire if you try to like predict which group is going to be powerful in the future and like coordinate now to stop them, because then you, that never really ends and you end up just like fighting over their future in detail all the time. We have this like much more like live and let live thing, which is very counterintuitive, but in a sense I think we've sort of maybe taken it too far and are just like, no one should steer civilization, no one should like try to be agentic at the top level again. That seems like super, super scary because if we ever do manage to make an agentic world government, we probably can't ever come back from that. But my basic aim is like probably someone or something, probably there is going to be some sort of like global agency and we sort of need to like practice and upgrade ourselves and sort of grow up to be able to handle that kind of responsibility and planning. And I'm not sure we can, and Russia is going to go very well, and it like, it's again very scary. But the basic vision is learn how to govern so well that we can make governments that will treat their people well even when they don't need them.

First, I want to like attack my own position, which is to say like, I keep talking about this world government, but yeah, maybe this is one of those like things where the naive analysis always thinks that there's going to be this like world agentic thing, but actually in reality, life finds a way and it's more complicated, right? If you look at our like planet, it's not like there's one being that has filled all the niches and it's like also we've had like empires rise and fall. But you know, when one political party becomes really big, often it like splits in two or there's like a new religion that is founded that then like, you know, somehow takes over. And so maybe the equilibrium isn't that there's like a global agentic thing. It's that empires rise and fall and there's some more like chaotic like attractor sort of thing. Um, in which case I think then it's kind of even worse though for the people who aren't productive in that world because it's like, well, then no one's going to be able to like fund the like, you know, foundation that keeps them around forever if empires rise and fall, right? Like that's one thing that happens when the empire falls is all your like foundations get looted or whatever. So, but anyway, but I guess I'll say like, I again, like liberalism is great and I think it's sort of like still underrated in a sense because people don't understand how amazingly positive some of it is today. But I just don't understand how we're supposed to stick around for all that long when there is like competition at the top level because the politics or like the governments that focus on machine economy and machine growth will just dominate against the ones that spend significant resources on like paying the human pensions. I mean, the hope is that like growth is so... there's one way that forward is that like growth is so fast that our demands are so small that it doesn't even matter. It's like, oh yeah, just give the humans whatever they want. It's still a rounding error. Then that's kind of a feasible future where everything's okay. But then it's also like such a crazy future where everything is changing so fast that I'm also kind of like, I'm not like, we're like a mouse in the bed with an elephant or whatever, right? It's like still scary from that point of view.

I don't know. To me, I actually really don't like plans that rely on the AGI being smart enough to solve global coordination problems. And this is also actually a huge crux, for instance, between me and like Ryan Greenblat and I think other people that say sort of AGI will solve it, is from their point of view it's like, well, maybe our institutions aren't very aligned with us, but if you could at least give everybody their own AGI that was aligned with them, they would be able to come up like cut some global deal where everyone agrees to like divvy up the resources and the growth and it's sort of fine from there on. And that's plausible and I don't hate it as like a scenario. It's just that it's like, well, what if we have a slow takeoff? What if we just have a bunch of APIs and each human is like, hey, I want you to help me, you know, get a good bid on this market or get a date or like play the normal sort of like messy games that are somewhat zero sum. And none of the AGIs has the ability to stop and say like, oh, can't you see that we all like have this horrible like dating culture or market or whatever that's leading to bad equilibria? Let's stop and redesign everything. That's possible. We've done it before. But it's also just like hard. It might be harder when most of the market participants are even more sophisticated. So like concretely, like let's imagine that I'm like a pre-tenure academic and my AGI is like trying to help me and I'm like, "Hey, I want you to help me get tenure." And it's like, okay, I'm going to help you like spam, like maybe I'll help you do good research, but also just like spam the citation metrics or whatever, right? Like hack the incentives. And this just like makes the global system work because everyone's like now their AIs are all spamming and everyone has like a thousand papers a day. And the question is, can some academics get together and say, stop, stop, I'm going to risk my tenure, stop, and like find some new global way of evaluating contributions or something like that? Or like a dating market. It's like, help me get a date. It's like, okay, I'll spam all the like dating sites. It's like, no, no, no, the real solution is to like step back and like fix our entire culture or something, right? It's like, well, maybe we could do that, but also it might be this sort of like very tough global coordination problem that no one has ever empowered or incentivized enough to actually fix.

Not really, but I just, I have a like half-written sort of post about like what is stable at the top. Like there's the case that we should expect, you know, Earth to be dominated by some global thing because once you have global power, you can use it to like purge all dissent and, you know, control everything to make sure that you stay in power. On the other hand, when life finds a way, when modern humans get cancer and die all the time or like infections, like things where the sort of like little guy realizes he can just start growing and, you know, you can just do things then suddenly, or like there's a new religion that starts or, you know, that changes everyone's minds and they decide that we don't want to be the global hegemon anymore or whatever. So it's actually not clear to me what's actually stable at the top level. And at the very top level, of course, like there's probably other alien civilizations out there that we'll eventually meet. So there has to be, if we think really, really long term, there already is some sort of like multipolarity there.