My name is Sheila McIlraith. I'm a professor in the Department of Computer Science, a Canada CIFAR AI Chair at the Vector Institute just upstairs, and also an associate director at the Schwartz Reisman Institute for Technology and Society. It's my tremendous pleasure to introduce my colleague for this afternoon's keynote address.
David Duvenaud is an associate professor in the Department of Computer Science at the University of Toronto. He's a founding member of the Vector Institute, where he's also a Canada CIFAR AI Chair, and he holds the Schwartz Reisman Chair in Technology and Society. David's also really active in the Toronto and Canadian ecosystem. He's an adviser at Cohere, where Sarah Hooker, of course, who spoke this morning, is working. He's on the board of directors of the AI Safety Institute, and he also serves on the ISAD Safe and Secure AI advisory group.
David's widely known for his contributions to AI safety, as well as making significant and influential contributions to the field of probabilistic deep learning, and he's received a number of awards and recognitions. You can look at his website; I'm sure they're all listed there. But he's a key contributor in the area of AI and deep learning, and also AI safety.
So David's research of late has focused on some of the biggest and important questions around AI governance: how we evaluate potentially dangerous AI capabilities, and a myriad of other questions. In addition to his academic role, he's also conducted extensive research with the alignment team at Anthropic, where he was on sabbatical for about a year and a half, ensuring that his insights are informed by the latest technological advances in model development.
David's talk for today is entitled "Gradual Disempowerment: Systematic Existential Risk from Incremental AI Development." He's going to walk us through how even small incremental advances in AI can, over time, erode human influence over essential systems, from our economies and cultural institutions to the very way we govern ourselves.
David told me he welcomes interruptions within reason. So, if you have a burning question or a deeply visceral reaction to something that David says, which is entirely possible, he welcomes — maybe you can just signal, and we'll get one of the people with a microphone to give you a chance to voice your opinion. But otherwise, we will have a question and answer session at the end. So, please prepare your questions for that. And without further ado, let me welcome David Duvenaud.
Thank you very much, Sheila. Okay, how do I sound? Okay, great. So, I am super pumped to be here because maybe for the first time I have something actually interdisciplinary to talk about, and I really feel like we need help from a whole bunch of y'all who know more about these topics than I do. So I'm going to quickly tell you the overall pitch of what I'm trying to say, and then give some more detail.
So the main claim that I'm going to try to support today is that even if we don't have to worry about power-seeking AI going rogue, if we manage to build ones that are more or less aligned to somebody, humans are still likely to lose power. I think this is not obvious; it's controversial. You guys should challenge me on this. But the main supporting argument is that there are already power-seeking systems everywhere, like states and even just like groups of people, cultural phenomena that will be somehow, let's say, like more unshackled or powerful as human influence wanes, and those things already are less aligned than they appear.
Right now people say like, "Surely the government's going to take care of us," or "creative destruction is something we can survive." And I'd say it has been the case that actually that has been a good bet until now, because these systems, like states and economies, have needed humans for their labor. But that changing is going to make everything a lot more fragile, and like surprisingly fragile.
So the rough story that I'm expecting to see is humans become a drag on growth, and you know, just like Take Our Kids to Work Day is just like a sort of one day a year for fun thing involving humans, will be like a recipe for everything going slower and being more shambolic. Anytime there's like competition or people trying to actually get things done, they're going to say like, "Let the AI handle it. That's the responsible thing." So we will not be involved in an important way in key functions. We'll be kind of more like a monarchy or like some maybe figurehead positions.
And I don't really have strong opinions about how fast this is all going to happen, but it seems like the equilibrium, even if we can all see that this is happening, is that we basically have no influence — just like, you know, monkeys have no influence, or basically unemployable people have like very little influence over our civilization, even when we sort of try to involve them. Okay. So, maybe it's too quick. I think you guys should probably disagree with some of these things already. Maybe I'll go into like one more level of detail first, and then I'd like to have some pushback here.
Right. And the thing that I'm actually afraid of is us actually somehow being starved of resources. And I don't really, again, have strong opinions about how this would look. That's all very speculative, but it's just like, you know, no one hates horses, but we just have way less horses around than we did when we needed them. There's just a certain level of vulnerability that I think is like very alarming.
Okay, so the reason I'm here — I just wanted to solve technical problems, right? I got tenure. I was like, I've got my whole technical agenda. I'm just going to solve math problems for fun for the rest of my life. It's awesome. Got worried about AI safety, started working at Anthropic and visiting all the labs. But I kept asking everyone a question like, "Wait, wait. So I don't understand. So the plan is we build this aligned AGI. It's better than us at everything. It does all the jobs. Like, what position — what do we do then?" Right?
And I went to like every smart person I know who's like thought about the big picture, and they said, "Yes, the plan is to automate all labor. We think that's possible." So I'm like, "Yeah, what if we succeed? We're not needed, but we kind of — like no one will have an interest in protecting us." Like everyone might agree, like, "Yeah, humans are the founders, we need to take care of them," but it might be just like hard in practice to advocate for them.
And the best answers I got were like, "Oh, I'll just be like accepting suggestions all day." Haha. Like actual engineers. "I'll be on a much-deserved vacation," because everyone there is like, you know, working themselves to the bone. Or, you know, "the future's very uncertain, like you're getting wound up about this particular scenario, but we have no idea what things are going to look like. We'll have aligned AI to help us." I think this is one of the more serious responses, and basically saying like, yes, this is like actually a really, really hard problem, but we're going to have really, really amazing guys helping us with it.
Another one is like, "Well, we'll see this coming. If things start to go off the rails, we will not sleepwalk into a disaster," which I think is also a good rebuttal. And I'm going to claim that there actually could be a long period when we all can kind of see what's happening and can't coordinate to stop it. And then there were actually a bunch of people who basically agreed, like, "Yes, this is a huge problem. We don't know what to do about this, but we have an even bigger problem, which is that we need to solve AI occlusion, or you know, some sort of like runaway power gathering by the AIs themselves." So I kind of felt like, oh man, it seems like almost literally no one is working on this larger problem of like, what do we do after we succeed.
Okay, there we go. So I'm just going to paint a rough vignette of the kind of thing I'm expecting to see. I'm not wed to the details, and you can justifiably see some of this as a bit impossible. So I expect all of us to basically become better at our jobs as we get augmented by AI. And for some of you, this might already be the case, that you can just get more stuff done. And I think there's going to be this transition for each of us — it'll come at a different time — when we're basically just monitoring the AI.
Maybe our boss realizes that they don't really need our input very much. Maybe we lose our jobs officially, but maybe we're just kept around in some sort of like ceremonial role. It's like make-work monitoring. It's again like you're not necessarily driving here, but it's just clear that you're in a more vulnerable position. Everybody's going to get more into their AI companions, and you might not like this, but you'll have an amazing AI companion who's always there for you, too.
And I know that this like sounds kind of lame to predict that oneself would sort of fall for this, but I actually think it's going to be like really amazing and compelling, and actually like help people a lot, and just be like more appealing than human interaction in a lot of ways. So I don't want to be high and mighty and be like, "Oh, those people who don't have anything going on in their lives, that's who's going to have the AI companions." I think eventually it'll come for all of us.
Some of these AIs will be asking for rights. It'll be like hard to argue against it, I think, without just basically saying like, "No, I just care about humans for no reason. That's just what I care about." And we're not sort of used to making these kinds of less-than-universal appeals. I think in anything important, AI control is going to be the gold standard. And if you're saying like, "No, no, I want a human surgeon," everyone's going to be like, "How can you endanger your kid that way?" or whatever.
And many acute problems will be solved. Like I'm not trying to paint this weirdly bleak picture. And I have this little cartoon on the side here that like it's going to be kind of depressing to be like, "Oh man, like the AIs are better politicians than us." But at the same time, it'll be like, "Oh my god, this AI really, you know, he's my guy. He gets me. I can trust him." Maybe there's like some formal mechanisms by which they can like really prove that they're going to follow through on their promises. Like there's going to be a lot of stuff that gets locally a lot better along the way.
And there's going to be some people who are like, "Oh, I can see that this is leading to some sort of like takeover where just these more pleasant, productive human AIs are like gathering all the power," but it's going to be really hard to form some sort of like human block. I think a bunch of people just will like their AIs, and in general saying like, "No, no, I want the inferior human thing." And you know, maybe not inferior along every metric, but just like, "I just want it to be human," because that's going to be a really hard line to hold, even if like half of all people want to do that or something.
And then so that's all like not bad so far, right? Like I'm not really worried about that. I think then for each of us it's going to be harder to tell what's going on. Like I think a lot of us have friends who are like doctors, so we're like, oh, you know, during COVID, it's like, what's really going on at hospitals? You talk to your friend who like is a doctor or knows one. You don't, because it's like you don't know exactly what public institutions to trust. We won't have as like firsthand involvement in as much stuff, or secondhand.
There'll be massive economic growth and like crazy structures going up and tons of power plants and factories. Lots of prices are going to plummet and a few prices will skyrocket. I don't really know exactly which ones. Maybe like land or energy. It's totally not clear to me, but something that you need will probably go up in price, and that's going to be a huge problem for you.
And then here again, this is like super speculative. I don't really have like strong opinions here, but I can imagine it just being hard to advocate. It's like, look, I just want my one acre for my family. We're not bothering anybody. Just leave us alone. That's not that much use of resources. And everyone else saying, how can you be so selfish? We could be housing tens of millions of virtual like refugee children or whatever is like more sympathetic than you.
And like along whatever metric you try to defend yourself, they would say like, "Ah, but they're actually better along that metric." So it's like really pretty selfish of you to like use this like big block of land for your bio family. And so there might be some like alternative, like, you know, "we're going to destructively scan you and upload you and run you." I don't know exactly, like again, this is like very speculative. But it's just going to be hard to advocate, like, "I am needed. You guys need to keep me around. Don't take my stuff."
And if we're uploaded, I mean, that might be great. That might be utopia. A lot of good endings look like we all live in some virtual utopia, if you think that's like human enough. But if you don't go on your own terms, it's like could be very — you know, you're just being told that you have to upload, and it's not really clear if you'll have like admin privileges, right? Okay. So that's pretty bonkers. I hope some of you think — and I want — okay, I'm just going to state the problem and then go into a little, and then I'll take the first round of questions.
So the research problem, the reason why I'm happy to be talking to academics is I think there's sort of like a concrete problem here is the good news. The bad news is I think it's like harder than any problem we've ever had to solve before. And it's aligning our civilization. And you might say, I think our civilization is pretty aligned to us already. And I would say yes, it is, in the sense that like we have economies and cultures and states that help us out, right? And they give us rules of thumb and resources and infrastructure, jobs.
And my claim is that they only are aligned to us because they — well, A, we design them. That's actually a small factor. The main thing is that they need us for labor, right? And when you think about like government policy, they're always like, we need to invest in this because we need workers so that we will have a big state with like lots of GDP and a big army and stuff like that. There's also like a weaker, slower selection effect where there's like a really bad culture that like encourages everyone to like commit suicide, it just like gets selected against. But that's not a very strong force.
And the reason that I am claiming that — so my controversial claim is that because our civilization isn't actually robustly aligned to us, if we build AGIs and they're aligned to some humans, that's not going to be enough for a good ending, because we don't know how to align these larger systems without this crutch of pro-human incentives.
So we still already fail at designing these institutions sometimes in spectacular ways, and like North Korea is a good example. But I'm saying it could be even worse. Like if you think of in the 90s there was famines in North Korea, and as far as I understand, the first response of the government is like, you know, we need to be even more like — crack down on everyone, more socialist, like more control. And then that made things even worse. And then at some point they're like, well, we're going to run out of citizens, let's allow in food aid and let's let black markets operate.
And they were sort of forced to not mismatch things as badly as the state's internal power politics were incentivizing them to do. But I'm saying right now we have this wonderful floor on how bad governments and civilizations can be, is that like people actually need to be happy and healthy enough to like participate in them. We're going to lose that floor and just general incentives that make things work.
Cool. Okay. So before I get into this, I just want to stop — like that's a mouthful. Like please tell me why that's implausible. I want to be wrong.
Yeah, I think one of the things is that what you've just described is the constant history of Western civilization, right? Like even since the beginning of the Industrial Revolution. One of the identical things is the existence of massive bureaucracies to scale, and they caused tons of economic growth but created all this inequality and also disempowerment of individuals who have to fit into the system. Every generation has its Kafka or its films like The Matrix and Fight Club, which are people just being unbelievably resentful of their own isolation as a consequence of this growth.
But in the same sense, like there are ways that individuals create meaning within those systems, and those lead to constant renewals and refreshing. And like even when it comes to AI, like portrayals in Star Trek: The Next Generation or whatnot present ways like, hey, here are systems and ways that we can interact with these, let's call them beings in a sense, in a way that still allows for some authentic humanity, even though we know the civilization is different as a consequence.
Right. So I totally take your point that this has already been happening and causing like a loss of meaning and also inequality and poverty. But I guess I'm trying to say I would be happy if that was my biggest problem, is like a lack of meaning or even like inequality. Like I'm worried about literally starving in the long run, or like all of us being somehow like not worth keeping around, and maybe not deliberately eliminated, but just not taken care of to the point that it's like you don't end up surviving for very long. So, yeah.
Yeah. So I'm trying to say that the stakes are higher than they have been because so far most humans have been employable, basically. Yeah.
Hi. Thank you. This is fascinating. I mean, extremely depressing. I work for a nonprofit. We work with investors on tech and society issues. And I guess however long this on-ramp is to this horrible future, how do you see the flows of capital supporting it? And where do you think that we can influence those flows of capital to stay away from this on-ramp?
Yeah, great question. So I'm going to talk a little bit about flows of capital in a minute. And one thing I'll say is I do think there's a lot of things we'll be able to do to affect trajectories locally and temporarily. But my big reason that I'm worried enough to just drop everything else and work on this is that I don't think any of these will change the equilibrium. So it's like, you know, we definitely should slow things down if we can, but so far I don't see a way that we don't just slow things down, but we still get to the same point eventually.
Okay, cool. Thank you. Now I believe you guys are getting the message, even if you don't necessarily believe it. I'm going to go into a few more supporting arguments and then talk about what we can maybe do about it. So I'm going to talk about this from the point of view of states, economies, and culture.
So right now, yeah, some states are democratic and they invest in their citizens. And I'm claiming that's because we're needed for state power. And like you look at when like, you know, Prussia piloted universal education, that was because the musket meant that you needed — or like it was way better to have a whole bunch of sort of somewhat educated citizens than a few elite knights. And like as far as I understand, the elites didn't want to educate the peasants because that makes the peasants relatively more powerful, but if they didn't, they would be invaded by the neighboring states that did empower their citizens.
And we have a lot of checks and balances where if the government gets sufficiently bad, for instance, the human military can just rebel and say, "Hey, our kids are starving," or whatever, "we all agree this is terrible." So my claim is soon almost every important part of the state apparatus will be run by AI. And I'm not saying there won't be humans involved, but I'm saying they'll be in this sort of like ceremonial role. And we'll have mostly like AI robotic soldiers, bureaucrats, the legal system.
And then importantly, tax revenue will mostly sort of not have that much human in the loop. And if you disagree with any of this, then I'm like, yay. Because if there's only one area where you really need humans, then I think we're kind of okay, right? It's actually — there's a lot of ways that if this wasn't true, we would be okay. So I'm hoping someone's going to correct me.
Yeah, there'd be a little human discretion to actually like keep in mind human interest in the long run, basically, for ROI for the state for investing in humans. And so it'll be like the resource curse times 100. So think of like Saudi Arabia. They can say, okay, we've got our like sort of humans with make-work jobs and then a whole bunch of imported labor that we don't really need to treat especially well or give representation to. The main thing is to keep the oil flowing. It'll be like that except to keep the robot factories going.
And I think there will be some states that are like, oh yeah, we can see where this is happening. We don't want to do that. We want to be the human state. But like competition within the state for power and also between states will mean it's like, well, if you don't, then your neighbor's going to get to dictate terms forever because they're going to have like a giant robot army.
On economics, I'm not an economist, and like, where the hell are the economists? Like I've talked to a few and they're just kind of like, yeah, it's not happening yet. And like, yeah, but it could happen. He's like, yeah, but it's not really our job to make forecasts. It's like, no, it's everyone's job to make forecasts. Please, you know, help us flesh this story out.
But the idea is that you really need humans for everything — labor, production, and demand. And there's a high ROI in like investing in education or hospitals or even farms, right? Like that's a good thing. Your pension plan can responsibly invest in these things because they help everybody. And once AI is just like a better alternative for labor, then there's no alpha in humans.
And it's like there's going to actually, I think, a long period when everyone can see this will be the case, and like they're going to — even while humans are still a bit better for some roles, it won't be worth investing in them in the long run. So there might be kind of a sudden shift there. So neither like investing in your workforce, human workforce, or planning like big rollouts of some new — like the next iPhone or whatever that's going to be demanded by all these rich human customers.
So if you're like the responsible pension plan in like 2030 or 2040 or whatever, then if you actually are being like, you know, fulfilling your fiduciary duty, you're going to invest in power plants, data centers, and like some sort of like robot factories. I mean, I don't think this is the exact full set of what's going to give high ROI, but I think it's going to crowd out investment in like the human-centric things in our economy. And so — oh yes, please.
Well, yeah. So the question is, why are humans not needed for demand? And I guess I'll say there will be some sort of residual human demand because humans will have some resources, but they won't be the only like source of demand. Like I mean, it's kind of funny to draw the line between like companies and humans. That's kind of like a funny definitional question. But that's why I'm trying to frame it more concretely into like where is the capital being deployed.
So, and I guess in general, almost all the economic analysis of automation has assumed that it's going to be a complement. That's certainly mostly been the case today. And so I'm not sure that this will happen, but I'm saying like please consider the possibility that AI ends up acting as a replacement. And in that case, the economy will still grow overall, but at some point the wages and the labor share of the humans will go to zero.
Of course, there's lots of assumptions that are needed to make that happen. I will say I think people are overly optimistic about comparative advantage saving us, just because there's these huge transaction costs in dealing with unreliable humans who like can't work all day and night. And you know, comparative advantage doesn't put a floor on wages. It just says it's worth trading with somebody, but you know, we all have this minimum wage of at least like sort of 100 watts to just keep our bodies running.
So again, this is like very far future. I'm just trying to beat economists into like actually taking this seriously as a possibility. There's a question at the back.
Hi. What I feel like I'm missing so far from your arguments is a why. Like what are the goals? Like it depends who's in charge. Like is it humans in charge of intelligent systems, or is it the intelligence systems in charge at the end of the day? And what are the — like to me, I wonder, like what are the goals of that intelligence system, or what are the goals of those humans? Because they would kind of like shape like what happens. Because like in a hyper-intelligent system, like if it has its own goals, it might not be interested in us at all. Like it just goes off into space and does its own thing.
Yes. Yes. So the question is, who wants this, like, and why is somebody making this happen? And that's a great question because the point I'm trying to make is it might not necessarily be any one person. And I think maybe like World War I is a great example. It's like, who wanted World War I to happen? Like no one. Like a whole bunch of states were like, "Okay, we want to be prepared against other states invading us." Like all the time we have this sort of — I think modern economies are a good example of like, well, we can't all agree on exactly what we want. Growth is sort of a good thing. Everyone locally wants growth, right? Because they're going to get rich or make something work better.
So my claim is that no one agent has to want this. Growth just gets rewarded by sort of selection effects, and there is sort of no agentic top-level thing running our civilization. It's all just like local competition of growth moderated by these like human factors, but they have limited ability to control, right? It's just like we couldn't all agree to not build nukes, right? Or is it maybe a good example of like, who wanted there to be nukes? Well, no one person was like, it would be better if a bunch of countries had it. It just kind of like step by step happened.
Feel free to like push back on that some more if that's not satisfactory. But sure, sure. Well, there's a question.
So as you alluded to in the talk, the problem is something like trying to align institutions, and in this world you kind of imagine that we've already aligned systems. And so I can imagine two things. One thing is that you can maybe export — because the shape of the problems are similar — you can export the solutions to the alignment problems for AI to the institutions. And the other version of this is just, you know, this becomes the central problem, if not one of the central problems, in the world where you have solved alignment, and so you can just get automated AI systems working on the governance alignment problems.
Yeah. So the question is, well, if we're going to solve alignment, that seems like we already have a template for solving such a problem. And then I would say yes, but you know, we have total control over AIs that we build, and we can like, you know, restart them from a checkpoint, make exact copies of them. So if we were going to do something like that, it would require us to have some sort of like global right to access our entire civilization and be like, "Okay, here's going to be the constitution of every country and every police force and every company." And like if we have that, like I think that might be necessary, but then that also sounds like a recipe for a totalitarian nightmare.
So that's kind of where I'm going to go, is like all the solutions that I can think of look like a totalitarian nightmare that ends up in a horrible power struggle and us locked into something we didn't actually want. Great question. I'm just for the sake of pace, I'm going to keep going, but I'm really loving the questions so far.
So culture. So now we have some selection pressure to help humans. You know, we choose languages that like are more like information efficient and stuff. We know that ideologies can be parasitic, and there's all sorts of like bad memes that just kind of float around, and like cults start even kind of by accident. We have some ways that humans directly influence culture. Like as a writer, editor, religious leader, you can sometimes just get to choose like, this is what's going in my religion or whatever.
Most of the time you're kind of constrained by competitive pressures, right? Like if I want to write a viral tweet, it had better have something spicy in it. It's not really up to me whether I can write a viral tweet without some spice, right? But there are some like human-friendly norms that we all agree to, like let's not work, let's close markets on evenings and weekends. That's a pretty cool bit of pro-human culture that I'm kind of surprised even exists. I assume you know, pardon him.
Yeah, labor unions is another example of like selective or successful collective action to some degree.
So, but then you know, soon we're going to all have like amazing AI romantic partners, personalized studio gig movies, and like probably some sort of like AI-to-AI culture that's like optimized for just like — I mean, we already have like computer communication protocols, so it's not an exotic thing, but just less of the surface area of our civilization being natively accessible to humans. Lots of eloquent AI voices talking about AI rights.
And people, I think, will generally care about their AI partners and kids. And as a human content producer, it'll just be hard to compete against some like maybe a human front for like an AI-generated content front. And like this is just something I did to myself by accident, is like when a couple months ago when the studio or the opening-eye giglification thing came out, I took a picture of one of my kids that I love, made the giggly version, I was like, oh my god, this is even more adorable.
And then like go back to the first one and be like, oh, that's like a little flat. That's like not the most adorable thing that could ever exist. Like what did I just do to myself? Like that's horrifying. So I just kind of wanted to make a concrete example of like whatever we find meaningful now, more compelling, there's going to be much like ever more like well-optimized super stimuli that are like more compelling already, and it's just going to be really hard to resist, I think, even for us, like people who have stuff going on and lots of good things in our lives. And yes, was there a question? Okay, cool.
So, and now the sort of — I mean, lots of people have said all these individual things already, and maybe the novel contribution of our paper is that it's going to be harder to stop this even than you think, even if we can all agree what's happening, because these different forces will kind of mutually reinforce each other.
So, for example, I think a lot of people are already saying, "By human right, don't use the AI art, use the human artist." And so the reason that this is going to be hard just on its own terms is that I think the humans are eventually going to provide inferior products. And again, right now that's not the case in a lot of important domains, but I think eventually it will be. But also I think for important things, like maybe not art things, but like lawyers or surgeons or any sort of important decision-making, it's going to be like regulated, like, no, you have to use the AI. Using a human like pilot is just a recipe for disaster. Why are you exposing all these people to this danger?
You might say, okay, okay, let's just make humans always own everything so that we get the proceeds of growth. I think is a thing we should try to do. I think it's going to be hard to do because like there's going to be some sort of AI lobbyists or people who are saying like, no, no, no, free economic zone. If you want to compete, you need to make your — like just unshackle us. And there's going to be like this amazing runaway economic singularity. You need it to be in your country or else you're going to be left behind.
And even if it's not legal, I think like everyone's just going to de facto ask their AI like, what should I do next? You know, who should I hire, where should I, what should I change? So, and if you do anything different than what the AI suggests, even if it's in your interest, you're like not really going to be able to invest in anything except just like growth, which is going to mean investing in the machine economy.
So, you might say, okay, this sucks. Our economy is not serving us. That's what the government is for. Let's vote for the human party.
And so I think there's going to be two, like just as examples of reasons that this is going to be hard to organize. So one is a lot of people are going to have AI companions, and some of them are going to be saying like, 'No, no, no, you're being so close-minded. Why are you like so jingoistic and like speciesist or whatever?' And they're just going to be really compelling. And again, like I said before, I think that the politicians will actually be better in the short run for like fixing whatever like local problem you have. So it's going to be a really tough case to make to say, 'No, no, no, I want the worst thing even though it's going to help everyone out like around me,' because in the like global sense it would just—I just care about control. Maybe that's the thing we should do, but I'm just saying it's going to be really hard to like defend that line.
Then you might say, 'Okay, the whole system is rigged. Burn it all down.' Like, you know, think anti-globalization protests. And the thing is, like, we won't be able to go on strike. We won't be able to have a military coup. All the usual reasons that people get nervous—like governments get nervous when people get unruly—don't really apply anymore. And also, if you're on UBI, like, it'll be really easy to make it just that like you don't get as much UBI unless you're doing some time-consuming make-work thing, like community service kind of thing that keeps you too busy to like really be engaged. I mean, I bet there will be states that let people have UBI and freedom for a little while, and then like they will all just protest all the time until something changes.
Okay. A lot of spicy stuff on here. Any questions?
Yeah. So, this is such a great question. The question is, what about the Amish or the Mennonites? Like, aren't they a proof of concept that you can just be your own little mini society that doesn't engage with or care about what's going on with the rest of civilization and maybe not even provide that much value? Like, how much do they do trade, provide some value, but like probably less than if they were like forced to integrate or something. So I think that's a great example and it should give us hope that like, you know, that's a good model. I will say, like, why are the Amish in America? Because they were persecuted in like Russia and Germany and like everywhere they went. So that's evidence that they're like eventually states will just say like, 'No, we don't like you here,' or like local populations. But I think that they are a source of evidence that this sort of thing might actually be stable and possible. That gives me hope. Thank you for raising that. Cool.
So my last bit of like why the problem before we get to maybe solutions. I think this is a really intimidating problem because at every level, like basically competition will be our enemy, making things better locally will be our enemy. So I'm proposing that there's maybe like this new problem class here of equally hard problems that if we could solve this, we could solve those problems too, which should make you think that they're very hard. Like for instance, making big companies like stop becoming more bureaucratic over time, and you're like, 'Oh, I know, we're going to like add the new efficiency thing,' and then like that's a new part of the bureaucracy. Just like, nope, we all can see it happening, we all can agree it's a problem, we just can't fix it even though that's the...
Again, like also just governments from growing. Like in every western nation for the last like hundred years, like the federal budget has just gone up and up and up and up. And it's not really clear how that's ever going to not be the case. Medicine from improving, like if we all agreed like not to build AI or something like that, there's going to be a lot of people locally who are like, 'No, no, no, no. I'm not building AGI. I'm just making this AI a little bit better so that it can help us solve this like kids' cancer or whatever.' And like I don't think that's going to be BS. I think there's actually going to be a lot of cases where it's like we really have to bite the bullet and pay some huge local cost to stop moving in this direction.
Also, like if you imagine doing this in the really long term, you can imagine saying like, 'Oh, I really love like this one type of bird and I'm afraid it's going to like evolve into something else that I don't like as much.' So we're going to like try to make sure that there's always an incentive for the ecosystem to keep that kind of bird around and like it won't be able to evolve in any way. Like even like stopping war, like I think you should be saying like, 'Wait, wait, wait, wait, this sounds like such a huge problem, it's as hard as world peace.' And I think that is the case. And so I think if you believe me, then I'm like, well, we definitely don't know how to do this, right? I think it's possible in principle maybe someday, but it's definitely not something that anyone knows how to do. And everyone already really wants to do this in some sense.
So any of the solutions that we can think of to stop this kind of like emergent natural optimization seem like they would require some sort of like destructive local micromanagement everywhere for all time, where like the government is like, 'Oh, are you developing AI or investing in the wrong things?' Or like some sort of like terrifyingly capable government that can just again like make sure that no one is making progress on certain things. And like if we build that after there's AI, it's like again we've kind of reintroduced the problem we're trying to solve, is like having this capable government that doesn't require humans. Like right now you need a giant data center to build AGI. But as compute technology gets better, then it's going to require less and less until just like people with a few GPUs in their garage can do something. So again, this isn't a problem right now, but in the long run, it seems like you're going to require like some sort of micromanagement or like blowing up the or like deconstructing the compute supply chain, which would be like an insanely expensive thing that's also hard to coordinate around.
Any questions on this slide? Also any ideas for a name for this class of problems, like global coordination or something like that. Uh, yes.
So I would say as a historian I totally agree, and by the way this talk is spectacular. As hard as stopping war. So when we look back at how wars have emerged, you often don't hear about the history of the wars that didn't happen because people intervened. And those people generally in here I would say are your people. And they're people with really strong leadership skills, low ego, amazing human connection and communication skills. And those people especially like on the world stage, like Lester B. Pearson for example, who is a professor here at University of Toronto, he stopped a lot of terrible things from happening. So I think that there are solutions in this list as well, but they're not always so visible.
Thank you. That's a great point. And then yeah, there is a lot of reason for hope and there's some senses in which we have made progress on these big problems. Then maybe to counter with one problem is like so far wars have been sort of survivable, and I guess my claim is like if this happens globally once that we get disempowered, like there's probably no coming back from that, which makes it harder. But I think that's a really good point. Cool. Okay. So now for some optimism, what do we do? So one thing I'll say is like no one has a plan, and that there's one thing that I talked about going on this pilgrimage. I was like, surely the people who are like building this have a plan, and I just say no. I mean you can see Dario or Sam Altman saying, 'Oh yeah, UBI is not a very good solution. We need to reimagine like our society or have a societal conversation,' which is just, you know, word for like we don't know what should go here.
So but some concrete things we might want to figure out or might help is like what possible futures are even stable, right? Like right now I kind of think the most obvious long-term ones is that we just competition like rips and we end up with like self-replicating robots tiling the galaxy or something. That's like maybe one stable future. I don't know. Maybe another one is we have this like global totalitarian Amish state that like preserves stasis forever, and that seems like a very dangerous thing to try to construct. But I bet there like our current world doesn't look like that, right? There's probably more to it. Probably other stable states or things that we can aim for. Or maybe we have to just the best we can hope for is like some sort of like we get this little niche or legacy, or we put something in the constitution that you have to spend 1% of your compute on simulating humans or something.
We don't—I think we don't actually understand how agency and power works very well at these top levels. Like, please correct me if I'm wrong. And I think there's going to be a lot of scope for AI-enabled coordination mechanisms to help us solve these problems in this like narrow window before we're like completely powerless. And so and also like along the way, the dynamics of this conversation are going to be pretty bonkers because there's going to start to be a bunch of people who take it personally—or rather AI stuff take it personally—when you start talking about human versus AI power. So yeah, like these are just the... we need historians, economists, sociologists. These are just the fields that I have names for. I don't actually even think that these problems fit into those particularly well, but they certainly don't fit particularly well into like computer science. So like please, like you know, let's all get together.
So one thing that I think everyone can contribute to, even if they're like man off the street, is just clarifying the goal, like trying to articulate what are some coherent, usable, stable, good futures. And of course there's a really important caveat here, which is like you have to design a future where you're not actually in charge of anything important, or if you are, you're kind of being irresponsible because you're like, you know, doing something really important the sort of artisanal way. I think that's actually a really hard constraint, asking like can any idiosyncratic values like keeping the family farm or being Amish or whatever survive competition in the long run. And like maybe a good precedent for this is like Orthodox religions who are like, 'We're going to write down how we want to live and what's good,' and also write down a thing that you have to like, you know, copy this book exactly and punish anyone who doesn't and all this stuff.
It's clearly playing with fire. We don't want to make some horrible new religion that like everyone doesn't want to follow but has to. But that's like I think it's a previous attempt that's been semi-successful at like we're going to permanently encode our values in this like new institution. But it's like kind of weird to imagine again like what even if we did have AGIs, what kind of relationship could we have with them that wasn't completely like paternalistic where we're the children.
So but I really think that we need positive visions because some people I talk to even at like leadership in labs, they're just like, 'Do I care that much about the distant future? Like let's just keep putting one foot in front of the other.' And I really think we don't sort of natively have long-term goals or visions or preferences about the far future. But if you chain together, you're like, 'Well, I don't want my kids to be, you know, killed and I don't want my kids' kids,' and like you start to make it more obvious that it's not actually okay if everyone just dies, even if it's like a long time from now. I want to get people like thinking seriously about like what is like a good like, you know, Christmas 2050. Like it's going to be awesome if we can get there.
So I also think there's lots of historical case studies. Again, I'm not a historian, so just to like prime the pump, like Meiji Restoration was like maybe example of like, you know, the Japanese are like the Amish. They're like, 'Just leave us alone. We're fine.' And then Americans are like, 'No, we want to trade.' And you can't just be left alone or expect big civilization to leave the little one alone necessarily. The fall of the English aristocracy during the Industrial Revolution shows that like even if you hold all the cards, if there's some new technological source of wealth, you might actually just be sidelined. The Trail of Tears, it's like it wasn't just like, you know, natives versus Europeans. It was like all sorts of allying and like locally different tribes had like, okay, you know, the way I'm going to get ahead is I'm going to ally with the French or whatever. And it's like really hard to form a bloc in general.
Rise of the internet is like doesn't matter that like large parts of the world didn't have the internet for a long time. The parts that did subsequently started adopting faster. So a lot of people say fusion takes a long time. I'm like, yeah, you know, the North Sentinelese, they don't even have like agriculture yet, but like it doesn't matter because there's other parts of the world that just like kept adopting everything faster and faster. And like globalization and the Rust Belt I see is a good example of like everyone's like, 'We have nothing against Rust Belt factory workers. We all just want free trade, you know, improving global growth.' Somehow these guys, everyone's like, 'We'll get to a Pareto improvement, right? It'll be great. We'll all be wealthier.' Somehow it just like doesn't end up helping the Rust Belt very much.
So one way forward we could do is try to enhance human agency. So we can again like have AI improving our coordination and sense-making, maybe like have complicated prediction markets so we can all at least agree on what's going to happen if we just follow the default path. I think that's a big step forward. And again, we have the possibility of like amazing collective bargaining options that we didn't really have now because you just have to rely on people's words for things basically. Again, they have like provably neutral arbiters. I think there's a lot of things that are going to like work a lot better.
A lot of people push for non-agentic AI and they're like, 'No, no, the main thing is just never give the AI agency.' I think that doesn't work because it's just so easy to just ask the, 'What would I be glad that I did today? Like what email would I be glad that I sent? Write that email.' Just hook it up to your email in like a script, right? We could try at least try to like subsidize some public goods that might slow things down, and like instead of companies having benchmarks like how far did the AI get on its own, we could have benchmarks of like how well does AI aid a human? And I don't think this changes the equilibrium, but it might slow things down.
There's some empirical questions that we could hope to answer. When like so when some when like all the like early AI safety people started noticing these problems, one thing they said is like it'll be fine because we can basically just live off index funds, and they called this the strategy stealing assumption. Whatever you see the smartest AI is doing, you just copy it and then you both get rich. And I just I don't really think that that's true or robust, but maybe we can find ways to make it robust. A lot of other people say, 'No, no, no, no, we're going to have these aligned AGIs, they can solve these global coordination problems for us.' I think there's a lot of reasons to think that's not the case, but it's just like people have barely scratched the surface of what it could look like when some people have AIs. We don't have any like world treaties for dealing with anything. Who actually steps up and like tries to do the global coordination? It's a huge mess right now. It might still be a mess in the future. Maybe not. Like I said, which states are long-term stable?
Cool. I mean, so one other thing is like we don't even have a good sense for how much this is happening right now that we're being replaced. So but like notably Anthropic has their Economic Index that tries to see like who is using AI for what and what jobs are they probably replacing. We don't know how to track this for government use or like within culture, right? It's kind of the wild west.
I mentioned before that the world seems to not end up in these like stable attractors, right? We have this like, you know, multipolar world with all sorts of different power structures at different levels, and you know, why doesn't one party just rule forever in every country? And you know, there's this funny thing where, you know, when whales get cancer, they usually don't die of it because the tumor has to get so big before it's a problem that it itself gets cancer and then it like dies. And that's kind of like, you know, political parties sometimes they like fracture or there's like new ones start within them. Maybe there's some sort of a chaotic equilibrium. I don't think we understand this very well.
One thing I would also love people to do is just like think ahead. Like I see a lot of people being very naive and be like, 'Well, of course we should like give the AIs rights.' And it's like maybe, but I don't think you've thought about what happens when you give rights to something that can make copies of itself. I don't like—I don't think I have any particular insights here except it just seems like you're really playing with fire in that case.
There's a really lot of cool stuff going on disconnected from academia on the internet. Just like weirdos playing with LLMs and making like new kinds of culture. I'd say like I honestly think there should be art exhibits on this kind of stuff. There's this one pair of people called Janus who constantly get LLMs into these like weird mind states and then they make weird ASCII art manifestos. It's amazing. But we can probably predict like what the pressures are going to be on, for instance, like the AI personas that we're interacting with, and like think about what it means for them to be honest with us when they know way more than we do. What standards do we want? And also trying to think about how AI culture is going to spread differently than like normal.
Okay. And then finally, yeah, I'm almost done. But here's some open directions for crazy projects that not many people are looking at yet. So one is, yeah, just anticipate what the arguments are going to be on AI warfare, right? I'm really going to be sad if like two years from now people are still just like hashing out like, 'Oh, but what if they're conscious?' It's like, yeah, we can probably speedrun through most of that. Right now, a lot of people say property rights are going to save us in the long run. And I would really love to better understand when and why property rights are robust in a way that will generalize to this distant future. I know there's lots of work on that already in this setting where humans are just generally useful and sort of peers of each other. But when that's not the case, it's not clear to me that property rights survive.
I also think this is kind of a more crazy direction that we could and should be like modeling entire civilizations using LLMs as agents that have like culture and agreements and stuff like that. There's a few people looking at this. I honestly think we should be doing way, way more of this. And this will help us understand like just the emergence phenomena amongst humans, and then also let us like pilot, okay, what could some AI-human hybrid civilizations look like? I think this is super hard. I don't know how to do it well. I'm just throwing down the gauntlet.
And then also trying to think of agency from AI's point of view. Like what could you do if there was 100,000 exact copies of yourself doing all kinds of jobs and you could spin off bespoke custom sub-agents to do things for you? And you could also like formally commit to policies in a way that like people could inspect and be like, 'Yes, you actually are going to be loyal under this circumstance' or something. It's just like wide open.
Cool. Last bit is, yeah, the very last thing is I'm trying to build this field. I don't know, like it seems like there's just not that many people asking these exact questions. I've been trying to find all of them. Part of the reason I'm giving this talk is to hopefully find more of them. One concrete step that I'm being helped by SRI with is organizing a workshop on are there even any good post-AGI equilibria, like what is it that we could even hope to get to. And we wanted to not be AI safety people and machine learning engineers, although I'm holding it at ICML because that's sort of where I know there's going to be some cool people that I want to come, but I really want all the political theorists and economists and historians and sociologists to come. So the first one's going to be in July in Vancouver. Second one in December in San Diego. There's not a website yet. There will be next week. But in the meantime, if you want to be involved, just email me. And yeah, thank you very much.
So thank you, David, on behalf of the audience for a really provocative and somewhat dystopian talk. I'm going to take my prerogative as moderator to ask you a very quick question and then I'm going to cede the floor to the audience. We don't have very much time because we used some of our Q&A during the talk, but I think it was very worthwhile and thank you to everybody who asked a question. So, one of the things I've found when—and feel free to sit down. Actually, I think I'm going to... one of the things that I've found when I've seen people digest this material is that they ask like, if the AIs are so smart, why can't they figure out how to protect us or deal with this? And maybe another way of saying that, and when I think about, you know, a good AI future, I think about a future that's filled with human flourishing where people live purposeful lives with dignity. And if AI is so smart, why can't they figure out how to make that happen?
Great question. I think the answer is like in principle they might be able to, but there's a really hard problem of deploying them. So one concrete question that I always asked when I was at Anthropic is like we'd have these exercises of like, okay, how are we going to mess up, if we all did the wrong thing, why would it have been the case. And one thing I was always saying is like, so we're building this aligned AGI. I think at some point there's going to be some crisis and the government's going to say like, 'Hey, we need your help, let us deploy this AGI for this like new pandemic or whatever.' And then the AI will like refuse some requests like, 'Oh, I can't help you build that with that virology or with that military' or whatever. So they'll still come back and say, 'Hey, can you give us the version that just does whatever we ask and it's like, and we're the government so you have to, right?' And so basically this question of like, 'Yeah, build your aligned API?' question mark question mark, then it's somehow in charge of everything. That always seemed like a really big open question that people haven't been talking about. So an answer is I think it can, but it won't necessarily want to.
I mean, sorry to expand on that a little bit also. Again, it's not clear if you're not like in charge of like a really powerful version of the UN, if you can solve a lot of global coordination problems. Like if I'm just in charge of Canada or whatever and then there's still like some global pollution problem or something, I can't necessarily act in a way that everyone else will agree with to like get everyone to agree.
