David Duvenaud - What are Humans Even Good For in Five Years? (Early Experience of AGI - Episode 1)

Yes. Yes. Yes. But, uh, if you care to stop and think and be like, oh, maybe I can go on vacation, I can just like leave Robo Dad or whatever, like, uh, you know, you'll just realize like, yeah, the kids might even have a better time without you there. Or if you're there, you're just kind of like being puppeted by this Robo. Yeah. Yeah. Yeah. Yeah. Like you, you, you, the way you best show up as a parent is to have, is to be conjured by, is to be prompted by a system that in real time is drinking in the biofeedback of your children, is taking into account the things that they've learned and their own goals and objectives and all that other stuff, right? Like this is what you're getting at without like advanced sensors and biofeedback and stuff, like which I agree is totally probably going to happen.

Tastes more delicious every day, brother.

This is Daniel Fidelli. You're tuned in to the Trajectory, and this is the first episode of a new series on what it might look like and feel like when early proto-AGI starts to make its way into the world and share a space with human beings. We're calling this Early AGI Experience, sort of painting a picture of what signs we'll have when we're really dealing with something transcendently beyond us, really kind of running the show. And our first guest, I am delighted to say, is none other than David Duvenaud. So David teaches at the University of Toronto, where he's been for a great many years, and has spent a good deal of time at Anthropic. His focus areas are around AGI evaluation, AGI governance, or the technical considerations around artificial general intelligence. And David is a co-author on a paper called Gradual Disempowerment, which will be linked in the show notes. I highly recommend reading it. In this episode he goes into what it looks like when AGI is genuinely better than humans at everything. Most importantly, he addresses not the banal things like designing logos or routing traffic or silly banal things people talk about with AI. What about the relationship between husband and wife, between parent and child? What about when AI is better? In other words, more fulfilling, more enriching, etc., even in those sacred areas. David looks squarely at these things with no cope, which for me is remarkably heartwarming because on the show and just in life in general, Jeepers, I sure do deal with a lot of cope. So I'm not going to ruin any of the special stuff, but tune in for the more profound shifts to some of the things that are normally seen as sacred that David brings up, and I'll save my commentary for the end. Without further ado, this is David himself here on the Trajectory. So, David, welcome to the program.

And I was just mentioning to you off microphone, like I've thought less about what does the emergence of AGI look like and how does it kind of like create some shudders in society than even I wish I did. And I picked up a lot of nuance that I was really happy to see in Gradual Disempowerment. I want to kind of open with what's profound for you and what you think matters most in terms of like the painted picture of what fundamentally shifts in like the work world and the personal world as AGI, the dawn sort of emerges on us. Like where do you want to kick off? What should people know?

I mean, one of the reasons that I started reading your stuff is I was like, this is a guy who's actually sort of maybe gone through the five stages of grief and is able to think clearly about this to some extent.

Yeah.

Yeah. Well, well, let's, let's unpack that because I, I think the whole like the stages of grief and the parts of that that are really disturbing, we'll get there. Also, you mentioned something really interesting about even at the big labs, the whole, I guess I'll retire, chuckle, chuckle. We're going to double click on that a little bit later. But among those sort of like machines better than people across the board, you know, there's a lot of mostly what I consider to be sort of inexcusable cope around sort of these eternal barriers of like, well, no, no, no, it won't at these things. Well, no, no, no, it won't at these things. And of course, David, if you recall, and certainly you do, you go back even two years, never mind five years, never mind eight years of all the nevers, right? All the surely-nots that we have just obliterated already, right? They've just been plowed through. But what are some of like the, maybe we could start on the work side or personal side, wherever you want to begin, but what are some of the bigger sort of like machines are likely to be better at this thing that you guys, through, you know, your research, your paper, looking at trends, sort of came to a conclusion of that you think maybe even some people listening are kind of resistant to and don't think is going to happen? Like what are the jarring ones?

It seems like the way I think about, maybe you've got a different response, and I'm sure even in communicating what, you know, you guys came up with in Gradual Disempowerment and other formats, I imagine you get a lot of people pushing back on it. But the way that I see it is like the tide is somewhat clearly moving in a certain direction, and like people look at, it's insane to me, they look at the latest models that were produced and sort of where they benchmark, and they're like, oh yeah, man, Anthropic is done, I guess, man, they'll just never catch up. And then they'll be like, oh well, it's clear AI will never do this. And then the next day comes around and something else happens, like this company's the best, they can do this. It's like, why, why not just see like over a six-month, never mind an 18-month span, what kind of progress in these domains we've made and just roll that forward? I don't care if Anthropic, OpenAI, or like my mom is the one that does the next breakthrough. It doesn't matter. It's just like if this is where it's going, where does that take us? It feels hard for people to do that. Like why? Because surely again in communicating what you guys have come up with, people have given you very banal denials of current trends. Like what do you think keeps people there?

Yeah. So, so it's almost like a kind of cope of like, well, no. And there's all these funny memes of like, you know, like a giant machine god sort of finally devouring the earth and somebody being like, it's not even really reasoning, you know, like just dumb [__] like that. It's like some weird little petty petulant ego victory like in the face of like obvious oblivion and sort of like irrelevance.

Yeah. So, well, let's, let's walk through work first. I just at least want to make this visual, and if you even pull up Gradual Disempowerment and want to pick your favorite examples, great. But I just, I want to like, I want to get some tangibles in people's hands because we're going to do that with the personal side in a pretty robust way in a second. But like around work, you know, things that feel like, again, neither you or I know exactly what dominoes are going to fall first, but maybe it's like, you know, drafting legal documents. Like this is low-hanging fruit. But then there's other things like, okay, determining based on inventory in our factory, like, okay, based on our marketing and campaigns and annual goals, based on present inventory levels for these different things, based on the pricing and inventory requirements to produce this many product that we think we're going to sell, like when should the orders for these plastic pellets and these metal springs get sent out and to which vendors at which yada yada. Like these are the kinds of things that right now AI might help with snippets of, but it's not orchestrating the process. Are there examples you could articulate for jobs that feel very safe that you'd be like, yeah, well, here's how it would just get buffered out, you know, like what do you want to cover there around work?

Well, what you just brought up brings an analogy to mind. I want to see if this analogy clicks, or maybe you have another few analogies that we could unpack and explore. The analogy is like, you know, I took an Uber here in Austin today, closer to the city, and it was a Waymo, so I just like jumped in and [__], you know, and in Boston they don't really have a lot of that, so it's kind of novel for me. But it struck me as like, yeah, of course, well, you know, you have humans do a bunch of driving, and you kind of figure out like whether it's paths or routing or intersections that are traditionally very troubling, or intersections that are simple, or whatever, right? All the context, you know, how to do it at night, how to do it in the day, what to watch out for, where there are a lot of children playing, whatever. And then at some point machines can run it. What you're saying is, hey Dan, that just scales. In other words, if you have your AI assistant sitting next to you as captain of procurement who has to take marketing into context, who has to take vendors and vendor pricing into context, who has to take shipments and different levels into context, who has to look at spreadsheets and check certain things, like at some point the AI is updating the spreadsheet in real time based on data sources you even as a human couldn't operate and find, and it's already finding lower prices than you could find. Like this is already updated, and then at some point, you know, it's taking weather into account or like sea conditions for stuff that has to come from Japan or whatever the case, it's actually factoring for things and really simplifying and maybe streamlining and making more accurate some of the details. And then as you start to bounce from the details, you say, man, like four in a pallet should be enough, and then it'll sort of look at like historical marketing promotions versus the marketing promotion you have coming up and say, you know, it suggests, well, look, no one's hit target for one of these promos in like the last four years, and it doesn't seem like we're doing anything fundamentally different. So I don't think we're going to have a sellout here. Like if you want to be safe enough to still have, you know, plastic pellets, but you don't want to have two warehouses full of them, like this is a range that would really make logical sense based on just what we've seen. And increasingly what you're saying is those suggestions might start off missing some context, and then humans will give AI more context to those things, and then those suggestions will be like a good addition to our cognitive process, but then those decisions will be like almost ubiquitously what we should do, and that at some point it gets so narrow that like, you know, there's no more contribution. Is this a proper or improper distillation of what you said?

Okay, go on.

Yeah. I mean, because and it's somewhat, it's somewhat incredible that this isn't extrapolated to like, you know, you see videos of, and we're going to get into why these things aren't talked about openly, but you'll see videos of folks within big tech that are like, yeah, it'll be great, there's going to be humans, you'll be managing a lot of great AI, you know, workers that are going to yada yada. And it's like, you don't even have to extrapolate trends in a major way. You can just be like, what happens when they can manage things?

So, so it's sort of like, and we'll talk about why the truth is obfuscated. I'm really going to want some of your perspective on some of that. I have some hypotheses I actually would love to see if you can validate or invalidate in our call here today about that. But yeah, like people don't see that. It strikes me like here's how I will see it rolling, and I want to get how you would see it rolling out now that we've distilled kind of the same essence here. I would see it as there are some people through whom this kind of constant series of AI prompts kind of just being good. I think it'll work for work. In other words, like, hey, what should I do with this inventory level thing, right? But I think it'll also work for the personal life. What should I eat for dinner? You may feel like eating pizza, but like it may know sort of your satisfaction levels via, you know, like eye movement and BMI or like EKG or whatever, like various and sundry bioactors at some point, your personal assistant, your [__], you know, whatever little bio ring you're wearing or something. And it may make suggestions for, you know, films or places to go for a walk, or, you know, it'll conjure virtual media that's never existed just for you that'll be humorous explicitly to you given the mood you're feeling today and given your real-time eye feedback and stuff. So when you say I want to laugh, you don't say show me this Dave Chappelle sketch because it's not going to be as funny as what the AI gives you. And so there's a degree of like handing off the baton almost entirely. And I think most people would love to just pass it off, right? And just, man, just get washed over personal, professional with whatever the AI suggests. I think on the work side there will be some people who try to ride the tiger for as long as they can, which I consider admirable. There's kind of this world eater, lotus eater dynamic I talk about all the time. There's an NSA called World Eaters and Lotus Eaters. The world eater, the lotus eater side is just like let it wash over me with whatever suggestions. I'd like to get as deep into pleasure and as far from like struggle as possible. And like you talk about, David, many people say, oh well, yeah, it'll give us a great vacation, we'll be on vacation all the time, like, and it's like, ooh, buddy, you know. But the opposite side would be, okay, it can tackle all these things. I want to apply my volition to that razor's edge of where like vision, goals, agency, impetus, values, and like judgment and strategy. I want to pinch my 80 hours a week of work completely into that, right? Get rid of like banal email responses. Get rid of, you know, even inventory as a consideration. Focus on the screaming edge of product and market share. Focus on the future of this product line and new source materials. Focus on the impact I want to have in the world and try to just [__] scream it out of that top band. But at some point, even those world eater people squeezing 80 hours out of what, you know, even the most ambitious of us, like Musk today, for example, will only spend 20% of his time on that screaming edge. There's going to be people pushing 100% of their 85-hour work weeks just into that. But at some point does that go to [__], you know, like what are your thoughts there?

Oh, that's funny. It's dodgeball.

But you're not even saying that to real humans. You're saying that to like your AI friends that are way better than your human friends.

So, okay. So you kind of see it that way, that some people will do the squeeze, but what you're saying is like at some point you are literally just not, it doesn't make sense for you to be doing those things.

So yeah, well, and we could think about the areas that I see people kind of defend are like areas like, well, what if I'm like an author and like that's not optimizing for something for me, it's like I like to tell fiction stories that are really yada yada, and I'm just going to keep, or like, well, what if I'm like a philosopher? What if I really, right? Because I could try to cope, David, couldn't I? I could say, well, I'm, by the way, I'm no formal philosopher whatsoever, but I am trying to paint a picture. I could say, well, you know, my ideas are so important. Why, surely, you know, an AI can optimize for inventory and eventually win market share and yada yada and invent new industries, and humans aren't even involved in those industries, that can all happen. And like, it's not going to have the same curiosity I do about these areas, but it's sort of like, well, what if anybody that wants to know about, let's just say in my case, the future trajectory of intelligence and sort of posthuman life, whatever, what if just asking even the worst possible models about like what those trajectories are, where they're going, what's likely, what's not likely, what humans might do to get there. Like, what if all of those ideas are just more interesting than anything I ever have to say ever? And like more rich and robust and grounded in different ways than anything I've ever written for the entirety of my life or ever could create. It's like even the areas where it's like, well, I'm defensible because I'm not to be optimized away. It feels like even that could be optimized away. I mean, are there areas you would agree or disagree with that supposition?

Yeah, I mean, spend a lot of...

And even when people say, well, no, but I would keep doing it to enjoy it, I would keep creating because that's where I get fulfillment, it's like, would you though? Like if you have a system that's completely trained on what fulfillment is, whether it's social accolade, sexual gratification, humor, whatever, all the rich components of fulfillment, not just bliss, not just [__] something lame like eating chocolate ice cream, and you know, like if it's the full range of blisses have been calibrated in terms of your agent experiences and biofeedback experiences and, you know, eye trackers and everything else, right? If that's all been taken in, there's a virtual world that is 20 times more fulfilling than whatever you think your actual work fulfillment is. And at some point you taste the lotus once and you're like, and you step out just like you said from the dodgeball game, you're like, yep, I'm out. Wasn't actually, actually wasn't even ultimately fulfilling. As it turns out, fulfillment can be optimized for. As it turns out, go figure. Go figure, Buck. As it turns out, you can optimize for that too. So, let's get into some of these uncouth things that you've thought through and that in Gradual Disempowerment you poke into on the personal side. So, there's two areas, and I really want you to lay this out because honestly, I think there's components of these you've thought about in ways that I haven't even. I would love the audience to digest this. Romantic love and relationships and parenting. You are literally, I'm not joking, you are the first married person who is a parent that has ever to me admitted that obsolescence and being buffered out of relevance in any meaningful way is full-blown possible in those domains. This makes you a wild anomaly. I think you're the first of many. I think you are an initial domino, my friend. But I would love to say like there's people listening in that are like, yeah, yeah, yeah, work, sure. My kids, never, never the fulfillment I get from them and what they get from me, that's never going away until our species is gone. That is not right. You can hear the cope. You can hear the white-knuckled grip of the attempt to find something permanent in the Lucretian clashing of void and matter and destruction of everything. You can hear the knuckles clicking. But like what for you are the factors that lead you away from thinking these are eternal permanent things?

Yes. Yes. Yes. But, uh, if you care to stop and think and be like, oh, maybe I can go on vacation, I can just like leave Robo Data or whatever. Like, uh, you know, you'll just realize like, yeah, the kids might even have a better time without you there. Or if you're there, you're just kind of like being puppeted by using Robo. Yeah. Yeah. Yeah. Like you, you, you, the way you best show up as a parent is to have, is to be conjured by, is to be prompted by a system that in real time is drinking in the biofeedback of your children, is taking into account the things that they've learned and their own goals and objectives and all that other stuff, right? Like this is what you're getting at without like advanced sensors and biofeedback and stuff, like which I agree is totally probably going to happen.

It tastes more delicious every day, brother.

Yeah. Well, we're going to get into like the societal impacts too. I do want to get into like bigger picture what happens here, goods, bads, uglies about this, but on the point you just brought up around sort of super dad, Robo Dad, whatever, I think I'll kind of refine what I had said there. I don't champion you for looking squarely at what is bleak. I think I champion you in being honest about what we're going to deal with because I think there is a way to say, yeah, yeah, yeah, some loser in his mom's basement will fall in love with some AI. My wife Stephanie, though, she actually loves me for me, okay? It's not about like, you know, I'm humorous or I'm this or I'm that. Like, we have a connection that's not just I can tell better jokes than a [__] robot. Like, it's way deeper than that. And my supposition is I actually don't think that's true. My supposition is there's a certain number of reward circuits Stephanie has. Some of them could be physical provisioning, could be how much you make, but there's a certain number of reward circuits, right? There's like, you know, there's like, you know, humor, there's sort of like intellectual engagement, there's like sexual satisfaction, there's maybe exploring certain hobbies and commonalities, there's, and all these are of different importance to different people, right? And they're all hyper-calibrated. Machines are going to be able to hyper-personalize experiences, including relationships, way better than humans can. Amazon already knows what I want to read more than I do. Spotify what I want to listen to more than I do. This is obviously coming. So my supposition is like when enough of those reward circuits, when it's like five out of seven of the reward circuits that made Stephanie say yes to a marriage, when like five of those are, let's call it 95 or more percent of the time better satisfied by the machine than the person, she may not marry the machine, but may be perfectly, no joke, perfectly happy just living in existence with a growth in that bond which is deepening and richening in all kinds of interesting ways that are super customized to her than with a human man. And that there are many people who are married today who will live to see that drift occur, assuming the societal collapse and the being eaten by AGI doesn't happen first. If it does not, I suspect that that atomization is essentially inevitable for the reasons I just laid out. Do you see the mechanisms operating the same way? Do you see them operating in a different way? Like how does your vision differ from what I've articulated?

Well, I like this idea of pre-commitment. We could talk about this a little bit. My suspicion is a lot of this stuff is very green eggs and ham. So if you're familiar with the Dr. Seuss story, you have children, right? So I will not eat it in a box, I will not eat it with a fox, right? But then you eat that [__], don't you? Right? And then that [__] ain't so bad, is it? Right? So it's like, you know, you throw the burkas on everybody, but like, you know, the faithful husband guy might go home and watch some very obscure adult content. That's strange, right? But like, so the same guy might have all the virtual burkas on all the women or whatever and make them all sound like werewolves or something through some VR lens so that they're all unattractive to him except for his wife. But it's sort of like, you know, you're just curious once you're like, well, I don't know, Mariah Carey 1998. I mean, that was like a really [__] good time. Like that was a prime year. Very formative. I'm not saying this about me, David. Bear in mind, right? I'm saying this is somebody else. I would never think that. So this is not, this is, I hope you're not thinking that I was thinking because I'm not, okay, David. I'm telling you right now. I'm really not. I'm using as an example a friend person I don't know might say something like that, right? Well, would you know, yeah, like a hot tub one time, like I don't know, maybe that would be right. And like green eggs and ham, my good sir, now it's like you will eat that in a box, my dude, you will eat it with a fox, my dude, as soon as you have it. And my supposition is how does that wave stop? To your point, it's unstable. Whatever your pre-commitments are, incredibly unstable, right?

Incredible, incredible. I mean, not surprising in even the slightest, like not even in the slightest of the slightest is that surprising. And but the splendid thing about you is that you're willing to admit that, like I almost can't believe that a married guy, because I've never talked to anyone who's married who hasn't said no, no, no, like brother, you've seen all my Twitter threads, we've been in touch for a long ass time over there, right? Like no, like Trisha, she actually loves me for me, it's not the fulfillment of some circuits. But I think if you carry that to your children, it also, people don't stretch it, no, no, no, no, my blood, I'll always love them, yada yada. I have two big oppositions there. I would love yours because again the parents thing is so uncouth. The fact that you have children, you're willing to talk frankly about what we need to deal with as a species is unbearably rare. And I hope just in listening to this more people like you will emerge. But with that said, like the two things about that. Number one, like is there not a scenario where there is either something to nurture or care for or spend time with that could be conjured to be minute-by-minute astronomically more fulfilling than your children, where let's just say it learns from you in a more ardent way. Maybe it doesn't have those traits you really wish it didn't have, like, you know, crying fits or like really embarrassing traits, right? Like stuff that embarrasses you in public like all the time. There's plenty of parents who like they're 75 and their child has been nothing but a net detriment to them financially, embarrassment wise, etc. for their whole life. But David, they can't admit that, can they, David? They have to say it was the greatest choice they ever made in their life, don't they, David? So is there no other thing to be interacted with where the circuitry of parenthood would be like just otherworldly more fulfilling, where like, you know, old people live longer when they care for a cat, you know, sort of is there that kind of like a pet or like a mentee or whatever that isn't just like sparklingly responsive in ways that are like way more fulfilling than children. That's number one. Number two, is it the case? Is it actually the case that like blood ought be the ultimate moral barometer for where we allocate time or attention for eternity? I'm not saying it's not, and I'm not saying it's irrelevant. I'm just saying is it actually the case that morally it is the highest ought that we have. Is it actually? So what are your thoughts on those two? Because again most parents listening are saying I'm...

different. I love in a different way and my children love me in a different way. Where's your take here?

Before we move on, I just want to talk. I've noticed I found it hard to talk honestly in that last segment and I think I was probably afraid of offending listeners. And so I think one reason it's really hard to talk honestly about this stuff is if I say I'm scared of the future because I'm not going to be able to compete, that sounds really weak and humiliating, right? It's like I suck. And then if I say to someone else, and you should be worried because you're not going to be able to compete either, that sounds like an insulting attack and I'm trying to get everyone to ostracize you. And same with it's like I'm scared of the future because I might cheat on my wife with like some sex bot. It's like that again, it sounds like this guy thinks he sucks and we shouldn't trust him.

And then if I say to someone else like I think you're going to cheat on your wife, and then it sounds like just like I'm, you know, think you're a terrible person. I'm trying to say no, no, no, no, no. It's like even like Gandhi would do this. And the point is that's one of the reasons I think why we don't, we have a hard time talking about this because it just sounds like we're attacking and insulting everybody.

So you're touching on a really important point to address what we're saying. To your point, when you say it about yourself, everybody could say, "Haha, yeah, because that's because you're some loser, right?" And then when you say it to others, they feel like, "What are you saying? Stephanie doesn't love me, right?" So, but for me, as you know, so number one, I mean, I think people super easily offended probably are not tuned into the trajectory. Certainly, in my opinion, shouldn't be. But the purpose here is to walk people through towards acceptance in many regards. I think that is indeed why people don't want to deal with it. I would say another reason why these elements, the personal super uncool things that you're talking about, which I think are crucial, isn't talked about is because the cognitive dissonance is way too much, right? So people can look into the future and they can see a world where their legal job is just not that important. That's actually probably psychologically they're going to hold together with that. If they look forward and like their wife has a chatbot that's just a better everything than you are, like that's way too much, right? Or like you yourself, who you now identify, many people have children, I'm sure with you this occurred, and by the way I'm not calling this bad, I'm calling this a human experience, they will have children and say this is like a fundamental part of who I am, period. Like when I wake up in the morning, more than like the dollars I make that day, it is literally the growth, the fulfillment of like these entities I've created, whatever, right? It's such a deep identity thing. So to think like that I would just care for some, you know, opposite tree branch hit like what you talked about, right? Is so jarring as to be indigestible. And I actually consider that existential struggle to be even more primary than the weakness admission or accusation that you and I talked about. What do you think of that?

Draw specifically the analogy. So how do people talk about temptation and how would you like to see that apply?

Oh, so that I do like. I think the addition, the added cope, which I would disagree with, is that there is a clear right thing. And of course, of course this I think takes us away from the project of truth here. But I think the admission of like we all should be vigilant to this I think is crucial.

Oh, absolutely. Right. Yeah. And again, yeah, the temptation thing is too, right? It's like whatever your values are today, if you know you don't leave it up to yourself at like, you know, 1:00 in the morning to like make the plan or whatever, or decide whether you like want to save money or spend it on like gambling or whatever.

You just brought up proactively the idea of like why this stuff isn't talked about and we've addressed that now for some of these uncool kind of changes in personal life. And I think you and I are very much of the opinion of like if any of these values are to survive in a way that any individual human wants them to, they would have to fight for those values in the reality they're going to deal with. Like the risks of this stuff also is not talked about. So, you know, you come from, I mean, a big AI lab. You know, you're pretty close with some of the godfather folks of creating this technology itself. You've been involved here for quite some time. Part of the reason that your ideas are so interesting and I think from my opinion rather credible, looking under the hood within the big labs, I know you've said things like, okay, where does this, like let's say we get this right, like what would happen? And then what you're saying is people are like, huh, and then that's it, right? And these are the people that are about to birth the sand god, but they're just like, huh, like what are the forces primarily squashing any expression of that? Is it fear the boss is going to get mad? Is it an inability to stare at the void as a person like we talked about with the loving of children and things like that? Is it some economic motive? Like why do the people closest to this problem who get it, who get it that our own obsolescence and attenuation seems rather imminent here, right? You're clearly of that opinion. You've been under the hood, you know, within Anthropic. Like why is nobody just being like, yeah guys, like all the paths kind of buffer us out, like why does no one say that?

Please. I would love. So the pilgrimage sounds fascinating and I think this should be part of the story. So lay it on us.

Yeah. Yeah. And it's like think of like, you know, people running an industrial lab especially because now you're basically betraying all your reports by saying this because now if like leadership wants to like cut your department or whatever, you can't just like full-throatedly say like, no, this is important stuff, we need to do it. You have to say like, they can say like, but what about this thing where you said maybe this is like terrible. So and you know, so if you don't visibly believe in the mission, you're just being a bad leader even if you're maybe being a good person or whatever. And you know, like Geoff Hinton did his 180 and was in a position where he retired at the same time and I think like I, you know, full credit to him for like full-throatedly like just Mia, like wonderful, wonderful.

Well, but I guess it's the other thing is that that's what these people know how to do. Like I know how to do AI research and so if you tell me like there's a problem where that's not the solution, I'm kind of like, do you want me to like be the caterer? Like I just...

Yeah. Like what was your genuine impression and again your frankness and being able to reflect on this and your pilgrimage including your time working with the big labs and very closely with one of the great researchers of this entire field, I think really matters here. What is your supposition of, let's just talk about, so the lab leaders are their own dynamic and I am going to actually give you a supposition of what I think are the barring factors from AI risk being truthfully spoken. And again I don't blame them either, I see it as purely an incentives game. So I've actually never been able to blame anybody for doing what they believe to be in their own self-interest. All that I can do is hope that we can collectively understand some of the detriments of self-interest and build structures to bound self-interest. And to me that's all we want to say something about that.

Right. Totally. Well, also here's the way I see it. I'm kind of like precursoring a little bit, but like what are the choices? So, here's the choice as far as I can see them. This is especially for the lab leaders, but even the employees, and we're about to talk about the employees, who I do not morally blame, by the way. Now I do hope that some of them flip. But I don't necessarily wake up and say they are evil, I am good. I see way too much of that. Like it's, you know, the lonely loser will paint his celibacy as virtue. The poor and unindustrious person will paint their poverty as virtue. The person who has not earned their way into a job that's that important will paint the people in those jobs as evil or whatever. I don't actually believe number one that it scales, number two that it's even right. I actually think it's all self-interest. But the idea that I see is like, okay, let's say I work at OpenAI. I'm some kind of like a capabilities guy that's focused on, you know, how to make AI multimodal, yada yada, whatever the deal is, whatever the flavorful terms are of like this six-month period we're living in. Like very complex job. I'm an extremely smart person. I got kind of two choices. So, choice one is I leave OpenAI knowing that like any win here buffers humanity out of existence and maybe I want to live a little bit longer or maybe I want to think more carefully about what we're conjuring. What are the qualities of a worthy successor for example? What are we turning into? Is there a merger scenario we can explore? Maybe we should explore that before we birth AGI. Maybe I think these things. So I leave OpenAI and I start thinking about this stuff. Maybe giving some talks and focusing on some other academic fields or whatever that I think are valuable. And then someone either at OpenAI or at Anthropic or at DeepMind or in China births the sand god. So they birth it. They get a final crescendo of glory of what they've created potentially seeding the universe and then probably it eats all of us. But my final moments are looking at the glory of another person who's higher than I and being consumed by their creation. That's option one. Okay, here's option two. I'll give you option two. Option two is you stay at the lab you're in and you see a little bit around the corner of eternity before the lights go out. But like at least you've built that thing that's going around the corner of eternity, even if it eats you and your children at least. Now, if those are your only two options, and there are some people who say, well, that's not their only two options, they could stop and revolt against it. In my opinion, until there's international concord around this stuff, that's not actually an option, right? I think I do hope more Daniel Kokotajlos and Hintons and others come out of the woodwork, right? But I don't think we can expect it of them. If that's their only two choices, what's the deal? Even if they know what's going to happen, they see our attenuation, like what's my choice? Let me at least see a bit of destiny and maybe I'll ride at that crest of humanity in those final moments as opposed to riding under the crest of a greater man in my final hour, right? Maybe if those are my only two choices, I shut up about the risk stuff and I just cook, baby, and I look around that corner before my lights go out, right? Like, am I right or wrong?

Yes.

Yeah. And I mean when I was a kid, I remember reading about the Manhattan Project and just being like, oh my god, that sounds so cool. Like I really hope someday I get to be part of something like that. I knew like there was one guy famously, I forget his name, who joined the Manhattan Project. Then it turned out it became clear that like, you know, the war in Europe was going to be won and eventually the war in Japan was going to lose too. He just left. He was like, well, what are we doing here? And then he just like went home. But there was like one guy, right? A single dude, right?

When you look, when you think about the people you talk to deeply, you talk to folks at OpenAI, you talk to some of the highest level researchers that have ever lived in this field, you know, very close to leadership, you know, within Anthropic, you know, boots on the ground researchers, people kind of in leadership, like average of them, like I imagine a pie, you know, 100% pie here, 20% nothing bad has ever happened in their life and so they just assume nothing ever will because like it's been pretty good. And then maybe 40% they know but they want to see around the corner because like, god, what am I going to do? The other 20, now I made up those pie slices, what do you see it as if we went under the hood, you took your best guess? All the biggest labs, maybe only in the West you can speak to, but I presume it'd be similar in China. What are the slices? What is their reason for being zipped? Is it like I don't believe it's risky, I think we can control it forever? Like is that 20%? Like these are rough, but just break this out.

I mean to like, you know, to Musk's credit, he talks it real loud. To Musk's credit, right?

Sure. Sure. But we were talking about whether they're clear by course. Yeah. They explicitly say there's a whatever 20-something percent chance to go, which is like I still think burying the lead in some sense. I also think they're being dishonest about the ratios. I think if Musk really thought about like, you know, he's at the world government summit or whatever it is recently in front of the Arabic event there and like, yeah, you know, like 99.99999% of intelligence is going to be non-biological. We just got to hope that they're nice to us. Like squaring that with like 20% chance of death is in my opinion like it's the management of perceptions to the people you're leading, right? So it's like if I tell my team, guys, this crushes us, but we're the final human victors, but like guys, the only other option is to be crushed over by someone else's wave. We got to plow forward. All your kids are definitely gonzo. Like Musk can't be honest. So the 20% even to me is an admission as much as it needs to be, but it is obfuscating a much higher ratio which any honest person would be able to say in my personal opinion.

Yes. Yes. Yes. Let's go down one level. So the leaders, I know they get it. You know they get it. We go down one level. Go ahead. Let's break out the pie. Break out.

Is it, I'm going to throw a couple options out. So the next rung down is full of those kinds of mental responses. And maybe there's a few. What's the percent who's like, dude, I get it, but I'm going to see around the corner before I get eviscerated because otherwise someone else is going to eviscerate me. What are you going to blame me for it? What's the percentage that'll say is 1%?

Like, well, I don't even mean it in a vicious way. I just mean like, brother, I know this is going to eat me, but what am I going to do? Leave and have somebody else's thing eat me? Like, well, let's say it in a non-malicious way. I'm not coming at this from a misanthropic angle. I'm just looking at incentives. Where could I allocate my time? And I'm not being dark. I'm not being Machiavellian. I'm literally just making a call that any living human would make. Like, has anyone, like what's the pie slice percentage? We're going to talk about that bigger group that has the beliefs you just talked about. But within that bigger pie slice, like is it only two or 5% that's like, bro, if I leave, like it's still going to get built. Like I might as well see a little bit of the future. Like I don't know what to do, you know? It's all I got. Like how many people have been honest about that?

Oh god. But again, it's like I think there's just so many, there's a rational, like what I end up hearing is yes, these safety issues, these governance issues, these things people should be thinking about that. It's a serious issue. We need to look into that. It's not necessarily my expertise. So like please you or someone else like work on this while... So part two things dawn on me. One, so like when Machiavelli wrote The Prince, people treated it as if he invented those ideas. When actually, David, like yourself, he made his bag, he was not in the fray and he just said the stuff that's obvious that no one says. But people treat him to this day as if he is the inventor and the conjurer of self-interested action. And I think those people are idiots. And so my suspicion is your honest man who you speak of was simply in a rare blip of frankness saying what many other people think but don't say. But it sounds like in your opinion an even larger percentage are mostly just not thinking that far out and saying no, no, no, I don't think it'll go wrong and they're just kind of plowing ahead and going forward. Is this, would you guess mostly that is because a, they simply don't have an interest, and you talk about an interest in the far future, it's like dude, AGI in like a year and a half is not the far future in my opinion. Okay. So just let me just throw that one on the table for you. But number two, like you, so maybe they just don't have an interest in the far, maybe they can't, like they really can't extrapolate trends. They can work on today's problem and they can't extrapolate trends. I know a lot of people like this, absolute rocket science genius folks who really just cannot see the 20-year future being that much different than today. They just refuse to let it happen. I estimate most of that is cognitive dissonance. In other words, to look at it squarely would involve reorganizing so many of your values and so much of your life that actually not going into that closet is better than having everything fall out of it because you're going to have to clean up for like, I mean you're never going to get done cleaning up. You're going to die before you're done cleaning up. And so, do you think it's mostly cognitive dissonance, mostly a childish disinterest in even one year forward? Like, what is it that keeps most people at that band you talked about? Again, I'm not insulting anyone. I'm just saying what keeps them there.

Well, exactly. Yeah. Yeah. So, I guess I was just trying to say like these are normal people who I think are just sort of like correctly not thinking about the things that they don't have an answer for and would make it clear that like if everyone else could agree to not build AI, they probably would. So, this person is like defecting in that sense.

Yeah. Well, I guess I'll say there's been a few people writing vignettes about these possibilities. It's really hard to predict the future, but some of the more plausible things that I've seen people like Rudolf Laine talk about is also Sam Hammond. Well, so like obviously white collar work is the most vulnerable and like, you know, our kind of jobs that can be done on a computer. Yeah. And of course right now people doing white collar work are also amongst the most influential. So they will be able to try, like that will be the window in which something can probably be done because once all the journalists are automated and anyone who's like an influential thinker is probably like a machine or somehow like locked into some incentive rail to grow, then there's going to be a whole bunch of unemployed people who can see what's happening and no one's going to care what they think basically. But yeah along the way I guess I would say yeah there's going to be a lot of people fighting for sinecures. I mean just think of the dock workers. They're saying yes you can automate us and that would be better but we don't want that. So we're going to demand that you either employ us inefficiently or give us a big one-time payout. But the thing is you have to negotiate the payout before you get automated or replaced so otherwise it's too late. And then I think there's going to be in terms of like reactions, there's going to be two main factions that are going to grow. And I don't have strong opinions about which one's going to grow faster, but one is the like Extinction Rebellion, Butlerian Jihad people who I mean will maybe worry that we'll be driven extinct or like equivalent.