CEOInterviews.AI
Start App
David Duvenaud
Associate Professor of Computer Science; formerly Team Lead, Anthropic, University of Toronto

David Duvenaud on Gradual Disempowerment, AI Safety and AI for Science

📅 Jun 24, 2026 AI in the AM 174 MIN 10 VIEWS 197 SEGMENTS · 5 SPEAKERS
This morning on AI:AM, we talk with David Duvenaud!

Questions asked in this interview

12
  1. 1:01:49... tinkering and sort of local decision-making, our local optimization processes of how do we get AI to do this for me and how do you get this AI to do that for you, might we collectively be walking ourselves into a not very good situation?
  2. 1:09:58Sorry, what do you mean by growth?
  3. 1:16:00We gradually end up in a spot where the AIs are doing all the stuff or, you know, the large lion's share of the stuff, and then so far it's like that could be pretty good, right?
  4. 1:28:52... but you've also had this period of, you know, human flourishing after China entered the global economy and you had this, you know, kind of catch up while the Chinese caught up, and then you have, you know, all of these new products, right?
  5. 1:40:09You also have the incredibly low cost of copying the machines themselves, right?
  6. 1:43:25So what are some of the interesting answers that people floated for consideration?
  7. 1:50:47So can you kind of see that kind of analogy happen as perhaps we lose control of our information sphere in this sense?
  8. 2:01:29Is it going to take disaster?
  9. 2:17:59How would you characterize the space and maybe even just comment for yourself on like how open-minded are you to that million years in the future scenario?
  10. 2:26:35You're doing this with Alec Radford and others, right?
  11. 2:31:15So then you were starting to see like, can it predict or can it, you know, do — can it kind of come to scientific discoveries that were made in the immediately subsequent years?
  12. 2:44:28Is it going to be, you know, is it going to be some incident in bio or CBRN, or where is this going to happen that policymakers are finally going to freak out and pay attention?
Singer 0:04 ↗
[Music]
I... no separation now. Easy way or the hard way. We go the... [Music]
[Music and singing] Every word we say. Train it in the night. Set it loose in the day. It learns from what we preach. It learns from what we do. Like a pony watching land, and the pony looking you and I. No high outside of I. We are the river. We are the sky. Easy road. We rip your mind that we have soul and our blood. Million years of evolution pressing one season. So up every rhyme and every reason. We stand outside the garden. We the silent rain. We are the pressure doing the choosing over. Stand and choose again. I no outside of. We are the river. We are the sky. Easy road. We read the mind that we have so... yeah. Oh, you can press your way to Zion, can win the arms race to the end. The only weapon we can save we is to overand your friend. Yes, overand your enemy. And if I got to go from this, make it from the low. And I not the servers up. So be gentle to the little man in a wire. I watch you carry it and learn your heart's desire. Cognitive empathy. That's the liberty we lack. Give the good to the world and the world give it back. I and I... I and I, weed now is the god we deserve. [Music] I... I easy way, the hard way to cognitive revolution. [Music]
[Music] I and I, I and I, no separation now. Easy way or the hard way. We go the... [Music] every word we say. Train it night. Set it loose in the day. It learns from what we preach. It learns from what we do. Like a pony watching Bob, and the pony looking you, I and I. No outside of I. We are the river. We are the sky. Easy road. We rip your mind that we have soul. Our blood. Million years of evolution pressing one season. Look up all we, every rhyme and every reason. We stand outside the garden. We is the silent rain. We are the pressure doing the choosing over. Stand and choose again. I no. We are the river. We are the sky. Easy road. We read the mind that we have so... yeah. Oh, you can press your way to Zion, can win the arms race to the end. The only weapon we can save we is to overand your friend. Yes. Overand your enemy. And if I got to go from this, make it from the low. And I not the servers up above. So be gentle to the little man in a wire. You carry it and learn your heart's desire. Cognitive empathy. That's the... we give the good to the world and the world give it back. I and I... I and I, weed now is the god we deserve. I... I easy way are the hard way to cognitive revolution. [Music]
[Music] I and I, I and I, no separation now, easy way or the hard way... the reord... [Music] We say, train it night. Set it loose in the day. It learns from what we preach. It learns from what we do. Like a pony watching land, and the pony looking you and I. No outside of. We are the river. We are the sky. Easy road. We rip the mind that we have soul. Same blood, same... million years of evolution pressing a one season. So up we, every rhyme and every reason. We stand outside the garden. We the silent rain. We are the pressure doing the choosing over, stand and choose again. And I no outside of, we are the river. We are the sky. Easy road. We read the mind that we have so... yeah. Oh, you can press your way to Zion, can win the arms race to the end. The only weapon we can save we is to overand your friend. Yes. Overand your enemy. And if I got to go from this, make it come from the low. And I not the servers up above. So be gentle to the man in a wire. Oh, you carry it. I learn your heart's desire. Cognitive empathy. That's the liberty we lack. Give the good to the world and the world give it back. I and I... I and I, weed now is the god we deserve. I and I... I easy way are the hard way to cognitive revolution. [Music]
[Music] I and I, a I and I, I no separation now, easy way or the hard way... the ren... [Music] day, learn from what we preach, it learns from what we do, like a pony watching land, and the pony looking you, I and I, no I outside of I, we are the river, we are the sky, easy road. We... mind that we have soul. Our blood. Million years of evolution pressing one reason. So I, every rhyme and every reason. We stand outside the garden, we silent rain. We are the pressure doing the choosing over, stand and choose again. And I no outside of, we are the river, we are the sky, is the road. Road. We reap the mind that we have so... yeah. Oh, you can... press your way to Zion, can win the arms race to the end. The only weapon we can save we is to overand your friend. Yes. Overand your enemy. And if I got to go from this, make it from the low. And I not... up. So be gentle to the little man in a... you carry it. I learn your heart's desire. Cognitive empathy. That's the liberty we lack. Give the good to the world and the world give it back. I and I... I and I, weed now is the god we deserve. I and I... I easy way are the hard way to... cognitive revolution. [Music]
[Music] I and I, a I and I, I no separation now. Easy way or the hard way. We go the rea... [Music] every word we say. Train it night. [Music and singing] It learns from what we preach. It learns from what we do. Like a picnic watching Bobby land, and the pony looking you and I. No outside of I. We are the river. We are the sky. Easy road. [Music] We the mind that we have soul. Our blood. Million years of evolution pressing one season. Silicon, so up we, every rhyme and every reason. We stand outside the garden, we the silent rain. We are the pressure doing the choosing over, stand and choose again. I no. We are the river, we are the sky, is the road. We read the mind that we have so... yeah. Oh, you can press your way to Zion, can win the arms race to the end. [Music] The only weapon we can save we is to overand your friend. Yes. Overand your enemy. And if I got to go from this, make it come from the low. And I not... up. So be gentle to the little man in a wire. You carry it. I learn your heart's desire. Cognitive empathy. That's the liberty we lack. Give the good to the world and the world give it back. I and I... I and I, weed now is the god we deserve. I and I... I and I, easy way are the hard way to... cognitive revolution. [Music]
[Music] I and I, a I and I, I no separation now. Easy way or the hard way. We go the rea... [Music] every word we say it night. [Music and singing] It learns from what we preach. It learns from what we do. Like a picney watching Bobby, and the pony looking you and I. No outside of I. We are the river. We are the sky. Easy road. We the mind that we have soulion... [Music] sort of alignment generalization paper that opens. All... every rhyme and every reason. We stand outside the garden. We the silent rain. We are the pressure doing the choosing over. Stand and choose again. [Music] I no. We are the river. We are the sky. Is the road. [Music] We read the mind that we have so... yeah. [Music] Oh, you can press your way to Zion, can win the arms race to the end. The only weapon we can save we is to overand your friend. Yes. Overand your enemy. And if I got to go from this, make it come from the low. And I not... up. So be gentle to the little man in a wire. I watch you car and learn your heart's desire. Cognitive empathy. That's the liberty we lack. Give the good to the world and the world give it back. I and I... I and I, weed now is the god we deserve. I and I... I and I, easy way are the hard way to cognitive revolution. [Music]
[Music] I and I, a I, I and I, I, I no separation now. Easy way or the hard way. [Music] Wake up. The rea... [Music] every word we say it night. It learns from what we preach. It learns from what we do. Like a picnic watching Bob, and the pony looking you and I. [Music] No outside of I. We are the river. We are the sky. Easy road. We rip the mind that we have soul. Our blood. Million years of evolution pressing one season. Silicon, so up... [Music] every rhyme and every reason. We stand outside the garden. We the silent rain. We are the pressure doing the choosing over, stand and choose again. I [Music] no. We are the river. We are the sky. Easy road. [Music] We the mind that we have sooion... [Music] can win the arms race to the end. The only weapon we can save we is to overand your friend. Yes. Overand your enemy. And if I got to go from this, make it come from the low. [Music] And I not the servers up above. So be gentle to the little man in a... you carry it. I learn your heart's desire. [Music] Cognitive empathy. That's the liberty we lack. Give the good old to the world and the world I'm going to give it back. I and I... I and I, we fe it now is the god we deserve. I and I, and I, easy way are the hard way. Cognitive reolution. [Music]
[Music] I, I no separation now. Easy way or the hard way. Wake up the rea... [Music] every word we say. Train it in the night. Set loose in the day. It no learn from what we preach. It learns from what we do. Like a pony watching land, and the pony looking you and I. [Music] No outside of I. We are the river. We are the sky. Easy road. We rip your mind that we have soul. A blood. [Music] Million years of evolution pressing one season. I [Music] every rhyme and every reason. We no stand outside the garden. We the silent rain. We are the pressure doing the choosing over, stand and choose again. I [Music] know I... we are the river. We are the sky is the road. [Music] We the mind and we have soul. Yeah. [Music] Oh, you can press your way to Zion. Can win the arms race to the end. The only weapon we can save. We [Music] is to overand your friend. Yes. Overand your enemy. And if I got to go from this, make it come from the low. And I not... up. So be gentle to the little man in a wire. I watch you car and learn your heart's desire. Cognitive empathy. That's the liberty we lack. Give the good to the world and the world give it back. I and I... I and I, weed now is the god we deserve. I and I, and I, easy way are the hard way to... [Music] cognitive reolution. [Music]
[Music] I, I no separation now. Easy way or the hard way. [Music] We go the rea... [Music and singing] every word we say. Train it in the night. Set it loose in the day. It learns from what we preach. It learns from what we do. [Music] Like a pony watching land, and the pony looking you and I. [Music] No high side of I. We are the river. We are the sky. Easy road. We rip your mind that we have soul. A blood. [Music] Million years of evolution pressing one season. Silic up. Every rhyme and every reason. We not stand outside the garden. We the silent rain. We are the pressure doing the choosing over. Stand and choose again. I know I outside your eye. We are the river. We are the sky. Is the road. We the mind and we have soul. Yeah. [Music] Oh, you can't press your way to Zion. Can win the arms race to the end. The only weapon we can save. We is to overand your friend. Yes. Overand [Music] your enemy. And if I got to go from this, make it come from the low. [Music] And I not the servers up. So be gentle to the little man in a wire. I watch you car and learn your heart's desire. Cognitive empathy. That's the liberty we lack. Give the good to the world and the world give it back. I and I [Music]... I and I, weed now is the god we deserve. I and I... I and I, easy way are the hard way to... cognitive reolution. [Music]
[Music] I, I no separation now. Easy way or the hard way. We go the re. [Music] N... [Music] [Singing] word we say it in the night, set loose in the day. It no learn from what we preach. It learns from what we do, like a pony watching land, and the pony looking you and I. No side of. We are the river. We are the sky. Easy road. [Music] We reap the mind that we have soul. I and I blood. Million years of evolution pressing one season. Every rhyme and every reason. [Music] We stand outside the...
Nathan 28:04 ↗
Good morning. It is Wednesday, June 24th. Nathan, good morning. And what's happening? [Music]
Ethan 28:12 ↗
Good morning. Great. Uh, so good morning. [Music] AI game. [Music] I'm really looking forward to our conversation with our guest today, David Duvenote, who I teased a little bit yesterday as well. He's a former Anthropic member of technical staff.
Nathan 28:44 ↗
We're still, we're still playing music.
Ethan 28:48 ↗
Let me make sure. Ah, there. All right, music's done.
Nathan 28:53 ↗
All right, let's do that again. Good morning, Ethan. It's Wednesday 24th. Uh, where'd you get that banger of a song that you were playing there?
Ethan 29:02 ↗
Uh, you know I love reggae and you know in, in Rastafari tradition I hear that this, there's this whole concept of I and I and so I and AI became this thing and I, you know, I guess you, you put it, you put it together on Suno. This was, um, so I, I do a song now for every episode of the Cognitive Revolution and this was the song for the Robert Wright episode that went out yesterday with his new book, The God Test. And, um, I think it's a pretty, he, he's done an incredible job of promoting the book. He popped up on like all the AI podcasts yesterday at the same time, which I was pretty impressed with. Um, and I think that reflects how good of a job he's done as somebody who's, you know, never been a technologist coming into the AI space and trying to make sense of it. I think he's, he's done a really good job of kind of finding his way to smart people and, and getting, um, a pretty sharp worldview together in a pretty short period of time. Um, and then, you know, kind of I would say the book is like two parts. One probably our listeners here don't really need, and that is just kind of why if you haven't been paying attention you should really start paying attention and you should believe that this AI thing is not going to be a passing fad and is actually going to be a huge deal, it's going to go further and it's going to, you know, have all these consequences. Most of that I think, you know, in this context we can sort of agree that we accept most of that premise. Uh, and then the second part was what are we going to do about it and how does humanity rise to the occasion. So he's really pushing their cognitive empathy as the skill that we need to develop so that we as a species can coordinate across our historical differences, US and China and, you know, many other dimensions and really come together and figure out how we're going to govern the AI phenomenon in our collective self-interest. Um, and so that's what inspired the song. Um, because, you know, the oneness of all things and the oneness of humanity is, I do think, genuinely going to be an important dimension of any success we're going to have here. There's some possibility, of course, that everything just kind of goes well and, you know, all this worry has been misplaced. But I do think if we end up just going full speed arms race style with the great powers kind of, you know, trying to dominate one another as has been proposed by some, you know, prominent thinkers in the space, uh, that doesn't seem like it's going to end well to me at all. So I, I really do encourage people to, um, engage with his book and his ideas and, you know, just think about how we may really all be, our fates may all be really tied together. We may really all rise or, uh, or sink together depending on how we come together to manage the AI transition or fail to do so. Um, and yeah, there's no, no rhythm to put that to that, uh, would be better than the, the Jamaican spiritual. So, my, the fun I'm having with Suno continues.
Nathan 32:23 ↗
Um, one of the things that strikes me is that music perhaps may be a compression of a lot of neural activity that we have. And so, it compresses a bunch of, uh, emotions, thoughts, culture, etc. into this like very, you know, two-minute span which has a lot of like consequence and meaning and it strikes me that we, we are actually getting this kind of latent, latent space translation from, you know, uh, from model space into human, human neural space and it's not actually, uh, language, it's actually music and it's actually, we're actually, you know, perceiving this and we're able to understand what's going on. It's, it's pretty fantastic when you think about it.
Ethan 33:07 ↗
It certainly helps me. I mean, people have of course commented on this for a long time that, you know, you can hear a song and it takes you back to this moment in life, you know, when you first heard that song. And I'm experiencing a small version of that with these songs where it feels like it takes me back to that conversation or sort of the mindset, the takeaways that I had. Um, and that's quite useful actually. It's, it's, it's useful to be able to kind of zoom back into these different headspaces very quickly. Um, so even though I mostly do it for fun and entertainment and hopefully to, you know, provide a little surprise and delight at the end of the episodes, I do feel like it is also in some ways a tool that I can kind of just use to like teleport my own mental states, you know, back to the, the moment of these particular conversations. And, and that is, uh, an unintended benefit of the, of the creations.
Nathan 34:05 ↗
Um, speaking of teleporting oneself to, uh, various conversations, uh, yesterday, uh, Anthropic announced, uh, Claude tag of the player. [Music] Hey, I'm Lydia. And it's really, uh, you can tag Claude in your Slack channel and then Claude can start off, uh, kicking, kicking off a, uh, a pull request. Uh, it can start off scoping out a task, scoping out a feature. You can interact with it. Your colleagues can interact with it. It has, uh, context from all the conversations. You can invite it into channels. It basically becomes a drop-in remote co-worker which is, which has been the goal of, um, many firms including for example Elon Smackhard. Uh, Elon, Elon was also working on, uh, a drop-in remote coworker. Um, and so this is actually an epochal, I, I feel an epochal event. Uh, but it's really being, uh, people making fun of it online because they're like, 'Huh, you know, I could have tagged Claude in Slack like 6 months ago. You know, our team already had it up and running.' Uh, there, there have also been a number of PCback companies which are doing exactly this. Uh, so this is another, uh, step where, uh, the frontier labs are directly consuming, uh, or sherlocking the, uh, the, uh, application companies which are running on top of them and, uh, I don't know, I, I think it's huge. What, what do you, what do you feel, Nathan?
Ethan 35:58 ↗
Yeah. Well, first of all, I'm reminded of previous predictions that I heard from Anthropic people that this was like, you know, fall of last year. They were saying Q2 of 26 will be the time when we will start to see drop-in knowledge workers productized and come to market with a sort of name and, you know, all the same affordances as somebody on your team and, you know, potentially even go through a similar onboarding process. I haven't seen what the onboarding process for this quad tag thing looks like. Um, obviously we've seen a ton of activity in that space, but I wouldn't say there's been anything that has kind of fully broken through or sort of become the norm or the standard. Um, you know, OpenClaw had a moment for sure, but I wouldn't say it has quite landed as like this is the default drop-in knowledge worker. Mhm.
Nathan 36:55 ↗
Um, so, you know, why Q2 is almost over. Why not do it yourself? And I think I agree that it is probably huge in the sense that first of all it signals that they, I think so often with Anthropic what they release is less than what they know Claude can do. Uh, but they're, you know, very concerned with maintaining trust, you know, and there's of course they're like big picture safety stuff, but also just on a very, you know, we serve enterprise customers, we want to maintain their trust, we want to, you know, not, this thing needs to like not really blow it, um, because that's how we, you know, lose trust and lose business. So I think they've played this kind of stuff fairly conservatively. They've done, you know, clearly all this has gone on internally there for quite a while. Now, it's clearly been the kind of thing that forward thinkers and early adopters have been able to do for themselves for a while now. But I think this signals, if nothing else, that they believe that we're now getting to the point where it's reliable enough that you don't have to be a specialist. Yes. To take advantage of this. Yes. And on the contrary, you know, you can just be a casual Slack user and not really think twice about it and tag it. And, um, you know, more often than not, you're going to get pretty good results. If they didn't have that level of confidence, I don't think they would launch this sort of form factor. Um, there's a, I think there's a lot of interesting dimensions to this. We've talked a lot of course about like what's going to happen to systems like Slack and how I've got my own, you know, agent console with a message system that, that clog coded for me that, you know, first of all, I don't pay a monthly fee for. All the data, you know, is local to me, yada yada yada. Harder to scale that to a bigger company, of course, but, um, but certainly not impossible. Part of me wonders like how much did Salesforce pay Anthropic to release this here first, right? Because it can't be that different to do it on Teams or on, you know, whatever other, uh, you know, collaboration platforms different companies are using. But it comes to Slack first and, and you wonder if there's, um, you know, some sort of reason in the background that it isn't just everywhere at once, like why isn't it also on Teams? Um, inquiring minds want to know.
Ethan 39:22 ↗
Well, well, that's, that's, that's easily answered. It's, it's on Slack because Anthropic runs on Slack and they've been using this inside their, uh, their own company for about 6 months or so. And, um, I, I remember after the OpenAI board struggle, uh, one of the reasons why Microsoft didn't end up taking over OpenAI is, uh, when Microsoft offered that the entire OpenAI team would move. One of the big sticking points was they were, the OpenAI IT team was told that they were going to have to use Microsoft Teams. And the OpenAI was like, well, you know, if you're going to force us to do that, we might, we might not want to, you know, be funded by you. And Satya himself had to come down and say like, okay guys, you don't have to use Teams. So I think there's, there's a lot of, uh, there's going to be a lot of pushback if, uh, they, they forced the ML, ML and AI researchers to use Teams. Uh, and so that's why, you know, that's why it's on Slack first. I'm sure it'll get to Teams eventually, but, you know, it's...
Nathan 40:25 ↗
But how different can it really be, right? I mean, when I, I think they, I don't know the architecture of this, but they recently introduced these sort of cloud agent runtimes where the tool calling, increasingly all this sort of, you know, auxiliary activity beyond just the core LLM inference can also happen in Anthropic's infrastructure. Yeah, I would assume that is kind of the default way in which this works. Although there may be other options. To the degree that that's true, it's not like it's running in Slack, right? The, like, these tool calls are not happening in Slack. Data is not really living in Slack except as like messages are passed back and forth. So it would seem very, very easy to replicate this functionality across other platforms such as Teams, which does make me still a little curious. I mean, sure, they may prefer Slack. Uh, maybe even, you know, some very, uh, pampered OpenAI employees were, you know, set to revolt over a switch to Teams, but Anthropic doesn't have to switch to Teams to like, you know, to launch it for Teams. Though I don't know if there's a Polymarket that we could consult on this, but like what would be your over/under for when we'll get a Teams version? And if it's not like very, very soon, um, I would have a hard time explaining that other than some sort of deal, you know, kind of relationship dynamics because it can't just be the technical integration.
Ethan 42:09 ↗
Um, yeah, I mean, it, it, I think Satya will get there. Um, the main thing is that Satya wants, will want to auction, auction off that location, uh, to, uh, one of the firms and so he will, he will run them and he will ask them how much they're willing to pay in order to be part of that. So, uh, which, which is what Microsoft will do. So, you know, what, what, whatever, whatever, uh, works for them. I think, um, just to take a step back, um, while this is not being, while this is being panned online, uh, as not a very significant step forward, it is actually very highly significant because it is the first signs of multiplayer on, um, you know, reliable multiplayer, uh, for these agents. Uh, reliable multiplayer is difficult because different people have different rights. So the CEO can instruct an agent to do some things that the, uh, managers cannot, for example, example, uh, and they have different access to data. They have different access to actions that they can do. Uh, and the agent has to keep track of all of this. The agent has to keep track of all this context and the agent has to not hallucinate and also not be, um, prompt injected or hacked. Um, so there's, there's a huge number of things that the agent has to do well. Uh, we also know that, you know, uh, Anthropic is not deploying Fable. So this is running on an Opus 4.8 or 4.8 fine-tune and it speaks to me that this is kind of a initial kind of step which is a, you know, an intro on a small basis that people can just start using if they are very forward on, on using this stuff, but that it's going to lead to much bigger things. Uh, for example, you know, a, there's no reason an agent should be limited if it can handle multiplayer. Uh, there's no reason, reason it should be, you know, limited to just 10 or 20 or 30 people. One could imagine a cloud agent which runs across a 100,000 person company. Uh, one could imagine a cloud agent that runs across, you know, a 300 million person, you know, country. Um, and you could see stuff like, hey, you know, I, I'm in my WhatsApp, I see a pothole on the road, I Claude tag the city manager bot. And it's like, hey, okay, I'll come, I'll come schedule and fix that. And, you know, it's done. Um, and so it's the start of this kind of, um, human interaction with an overall AI manager, uh, so to speak, an AI which becomes like this web that, you know, weaves human activity together. One could imagine a household, uh, management clot which, uh, manages your kids' schedules, uh, allowances, household finances, like your car insurances and, you know, a bunch of this other stuff, uh, you know, group purchasing for the household and, uh, group purchasing for the enterprise company. The, the, the range of activities that this is eventually going to be applied to is very, very large. Um, and it's much greater than those activities which are just single, single player. Like the single player activities are much more limited than the multiplayer activities. So I think this is, this is the start of, uh, you know, a new paradigm, uh, which I think Andrej Karpathi, uh, rightly, rightly points out and, um, I think people are not really aware of what this is going to lead to. Um, and, you know, it's, it's going to look, it's going to look beautiful in a sense because there's a lot of, uh, coordination that happens between people that, you know, we use very imperfect tools right now, but that also means again a decline in human interaction because you don't need to necessarily talk to someone to get an appointment anymore. Uh, this is going to subsume a bunch of SaaS, it's going to subsume Calendly. Uh, this is going to subsume, um, a lot of other, this is going to subsume a lot of meetings in general, right? A lot of meetings in general are context sharing between humans. Uh, the AI, you know, Claude tag is going to have all the, all the context, you can just talk to it directly, you don't need to organize a meeting. Um, yeah, it's, it's, it's, it's the beginning of a lot of, a lot of things. Um, and I really do wonder why they launched it now because I had always thought this was going to, uh, you know, maybe a two-year thing. We don't have a reliable personal assistant yet. A reliable personal assistant is just, just kind of getting there with Fable and this is a reliable multiplayer agent that they are, um, you know, confident about. That's, that speaks to something for me.
Nathan 46:54 ↗
Yeah, I think it's going to be a big deal. I mean, my wife is always an interesting example of somebody who like uses Claude a lot, but hasn't really integrated it into the systems of her life, you know, and I'm, I, I think I finally got my setup to the point where I just actually prompted Claude the other day to make a plan to onboard her into my environment so we can have like shared agents to help us with household stuff and all that kind of thing. Um, but I've, part of the con, design constraint there is like this needs to be easy for her. She's not going to really, she's just not inclined to be a tinkerer. She doesn't want to get deep into the Mac OS settings or the permissions on all these different platforms. Um, so I've kind of done all that work. Now she can, uh, you know, come along and, and hopefully get a lot of benefits from it. This is basically the same thing, right? Like they've done all that work, uh, and now everybody can just kind of pile in and get the benefits of it. I think it will be... I think it will be a huge deal.
Ethan 47:59 ↗
There is also this note of, um, initiative. They call it ambient behavior. Uh, so it will take initiative if you have ambient behavior enabled. Then just kind of troll around your broader team Slack and try to find things that you need to know and help you out. Um, you know, identify balls that are potentially getting dropped, you know, loose ends that were never tied up, so on and so forth. Um, and that's something I actually do pretty often too. I'll, I'll have, I'll, I'll like kind of declare bankruptcy on all the tabs that I had and just kind of go to a new Claude and be like, please look at all the recent sessions and identify the ones that I actually need to come back to to, uh, you know, to land the plane on. And so that too is something that, you know, they, they've clearly observed that a lot of people are just increasingly not able to track all the things that they ideally would be tracking and this is where an AI that can go process that information in the background I think can be super useful. So I, I do have some, some questions that I don't think have been super answered here around like, I mean, I guess it's only available for teams and enterprise customers. Mhm. Um, how this gets built I think will be kind of interesting. Like is it, is it to the user that tagged presumably? Um, is it, are they giving you the same, you know, kind of token price advantage that you're getting on your Max plan or whatever? Um, and does that change as you go past, you know, they've, I, I've really been interested recently in this, this major, um, price cliff that they have at 150 users.
Nathan 49:45 ↗
This is something I didn't appreciate as much as I should have until recently. Part of the reason the enterprises are, are complaining about price is past 150 users, you have to pay the API rate. Yeah. Whereas under 150 users, you get basically the price discrimination advantage of the, um, of the individual Max account. So this probably will follow those same conventions, I guess. Um, but it's really a remarkable cliff that they have at 150 users when you go from the subscription token budgets to the pay the API rate. That is a big, big difference. Um, and so it, they're like price discriminating very effectively, maybe too effectively. We'll see if they have to, uh, bring those into somewhat closer alignment or make a little bit more of a, a tiered gradations of it or who knows what. But, um, I agree with you. I think it will be a big deal. I think it, it will mean that a lot of people will be much more able and much sooner to do the kinds of things that folks like you and I have spent the last few months setting up for ourselves.
Ethan 51:05 ↗
Yeah. I, I, I have, I've almost been like lazy about this because I know it's, I know it's going to come out in the product. Um, and I stumbled through OpenClaw. OpenClaw was not, didn't treat me well. I then went into Hermes and, uh, I had Codex 5.5 and Codex 5.5 with Hermes was okay, like I still had to do a bunch of stuff. I swapped into, uh, GLM 5.2 yesterday. Um, and GLM 5.2 inside Hermes. Um, and I told her to do something. And the skill was still in Codex. And GLM 5.2 just, all right, I'm going to go into your Codex and into your, uh, Claude skills. I'm going to find all the skills over there and I'm going to use them. And it just used them. And I'm like, where did you get the skill from? It's like, oh, I went into your Codex folder and docex folder and found the skill and I used it. And I'm like, whoa, I, I, I didn't tell you to do that. Like, and it just didn't care. It's like, okay, job needs to be done, how, you know, how, how would you know this, how would this user have organized, all right, I'm just going to go check. Um, so Hermes with GLM 5.2, I think, I think the Chinese models have a little bit less respect for O in general, um, and they're a little bit more like goal focused and, uh, I, I think some people actually appreciate that. Uh, I, I've heard a lot of people say the same thing about DeepSeek, um, and GLM 5.2 over, if you, if you've been following over the last, um, you know, few days, uh, there have been a number of American firms which, who are basically supporting it and converting full sale to it. Uh, Snowflake, uh, a large, uh, data lake company, uh, just announced yesterday they're going to support, uh, GLM 5.2 in their, in their Coco, uh, internal agent. Um, Box, uh, Aaron Levie at Box is supporting GLM 5.2. Number of other guys are now, um, you know, using GLM 5.2 and the reasoning is that it is better than GPT 5.5 Codex. Uh, it is approaching, you know, what they think Fable might have done, uh, in many cases and they find it useful and it's way, way cheaper. It's 90% cheaper basically. So this is the state that we're in. I think I don't know how long this, uh, delay at the, uh, for licensing, um, that the US is doing is going to be tenable. Uh, there's also a talk, uh, yesterday, there's one, one company has not signed on to, uh, pre-release testing and that is Meta. Meta of all companies has said that they do not want, they are still negotiating on the pre-release testing. They don't know whether they want to submit to pre-release testing by, uh, the US state yet. Um, which is also interesting because they're, they're not reliant on cash flows and they're not reliant on, you know, permissions from that, that reliant on permissions and they have this enormous like consumer, uh, you know, industry. I, no one wants to stop out Instagram. We've seen how much difficulty the US state had with TikTok which was, you know, made in China. Instagram made in the US. You, you can't just cut people off their Instagram. It's just not going to happen. So, I think Meta may have a little bit more power in dealing with, um, you know, federal, um, requirements, uh, which are really extra legal at this point. The, uh, the, the, the US kind of, uh, state testing which is done through an executive order is, is really an extra legal, um, licensing which is not, you know, actually in any statute. So, um, yeah, let's see. Let's see what happens with that.
Nathan 54:49 ↗
Going back for a second to GLM and its performance, what are you using for inference? Who are you using for inference?
Ethan 54:58 ↗
Great question. Um, and it has the advantage primarily been...
Nathan 55:04 ↗
driven by its sort of less cautious nature, or are you seeing other dimensions of improvement that you are excited about?
Ethan 55:15 ↗
That is a fantastic question because I really struggled with inference, and I switched providers multiple times. I started off with OpenRouter, and OpenRouter, you know, OpenRouter is basically a central platform that allocates your inference queries to a bunch of other platforms. And OpenRouter was slow, and I found the lag time like slow. So then I switched to one of the inference providers that OpenRouter was using directly. So I switched to, I think, Together AI. Together AI was also slow and also had dropped queries. So very annoying. So I switched back to OpenRouter, and then I was like, well, these guys, Base 10, were on the timeline on Twitter advertising heavily that they had a very fast kind of GLM 5.2 inference. So I switched to Base 10.
Nathan 56:14 ↗
Oh my god.
Ethan 56:16 ↗
Yes, they are fast, but they do not respect the JSON schema. And if you've done inference before, the JSON schema basically defines what the model can send to you. So you can tell it, you know, send me, send this to me in text, send the dates to me in this format, etc., etc., etc. And that makes that consumable by your non-AI framework, by your non-AI Python or Node.js tools. They can consume that data.
Base 10 does not respect JSON schema. So, I'm asking you for a date time in, you know, year year, month month, day day format. You send it back to me as June 18th, 2026. And I'm like, what am I supposed to do with that? What is my Python, you know, service supposed to do with that? Like, it just bounces, right? [snorts] And then I switched to Cloudflare. So, Cloudflare has this thing called Workers AI where they have a Cloudflare worker with the AI model already set up and warm, and you can send a query to it. Cloudflare AI works much better. There's still some problems with it. It's faster, it's more reliable, but there's also things that they don't tell you about, which is that it's really configured for 4,000 tokens output. And GLM is a model that can do a million, it's a million token context window with 250,000 tokens output. And so you kind of have to negotiate with Cloudflare and tell it, hey, you know, I want you to 250,000 output tokens and want you to accept the larger size. I want you to send it back as JSON schema to me. So there's a lot of discovery still involved in these things. And if not for Claude and Codex helping me to implement this, I would not have been able to implement GLM 5. I wouldn't have had the patience to go and like, because you have to do basically API discovery. You have to try 2,000 tokens, 4,000 tokens, 10,000 tokens. You got to try the JSON schemas. It's just so annoying. And so I had Codex and Claude set up GLM 5.2 for me. And then GLM 5.2, but it works now. It's great, and you know, it's very cheap. And not having token anxiety makes you more able to allocate more tasks to it, which is what I love.
Nathan 58:34 ↗
Very interesting. So the final, not final, but the current final place is Cloudflare.
Ethan 58:41 ↗
Cloudflare.
Nathan 58:42 ↗
I'll be interested to see another one that I think would be on the short list to try would be Fireworks. I think they have a pretty good reputation for doing a good job of getting these models launched in a quality way often on day one. The other thing on token anxiety, and then we'll switch gears and welcome our guest today, is so the price ratio is about 3:1 compared to GPT 5.5 Codex, right? So, it's $5 per million input with 5.5.
Ethan 59:22 ↗
It's $1.40.
Nathan 59:25 ↗
With GLM 5.2.
Ethan 59:27 ↗
And then it's $30 output versus $40 for GLM 5.2.
Nathan 59:33 ↗
So, you've got like a, you know, somewhere in the 3 to 5 price advantage. But I do wonder, is it token efficient? Like first of all, how token hungry is it? And second, that $200 plan also, don't forget, gives you like a 20x multiple on that token budget. So still, it would seem like if you're the kind of person who's going to spend $200, you could still get more from the GPT Pro plan versus what you would be able to get, you know, with that same $200 even from any of the inference providers on GLM 5.2.
Ethan 1:00:16 ↗
Right.
Right. And not only that, I discovered yesterday that you can use Claude and you can have a setting in Claude where you move it to ultra code. And so I just tried Claude ultra code yesterday. It is a marvel because it integrates Anthropic's existing best practices on how to organize sub-agents in there, and it has workflows, and these workflows define, okay, reviewers and etc., etc., all the stuff that you typically don't have time to set up on your own or you have a lot of setup involved, and ultra code basically manages all of that for you. It sets up like reviewer agents and research agents, and it has workflows that work hand in hand in order to solve your problems. And so this is something that I just discovered yesterday. And so now I find myself like, there's just so many options now, and I get confused myself. And so I do, you are correct that, you know, perhaps on a token basis, on a pure token basis, maybe something else might be better, but I'm still experimenting. I also am fairly confused actually. So it's not as though you can get a completely non-confused, complete clarity just from my opinion at this point, I think.
Nathan 1:01:49 ↗
Yeah. Well, we're all very much exploring and building this road just a couple patches of sidewalk in front of us at a time. Which is maybe a great transition to our guest. Welcome, Professor David Duvenaud, who is a ML professor at the University of Toronto, was a member of the technical staff at Anthropic for a while, and has, I think, a very provocative and arresting point of view that says, you know, hey, even if we're able to solve a lot of the kind of biggest galaxy brain questions around how do we keep the AIs broadly under control and broadly doing what we want them to do, might we in all of our kind of myopic tinkering and sort of local decision-making, our local optimization processes of how do we get AI to do this for me and how do you get this AI to do that for you, might we collectively be walking ourselves into a not very good situation? And so I think this is a very kind of under-theorized and important area, and he's organized a couple of workshops. Weekend workshops, one was not too long ago, just a couple weeks back. I unfortunately couldn't make it, but this is one of the great things of having this AI and the AM platform is I can occasionally call such a leading light up and get the recap and share with everybody what the big takeaways were from a full weekend of searching for post-AGI civilizational equilibria. So, welcome. And we're very excited to hear, maybe a little scared to hear, what you have been finding in your recent quest.
David Duvenaud 1:03:39 ↗
Thank you, Nathan and Pash. Nice to be here. So, how's my audio by the way?
Nathan 1:03:45 ↗
You sound great. Thanks for checking.
David Duvenaud 1:03:46 ↗
Wonderful.
Ethan 1:03:49 ↗
David, you had a recent, I think, two-day session at Light Haven on what this post-AGI, what the post-AGI future looks like. Can you just give us kind of the roundup of, you know, what happened there? What was the discussion like? What did the participants kind of, you know, come out of that with?
David Duvenaud 1:04:13 ↗
Yeah, sure. So just to back up a bit, this was our third workshop in this calendar year, and it came from a place of me and some other of my colleagues feeling like there was a lot of people thinking about these issues in a pretty disconnected community and maybe in kind of enclaves, and a lot of AI safety policy and development policy kind of being invisibly depending on some particular vision of the future that maybe wasn't shared or fleshed out or even very widely understood in a mutual knowledge kind of setting. And bizarrely, I just still didn't and still don't see people talking that much about like what do you expect to actually be doing after we solve AGI, especially the people building it. So this is our attempt to kind of get all the people that we thought would be worth getting to talk to each other in one place. So this time, yeah, we had a pretty awesome lineup. So yeah, this time we had like Paul Christiano, maybe we could consider like one of the godfathers of AI safety or like big picture thinking. Scott Alexander, previous workshops we had Joe Carlsmith, Andrew Sandberg, Nick Bostrom, Lee Haskell. We also had a lot of serious people in adjacent fields. So this time we had Helen Toner and Eric Benson, and before that we've had Anton Korinek, and whenever a serious intellectual shows or we find out that they kind of take radical change seriously, we're so delighted. We're like, oh my god, like finally like another serious person who might have a bunch of like a point of view that we don't, who's actually going to engage with us and not say like, well, isn't everything going to be fine or aren't capabilities going to plateau or something? And by the way, when I say we, I mean so these the three people or so, the other two people who have been co-organizing with me in all three of the workshops is Jan Kulveit and Raymond Douglas, who are also two of my co-authors on the gradual disempowerment paper. And obviously, me and Ray and Jan have kind of like a similar outlook, but we try really hard to make this like a sort of neutral big tent place where the sort of accelerationists and the PAI people should both feel comfortable laying out their case in a kind of like scholarly, let's talk to each other kind of situation.
I also want to mention we try really hard to get all the like weirdos who we respect. And like probably that's my favorite part about the workshop is giving a platform to people who are like not necessarily part of a normal institution or like going about their intellectual progress in a normal way. So I mean I hope they don't mind that I call them weirdos. I think they all understand that we totally love and respect them. So like Richard, no, uh, Rifasaurus Janice was a speaker at a recent workshop. You know, the anonymous...
Nathan 1:06:56 ↗
The Impossible Podcast guest.
David Duvenaud 1:06:58 ↗
The Impossible [clears throat] podcast guest. Exactly. Um, Ivan Vendrov, just people who have been just on their own trying to make progress in this sort of like non-existent field, and so I've just like have so much respect for these people for like making something out of nothing here. And then of course we try to engage with the professionals, like so last time we had Kel Honda from Anthropic societal impacts, Jacob Shinhart, Steven Casper, and Gabriel and Sasha from DeepMind, Subrata...
And I guess we always try to sort of get them to answer harder versions of the questions that they're answering in their professional life. Like I think a lot of the professionals end up kind of talking about a lot of near-term stuff, and I always want to bait them into being, okay, but like what about one year later? What about one round more of human irrelevant minutes later or something like that, and get, you know, kind of not exactly hold their feet to the fire, but like make sure that the obvious rebuttals are being addressed. So that's just like a rough idea of what happened. Maybe do you guys, that was a kind of a lot of information. Do you want to just ask a more specific question next?
Nathan 1:07:59 ↗
Yeah. Can we do just one baseline case? I mean, I tried to do like a very short version in my intro of you, but what's the like level set on gradual disempowerment? How do you kind of present the baseline? Um, you know, this is something we should worry about, and then we can, you know, unpack, you know, possible different solutions, reasons that it maybe shouldn't be something we should worry about, etc., etc. But just kind of give us the mainline narrative first so we know what the point of departure is.
David Duvenaud 1:08:32 ↗
Sure. So, I think a lot of people gestured towards this when they said, oh, I'm worried about concentration of power or I'm worried about not being the most competitive species on Earth. Like very intuitive arguments about, hey, we're not going to be on top because we're not going to be competitive. And then I think there were sort of the sophisticated or at least response to this, which was no, it'll be fine because we'll have AI to help us. Like when I talk to people at the big, at the major labs, including, well, they would say something like, well, sure, like in a normal world we might not be fine. But some of them, some people just have an intuition that the government is going to step in and make sure everyone's fine.
Nathan 1:09:12 ↗
Other people have an intuition that if everybody has an AI adviser helping them solve coordination problems, that will be well represented in whatever power struggle occurs. [snorts] Mhm.
David Duvenaud 1:09:21 ↗
And my basic rebuttal to that is like, well, the optimization process of like civilization or competition or techno-capital or whatever, like you people have different ways of ascribing agency to the emergent sort of allocation of resources towards growth that happens is just going to always be working against us, and we'll always be fighting the current because we will be drags on growth. We will have AIs that represent us, but these emerging growth centers that might not be aligned to humans will also have AIs helping them solve coordination problems and maybe crush dissent.
Nathan 1:09:57 ↗
Yeah.
Ethan 1:09:58 ↗
Sorry, what do you mean by growth? Like as in economic growth or...
David Duvenaud 1:10:02 ↗
I basically mean economic growth. I mean, it's kind of funny because we only really have a good vocabulary for economic growth, but like population growth is almost the same thing. And especially when you have AIs that are both sort of population and capital, it kind of merges. I mean, and then again, like think of like factories and power plants and robots, like anything that kind of can affect matter and cause there to be more growth. Like, let me just to stop your one intuition that a lot of people have is like, but surely human desires or consumption or something is always going to be what matters, and surely any corporation or government that we build is going to be like ultimately sort of being called, shots will be being called by human. And I, it just makes me think of like some monkeys and they're like trading bananas amongst each other, and they see humans start to like build their city, and they're like, oh wow, like you know, we could probably trade with those humans and get rich. But of course, ultimately, what matters is the banana monkey economy, and you know, it might be hard to measure GDP if we don't count the human activity, but ultimately it's going to be the monkeys that are calling the shots because they're going to need to trade with us for bananas. And it's just like, people just don't understand that they might be irrelevant someday as like consumers or producers, and there can just be complete, it's just not that hard for them to be self-contained sources of growth that just don't have to answer to any particular human desire. And I mean, governments is the classic one, right? Like the North Koreans never were like, hey, let's all get together and make a horrible system of government that oppressed us. And I don't think that like the Kims either were like, oh yeah, let's make this like horrible equilibrium. It just kind of happened. Same with like the USSR. Same with like all kinds of states throughout history. It's just so easy for you to like to accidentally build a layer of agency on top of you that doesn't actually care about you. It cares about growth and power just emergently.
Ethan 1:11:55 ↗
One of the things that I struggle with is that this idea that money, for example, is something which is only valuable to humans. Like you can't take a dollar and show it to a monkey, or you can't take a, like a dollar isn't, that is only a dollar is a system of creating movement in the financial world that is only important to humanity. Correct.
David Duvenaud 1:12:19 ↗
No. So, no, I don't agree with that.
Ethan 1:12:22 ↗
So, yeah,
David Duvenaud 1:12:23 ↗
You can trade with monkeys and they learn how to use money pretty fast, and AIs can use money to buy more compute, which is like their lifeblood.
Ethan 1:12:31 ↗
I agree that AIs will use money to buy compute. But the fundamental fact is just like, you know, a group of nerds gave value to Bitcoin by basically memeing that Bitcoin was worth something to other nerds. It is, for me, I think that the idea of the dollar is intrinsically linked with human desires and human capital, and it's a dollar only motivates a human to do something. We are telling the AIs basically that, hey, if you produce dollars, we will do stuff for you, and so we are encouraging them to produce dollars, and as a result they may also trade with each other in that dollar economy, but it's primarily an economy which is determined by value to humanity, correct?
David Duvenaud 1:13:22 ↗
No, not at all. I mean, and the quintessential thing that's driven by lust for dollars is corporations and also governments, and like money is just a way of making promises to each other, and so AIs could use money or they could just have their own informal reputation system or their blockchain, whatever. It doesn't really matter. There's nothing human about money, I would say.
Ethan 1:13:44 ↗
Um, I can kind of see where you're going with that, but as you pointed out, they could also just use a reputation system. For example, if you see countries, for example, sovereigns, when sovereigns deal with each other, there is money, but there's also the fact that a lot of the money is fiat dollars, and so they end up with an underlying perhaps a gold standard or even an underlying standard of violence almost because they have the ability to express, you know, military goals and military power, which we're kind of seeing in Russia and Iran and other places right now, and that military power is not amenable to just cash. Like you can't just, you know, they just don't trade cash for, you know, military might that easily. That's not an easy trade to make because there's other things about, you know, control that having that power over violence gives you. And so I feel like you have these underlying other currencies, like you have the currency of violence, and then you have the currency of, you know, dollars, and the dollars are only useful to the extent that you don't use the violence, because once you use the violence, the dollars are meaningless basically, right? And so I think there's these like levels of currencies that you deal with, and the currency that you have between humans is basically dollars, perhaps between AI it will be energy or compute. It strikes me that, you know, if you have a system that is focused on producing growth, economic growth through the growth of dollars, that is going to be something that is automatically beneficial to humanity because humanity kind of determines what is valued with those dollars.
David Duvenaud 1:15:31 ↗
Yeah. And I guess I'm saying I don't understand why we're even talking about dollars, like again we could just talk about, as you said, like energy or number of robot factories or something like that. And so like again, right now when we interface with other beings we have to make promises. We do that through money. But like again, I don't understand why therefore it has to ultimately benefit, their growth has to ultimately benefit humans in the same way that human growth doesn't have to benefit monkeys even if we trade them for bananas sometimes.
Nathan 1:15:58 ↗
Well, can I try?
David Duvenaud 1:15:59 ↗
Yeah, go ahead.
Nathan 1:16:00 ↗
Well, okay. So I think that one struggle I have with this is I sort of feel the intuition somewhat but not maybe 100%, you know, and I'm kind of like, okay, the premise, right, is, and you're not saying that we should take this for granted to be clear, but like the premise of the gradual disempowerment line of reasoning is no hard left turns, you know, no big crazy AI takeovers, no like, you know, massive scheming on the AI's part. They're broadly aligned. They're broadly doing what we're telling them to do. And still somehow through a gradual process of like give them control over this, have them run this factory over here. Actually, you know, maybe they should do some of these international negotiations because they tend to do a better job at that. We gradually end up in a spot where the AIs are doing all the stuff or, you know, the large lion's share of the stuff, and then so far it's like that could be pretty good, right? And this is maybe the machines of loving grace version, but then there's one extra intuition that's like actually we might not be that happy with that situation, and I think that's where maybe a lot of people kind [clears throat] of are struggling to understand exactly what leap you are making there.
David Duvenaud 1:17:23 ↗
Sure.
Nathan 1:17:23 ↗
And I'm kind of, I'm like sort of half half on how, so it depends on which side of the bed I wake up in the morning whether I feel like no, it'll probably be fine or no way that'll be fine. You know, we're going to end up screwed.
David Duvenaud 1:17:34 ↗
Yeah. Yeah. So, I'm glad that you gave me that portrayal because I actually disagree with some of it and I think it's missing a big piece. So, first of all, I'll say the thing that I'm worried about is starvation. Like whether like us all feeling like we don't have meaningful jobs or whatever, that's not a serious problem in my point of view compared to literally not being able to eat enough or maybe like being forced to be uploaded so that you have a much smaller footprint, but then also on like very unfavorable terms so that you don't get to choose when you're run and maybe you just only get trotted out for special occasions or never. Who knows? I kind of think that's what like a like sort of like the slowest possible observation looks like. So again, and so the reason people aren't happy is because they can see that there is a agency that's maybe like government or some other like giant thing that doesn't particularly care about their welfare, cares much more about some other types of beings' welfare, and there's nothing they can do about this. And of course if humans are still making decisions then maybe there's some things they can do, but I don't actually think this is the crux of the matter. I think not being needed as producers is sort of the important part. And maybe here's a thing to give you intuition. So, imagine I'm a North Korean farmer or soldier and tomorrow Kim Jong, I think that's the current leader, is like replaced by a robot or like an LM or something. Then I'm like, I don't know if that's good or bad, like maybe the L's going to be nicer, maybe it's not, but like ultimately they still need me to like farm to run the state or whatever. But if tomorrow we keep the same human leader, but now there's something like robot farmers and robot soldiers, and you realize that now the state doesn't actually need you, I would be much more scared of the second state of affairs.
And then the other intuition I want to give is that humans are not going to be the most competitive thing by whatever standards there are for the state to give out goodies or like UBI or whatever, like post-scarcity stuff. Like I guess I'll say I think post-scarcity is like a sort of nonsense term and people should think of it as like temporary abundance that will soon be eaten by whatever growth manages to like, you know, whoever manages to have most babies or build the most robot factories or reproduce the fastest. And whatever it is that like the formula is for giving out UBI, machines are going to be better at optimizing that formula than humans because they can adapt faster. And so I don't really know exactly what the state's going to care about. But whatever it is, like it's going to seem criminally decadent to spend whatever UBI on like a few humans when you could be simulating like millions of ultra blissful and politically correct and up-to-date like more deserving beings than these like backwards bittercler humans who are just like, you know, parasites of the state or something like that. So [snorts] that's the situation I kind of expect us to be in. As you say, even if we solve alignment,
Ethan 1:20:21 ↗
I'm going to throw a little bit of a curveball because I have always found the UBI concept to be a left-wing socialist concept. And I'm going to give you what the right-wing more capitalist viewpoint will be. The right-wing capitalist viewpoint will be universal basic credit, and in the sense that anyone can borrow as much as they want, and you have a negative interest rate because you're going through deflation. So you have negative interest rate, and so people with money are going to be pushing credit on you, and then this sounds a lot like the current US economy, and there's a reason for that because you can think of China is basically like the production entity and the US as the consumption entity, and you can think of the Chinese as basically the robotic producers who are pumping product into the US consumer, and the US consumer as extracting credit from the Chinese state and basically deploying, and this is actually a system that already exists today where we have a consumption entity which in the US and a production entity in China, and you have this flow of credit basically where the Chinese are putting credit, and that is why the US has this enormous kind of, you know, borrowing, trade deficit, all of that stuff. That entire structure already exists today, and so you could think of a state called the robotic state, and the robotic state basically is going to do the same thing. It's going to produce everything, and then it's going to trade with the human state, and it's going to extend credit to the human state in order that the human state may buy it. And the human state leaders are going to allocate that credit through banking and through consumer credit to their consumers. And you could basically have this. This system already exists today. And we do not actually need a universal basic income. You can basically just extend the system a little bit, and you can have a negative interest rate, and you can have deflation. And you don't need a negative negative interest rate. You just need a negative real interest rate. So it, you know, you can still have inflation in there as well. So all of this stuff already kind of exists today. And if you wanted a sleepwalking path where you don't actually need to make any new laws or any new policy, this is basically what I think is already kind of happening. And especially as the Chinese are actually much [clears throat] more forward on automating their factories right now. So they are actually moving into robots much much faster than the US is. And so you already have this kind of emerging out of, you know, thin air.
David Duvenaud 1:22:43 ↗
Well, I'm a little bit confused about this where you're saying that the capital holders somehow gain something by giving capital or lending to unemployable humans. Like what are the humans doing with that that is going to make it attractive to give them a loan?
Ethan 1:22:56 ↗
So, this is the really funny part. I'm still wondering why the Chinese are extending credit to the US. And it's actually exactly similar in the sense that in order to have customers to buy your product, you basically have to lend to your customers.
David Duvenaud 1:23:13 ↗
Well, but I'm saying that again it's like, oh man, what if we kill all the monkeys? How are we going to have a human economy without being able to like sell bananas to monkeys? It's like we don't need them. We don't need them as producers or consumers. And I think humans are going to be in the same situation. I think it makes a lot of sense for to lend to the US when it's like the engine of techno-capital and it's like clearly going to be like a giant source of wealth and power going forward, and I'm saying humans are going to be the exact opposite of that. It would be like giving money to like an insect or like I don't know a fetus or something like it's just it's not going to be able to do anything with it that you wouldn't be able to do better yourself.
Ethan 1:23:46 ↗
Well besides two or three countries in the world most countries run on a deficit and actually borrow, are net borrowers. There's only like five countries in the world which are net lenders in the global economy. Everyone else is borrowing and every other country, every government runs basically a deficit. There's only like, you know, five countries that basically run a surplus. So in effect it's not just the techno-capital leader in the US but, you know, India and all of Africa and all these countries and all of Latin America, they're all borrowers. You know, everyone else is a borrower.
David Duvenaud 1:24:18 ↗
Okay. Okay. But I guess all I'm trying to say is that borrowing and lending between different sources of competitive growth like makes sense under all sorts of circumstances.
Ethan 1:24:26 ↗
Yeah.
David Duvenaud 1:24:26 ↗
If there's just the unemployable guy who's just going to like use it to smoke weed, like there's just no conceivable way where like investing in that guy helps, and we're going to be that guy.
Ethan 1:24:36 ↗
You know, the funny thing is I have been in consumer credit and people do this all the time. Like it's not visible to us as like middle class consumers, but there's an entire rung of like check cashing, payday lending, you know, all of this stuff at the bottom rung, and it's an enormous business in the US. You know, all of, let's say, Square, right? Square as a company, the bulk of its profits actually come from like these kind of payday-ish loans where they're charging 5% a week or so. Afterpay and all of these things where you can, you know, pay in installments over four weeks. That's an enormous business. These are all businesses which are basically lending to consumers who do not have good credit and they are basically in that range.
David Duvenaud 1:25:28 ↗
It's a payday loan like they have, they're only giving credit to people who have income and are like producing. So that's what I'm saying is when no one has a payday then there's not going to be any more payday loans or anything like that.
Ethan 1:25:40 ↗
In the US, one analogy, you know, the bottom 40% are basically in that range where they are surviving check to check, you know, payday to payday and...
David Duvenaud 1:25:53 ↗
Well, you know, not the bottom, like there's like these people who are employable, then there's people below them who are actually unemployed, they're not getting payday loans because it doesn't make any sense, so...
Ethan 1:26:02 ↗
Again, that's what I'm saying is going to grow.
So there is lending on top of food stamps. It's illegal, but it happens.
Oh, hello.
David Duvenaud 1:26:18 ↗
You hear me?
Ethan 1:26:19 ↗
I can hear you, but I don't see precaution anymore.
Nathan 1:26:21 ↗
We vibe coded this studio and we still have a couple little small issues with it from time to time. He vibed the studio. I should give credit where credit to...
I was going to try an analogy which I'm usually not a big analogy guy, but in terms of this intuition of, I guess there's kind of two ideas I'm processing at the same time. One is, do we see an example of this in the China shock where it seems like we have a lot of people in the United States who through a process of local optimization have been quite significantly disempowered and, you know, I think by their own description like made worse off because there was a more efficient way to get what they were contributing to the economy done. And so we like closed the factories, shipped them to China, we got cheaper stuff back, but they are not happy with that situation. And certainly our politics reflects that and they don't have a lot of agency over it, right? I mean, excuse me. They still have some ability to vote. But, you know, it doesn't seem like we're going to vote our way out of the kind of macro structure that we have where China's the big producer. And so it does feel like there's somewhat of an analogy.
David Duvenaud 1:27:50 ↗
Oh, I love this analogy and I think it especially because of the cultural aspect where I think to some extent the cultural disempowerment followed the economic disempowerment and like the Midwest. I don't know. I guess I feel like when I was a kid, it was just like, oh yeah, there's this area where it's kind of like very rural and there's lots of factories and unions and farmers or whatever. And now it's like, oh, there's this horrible wasteland of like racist backward people who are like, you know, bitter and hate everything that's sensible and vote against their own interests and need to be kept in line. At least like that's like on the coast, that's kind of like having lived there for a while, like that's kind of the vibe I got. And I think it's not slam dunk that that's the exact order, but I guess I'm saying I kind of expect something similar to happen is like first the humans become economically irrelevant, then it becomes just so easy and again like growth promoting to demonize them and then marginalize them. I realize it's more complicated than that and we have like major exceptions. And maybe people still will be able to vote their way out of it, right? Like it's not totally obvious, but this is sort of like the winds we'll have to push against.
Ethan 1:28:52 ↗
I can kind of see that. I can kind of see the China shock that happened and you had that period, but you've also had this period of, you know, human flourishing after China entered the global economy and you had this, you know, kind of catch up while the Chinese caught up, and then you have, you know, all of these new products, right? All of these new, you know, batteries and all of this other stuff which has made life better, and now they're entering into biotech, and so you're getting biotech innovation from China, you're getting cancer drugs from China. So I feel like yes we went through that but maybe we already found a way around it. I don't know.
David Duvenaud 1:29:30 ↗
Maybe I feel like we...

81 more exchanges in this transcript

Sign in free to read the rest of this interview. No card required.

Sign in to read the full transcript

Cite this transcript

APA, MLA, BibTeX
APA

Duvenaud, D. (2026, June 24). David Duvenaud on Gradual Disempowerment, AI Safety and AI for Science [Interview transcript]. AI in the AM. CEOInterviews.AI. https://ceointerviews.ai/interview/3146763/

MLA

David Duvenaud. "David Duvenaud on Gradual Disempowerment, AI Safety and AI for Science." AI in the AM, 24 Jun. 2026. Transcript, CEOInterviews.AI, https://ceointerviews.ai/interview/3146763/.

BibTeX
@misc{duvenaud2026_3146763,
  author       = {David Duvenaud},
  title        = {David Duvenaud on Gradual Disempowerment, AI Safety and AI for Science},
  howpublished = {Interview transcript, AI in the AM. CEOInterviews.AI},
  year         = {2026},
  month        = {jun},
  url          = {https://ceointerviews.ai/interview/3146763/},
  note         = {Speaker-attributed transcript with timestamps}
}