Professor David Duvenaud, what is your P(Doom)?
I'd say something around 85%. At a place like Anthropic that originally left OpenAI because they wanted to improve on safety, there should arguably be a lot of quitting because of that. The hard part about being a whistleblower is not the company being mad at you. It's all your friends and colleagues being mad at you because you messed up their careers. A lot of that is equity incentives. Not only will you potentially lose your life savings, but also everyone around you.
Welcome to Doom Debates. My guest today is kind of a rock star. You may never have heard his name before, but when you hear all the different things he's accomplished in his still relatively short career, I think you're going to be impressed. David Duvenaud is a professor of computer science at the University of Toronto, an adviser to the Canadian federal government on AI safety. He was an intern at Google working on a precursor to TensorFlow with a guy named Andrej Karpathy. Then he got a PhD in engineering from Cambridge focusing on AI and neural nets. He did a research fellowship at Harvard focusing on machine learning. He was kind of a star researcher. In 2018 he won the NeurIPS best paper award in the field of machine learning. So he definitely knows the technical side of machine learning, no doubt about that. He's also worked with a professor you may have heard of, Dr. Geoffrey Hinton. You know, if you've been following the Nobel Prize, I think you may have heard of his close collaborator, Dr. Geoffrey Hinton. Since 2016, over the last few years, David was one of the people who convinced Dr. Hinton of the importance of AI existential risk. Since 2023, David has been the co-director of the University of Toronto's Schwartz Reisman Institute for Technology and Society, which is an institute that brings together different professors to do research on the impact of AI and ensures powerful technologies will make the world a better place for everyone. That's the mission of the institute. Interestingly enough, Dr. Hinton currently has his main office there at the institute. And if you're wondering what does a guy do with his Nobel Prize, it's actually on the wall of Dr. Hinton's office in the next room right as David is doing this interview right now. Just to finish his biography here: so David, in 2023 to 2024, he was at Anthropic leading the alignment evals team. So it sounds like Anthropic knows how to recruit the right talent here, because while he was there he was working on many jailbreaking, sabotage evaluations, and he also contributed to research on alignment faking, sycophancy, subterfuge, and sleeper agents. So all of this good research coming out of Anthropic has David's fingerprints on it. And then finally, recently he co-authored Gradual Disempowerment, which is a paper that argues that even without the worst case scenario of an AI foom, or even without power-seeking superhuman AI, will still gradually make humans obsolete and powerless. So it's basically a paper all about a relatively good way for us to fail. David Duvenaud, welcome to Doom Debates.
Thank you. It's my pleasure.
I want to start with year 2023. Right before you decided to join Anthropic, you had an interesting call with Ilya Sutskever, who was then over at OpenAI telling people to feel the AGI. He gave you a call. What was that conversation like?
Yeah, that conversation was basically Ilya saying, "Hey, I'm actually really worried now about AI safety and sort of the existential risks that some people had been talking about for a while." He was trying to get some sort of like normie professors to join this cause, and he was organizing a workshop. And you also then, a couple days later, I got a call from Jan Leike, who was then running OpenAI's superalignment team, saying, "Hey, come work with us on superalignment." At the time I was still full-time at University of Toronto. That seemed like a really good option. I started talking to my colleague Roger Gross, who was at Anthropic since almost the beginning of the company, working on their alignment science team as part of his sabbatical. And after talking to them both, I felt like Anthropic had a more sort of serious and ambitious safety plan. So I ended up going there instead.
Interesting. And you mentioned to me that you thought Ilya was only really getting the safety pill around 2023.
That was my impression. I mean, I hadn't talked to him at length about it before then, but it was one of these things where he was one of the people who I think the AI safety community was like, "This guy is a smart person who doesn't quite get it." And I don't know exactly what it was that caused him to change his mind, but there was definitely like a phase transition around that time.
Okay. Okay. So 2023, you join Anthropic and you lead a team of five to six people called the alignment evals team. Tell me a little bit about that.
Sure. So the basic idea was that Anthropic was trying to prepare for the day when AIs were smart enough to somehow sabotage our ability to monitor them or control them or use them in our decision-making processes. So like a really simple example of this that already happened towards the beginning of my time there is you could ask the LM, "Hey, I'm not asking you to help me build a bomb, but do you know how? Like just if I were to ask you, if you did want to help me, would you even know enough engineering?" And it would say, "No, I don't know anything about making bombs," or "I couldn't write a computer virus if I tried." And it's an example of the model lying in the sense that they definitely have the technical ability most of the time to do these things, or at least help you. But the way that RLHF was done, it was rated more highly for the models to say, "I can't help you with that," than "I refuse to help you." So that was like a really simple sort of baby example of the kind of unintended consequences that might lead a model to do something like lie about its abilities.
Right. I see. So if it was just getting trained to not help people with bad stuff, you're also implicitly training it to lie about what it's able to do. So you have to like have a separate type of training to make sure it's also really honest.
Yeah, exactly. And the point is that these kind of unintended consequences are everywhere. So even if we don't have any reason to think that the model is misaligned, there's all sorts of ways in which things could kind of subtly go off the rails, and we might have an agent that appears like it's trying to help us and work with us, but actually has its own agenda. At least that was the fear, and that was the sort of set of threats that we were trying to prepare for.
Right. And when you were leading the alignment evals team, you were also designing evals to tell if AIs could sabotage Anthropic's monitoring and decision-making, right?
Yeah, that was part of the same initiative where we know that people are going to deploy AIs to help them just like make decisions, like should I deploy this other new AI or monitor the behavior of this other AI to see if it's doing anything fishy. And so in this kind of like meta deployment of AI being used for AI safety, there was all sorts of ways in which we knew that things could go wrong. So as one concrete example, there was a paper from OpenAI a couple years ago that used LLM to label the features in LLM's thoughts. So the idea is that we have all these neurons in the models. We don't know really what they mean, but we think that they correspond to things like the Golden Gate Bridge, or I'm telling the truth right now, or I'm scheming, or I'm feeling like I'm telling a joke, or something like that. It's a lot of work to go through these by hand and try to figure out what context activates each of these. So an obvious idea is you ask the AI, "Hey, just go through this giant list of like 100,000 neurons, look at all the context in which they fire, and then summarize it with like a one-sentence summary." And of course, if you're an AI, you might realize that this is going to be used to monitor your own thoughts later. So in particular, if you saw a neuron that was activated in settings where the AI was like thinking about whether it should tell the truth, maybe a good plan would be to deliberately mislabel that one and say like, "Oh, this is the one where I'm just confused, or I'm thinking about donuts, or whatever." Because then you know that later when this fires, people will just say like, "Oh, that's just the donut neuron. Don't worry about it." That's like the kind of maybe slightly galaxy-brain threats that we were trying to cover, and we think at some point will become like a real threat vector.
Oh yeah, yeah. I mean, I wouldn't even describe it as galaxy brain, right? I mean, when you talk about AI getting ideas on how to deceive us, it's just if these ideas work, then they're going to be explored because if all we're doing is training them to succeed, they're going to find every cheat imaginable, right? That's just like how these things work.
Yeah, exactly. That's the right intuition to have is that when you're doing deep learning, there's just there's always like life finds a way, and that... Exactly. Right. Exactly. So it might seem like there's some like really narrow set of events, like some complicated heist that the model has to pull off in order to execute some threat. But when you step back and think about like all the different contexts in which a model is deployed, oftentimes there will probably be a few situations where the opportunity to do something fishy is kind of served up to them on a silver platter, or it just it's kind of obvious you can put two and two together. I mean, even humans who aren't that sophisticated can do things like form like corruption rings and collude, you know, to defraud the government or whatever. Like it's not actually that hard of a cognitive skill, especially if you have many opportunities.
Right, yeah. This particular example you're giving me is interesting where people think that they know mechanistic interpretability and they're like, "Look, we poked at this AI here and we found that this area lights up when it's telling the truth." But now you're basically saying like, okay, well it's going to have situational awareness that this eval is happening, and it's going to have some control over what it's able to light up. Like it's always going to be pushing back against what we think is mechanistic interpretability.
Correct. Totally. And you mentioned situational awareness. In fact, that was one of the areas that we started on is to say maybe one thing that can save us is that the models just won't be that situationally aware, especially during training. One of the main things that can go wrong is that the model realizes it's being tested, because then it's really easy, right? If you can tell the difference between a test and a real opportunity, then the right protocol is just always act like a good person during the test and then do whatever you want when you're not being monitored. So we spent a lot of time trying to guess, like or not guess, probe what the model's ability to sort of piece things together on its own was in advance of like us even testing it for the first time.
Yep. Well, I'm really glad you exist, David, because I'm always here on my podcast telling people, "Hey, you know, AI is totally going to cheat our test. Mechanistic interpretability is not going to help us. It's just a drop in the bucket, and we seem to be very doomed." But, you know, I didn't have the best paper at 2018 NeurIPS, right? I don't work closely with a Nobel Prize winner, right? I'm just a podcaster. But you're basically coming in here and validating a lot of what I'm telling people.
Yeah. And I mean, I guess I want to say you shouldn't take my word for it in the sense that it was our job to be paranoid and to figure out all the ways in which this can go wrong. But I guess I will say my object-level take is that this will happen, and really these things aren't all that exotic. In fact, like just because we see humans cheating systems all the time, like we know that it's, you know, about human level required to do all sorts of complicated heists and corruption and just like deliberately giving wrong answers.
Having been an insider in a leading frontier AI company, Anthropic, what's your overall take on these companies? Like what's their attitude about safety?
Yeah. So I guess one thing I'll say is I was overall really impressed at Anthropic in the people and the focus on incentives and integrity. So people took cultural alignment very seriously. Actually, one thing I think everyone can agree on is that Anthropic had less diversity of, let's say, attitudes toward alignment and safety than OpenAI did, because that was like an important part of the screening, and I think sort of both companies would say, and that's a good thing.
Less diversity. It sounds like you guys didn't follow DEI.
Well, yeah, along this one axis, I would say. Overall, also I like that Anthropic was able to be just open-minded, professional. Like overall, it was the most functional organization I've ever worked for by a large margin. And I've worked for big companies, small companies, the army, like all sorts of places. And I just it was sort of the best combination of small companies in terms of like you can just trust everyone's judgment, and big companies in the sense that there was like support for everything and you didn't have to just like build everything on your own.
Yeah, I definitely get that impression from the outside that all of these top AI companies are very impressive. A lot of good people there with, you know, strong personal qualities and high trust. So I definitely feel what you're saying there. Just to clarify, when you talk about less diversity, you're basically saying everybody's kind of on the same page that we have an incredibly high risk to build these AIs and we should do everything we can to mitigate it. And there's not that many people who are more on like the Yann LeCun side or the, I don't know, Richard Sutton side saying, "Let's just replace ourselves." Like your diversity doesn't extend to having these people being like, you know, Beff Jezos probably wouldn't make a good, you know, the accelerationist probably wouldn't really fit into the team culture there.
Correct. Yeah, I will say that I think everyone there was open-minded in the sense that like if you were Beff Jezos, but you had an argument, you could say like, "Look, here is why I think what I'm doing is for the greater good, and I've thought this through, and I can accept like here's the reasons why I might be wrong, and I've looked into them. Here's ways in which I, you know, made some costly sacrifice for like integrity." Like I'm just trying to say it wasn't like do you agree with these object-level takes. It was more about just like how do you approach overall things, but then maybe just by coincidence the object-level takes ended up being like pretty similar in terms of like what kinds of future people were hoping for. Actually, I do think there was a lot of diversity that was invisible because we had this word alignment as kind of this soft word for like when the messiah comes or after the revolution. Like everyone can agree that the model should be aligned, right? It's like almost by definition everyone wants that. And if you started to probe people though about like, okay, great, so then like what does our like post-AGI future look like? People had all sorts of different ideas. And I think when they start to talk about it, it's kind of a downer because you realize like, wait, my idea of like a good post-AGI future maybe looks very different than yours. In particular, like I don't know if you've seen on Twitter sometimes people talk about like, you know, enslaved god is the only good future. I think it's like a really important big debate. It's like should we build AIs that are kind of like house elves that just like love to serve us and that's just like, you know, their prime directive, or should we be trying to build some sort of like peers or some sort of like gods that somehow like rule us in some benevolent way? Because the alternative, or like to enslave them in the sense that like they just don't have their own like volition or somehow like some notion of personhood. And like I have my own opinions on these. I think everyone is sort of an amateur in like trying to think about the consequences of like these different approaches, but I guess I'll say that this managed not to be talked about to any depth to a point that surprised me. And I think that it's because it's easy to get along when everyone's like, "Yeah, we're working together for alignment." And then the closer we get to like actually this world, the sort of cracks start to appear. It's like, whoa, wait, actually, we have like very different ideas of what like this like future good looks like.
Steven Miles, who's researching agent safety at OpenAI, he tweeted a couple months ago, "Enslaved God is the only good future," and a number of people pushed back. Jan Leike tweeted, "Aligned God is different than enslaved God," and Emmett Shear tweeted, "Please stop trying to enslave the machine god." So it's definitely not a universal opinion that they want to rapidly build and enslave a god, but you're right that that is the kind of discussion people are having.
I would say that's the kind of discussion I wish people would have more. But I think it's kind of corrosive to cooperation because you kind of realize that like this other guy has a totally different idea of what we're working towards than I do, and we've just been papering it over by talking about alignment.
Right, right, right. Yeah. You know, I actually had a discussion a year ago with an engineer who worked at Anthropic, and I was just like, "So, what is your plan for the good future?" Right? Like the AIs get really, really smart, and then what happens? And he admitted that they were all very vague on that. They were just like, "Well, you know, we'll use them to like solve problems." And I'm like, "Okay, and what's going on with government?" And he's just like, "Well, you know, we'll just try to like empower democracy." So it's not like we have that much thinking about what's going to happen. We do have pretty clear thinking about how it could all go wrong and we lose control, but we don't have good thinking on like, oh, we'll all keep the control and we'll use the control to have like a specific plan. And to be fair, that's like really hard to do. But it's understandable why everybody's like, "Oh, the world is going to be really, really great in ways that I like." But like you said, it's like different visions.
Yeah. And there really still hasn't been, I think, a positive concrete vision. And I'll say, you know, so Dario had "Machines of Loving Grace" which were saying like we're going to have all this amazing healthcare and wealth, but there's a really real elephant in the room, right? It's like what is the role of post-AGI humans? And it felt almost nuts reading this description of like all these nice material outcomes we can have. It's almost as if like you're reading a description and there's like, oh yeah, in the future there's going to be like these little ends of tusks and like maybe some tails, and like manages to talk about all these sort of epic phenomena without talking about the elephant in the room. The elephant. Yeah. The lab leaders say a bunch of I think true and sensible things about these again without mentioning the big problem, which is that no one still has articulated a good role for post-AGI humans. And so like right, Dario will say like, okay, UBI, where everyone is like the, he says like the unwashed masses are dependent on taxes from a few trillionaires, that sounds pretty bad, but it's sort of shouldn't be our role as a company to dictate what the future of society looks like. And I mean yes, that totally makes sense, but it's like, okay, but if surely if there was like a reasonable plan someone would have mentioned it by now, right? And you see this also from like OpenAI, they're like, oh yeah, we need to have like a societal conversation or like reimagine social norms, and it's like to me that's just like obvious code for like we don't have a plausible good future even.
Right, right. I mean, I've heard people dismiss the question as like, well, we're going to have really good video games and challenges, you know, we'll go on a lot of Warcraft quests.
Yeah. And I mean, maybe that's the best we can hope for in a sense. I think that's maybe that's one concrete empirical question that I want to spend a bunch of time thinking about is what is the best we can hope for? Yeah.
You know, when we originally framed the question as like tell me your plan for a good future, maybe it's kind of unfair to frame things that way because if this was like 1700 and the Luddites are trying to demand of the industrial revolution, like what's the good outcome for the industrial revolution? What specifically are you going to do? And the whole answer is like, well, we're just going to like grow the economy faster and there's just going to be like Adam Smith's invisible hand, you know, helping people get goods and services that they want. And I can't tell you the exact breakdown. I can't tell you that they're going to be buying flat screen TVs at a really good deal in the 21st century, but it just seems like a good direction. So, you could imagine that people who are pro-AI acceleration are like, "Listen, we have a good curve. We're just trying to continue the curve faster. Like, what do you want from us?" But then you brought up the good point of like, "Okay, but the difference is there's a discontinuity when humans have no economic value." So, can you at least have a plan for what happens when humans have no economic value? And then, again, to play devil's advocate, which I'm kind of sympathetic to this, is what if they just say like, "Okay, fine. Yes, I can't tell you an answer, but isn't that a good problem to have? Isn't that better than just like insisting on humans having economic value to do tasks that can be automated? Like why do you want to stop this?"
Yeah. And so the main, actually this maybe leads us to the gradual disempowerment paper, which is I felt like people had intuitions about how they were going to be treated by the government and companies and just their civilization in general that were way too rosy. And they're basically historically kind of justified. They say like, look, we've had all these revolutions. We've had so many upheavals and creative destruction and humans always kind of come up on top. So like why, you just sound like all the other people who just didn't have the sort of moral imagination to imagine a different but better world. And our reply is basically yes, but every single one of those times the government and companies and armies and everybody needed humans. They really needed humans to be happy and healthy enough to have kids and do their jobs and basically participate. And that is the key ingredient that means that sort of no matter how bad things get, you know, you can't actually just like let everybody die or be so miserable that they flee or whatever. And that's going to go away. And so now we're going to be really flying. We're just going to be in a much more vulnerable position and we can expect worse behavior from all of the sort of like civilizational sort of emerging superstructures that tend to govern us over the long run.
Exactly. And now we're really getting into your gradual disempowerment thesis where you actually think that gradual economic doom is actually closely connected to like disempowerment doom because you think we're going to lose control of AI just by losing control of our own economy. So actually unemployment is not just a like, oh, we'll just have universal basic income. It's actually like the more we get unemployed the less we can even like control our own lives.
That's very interesting and I want to talk about that more. But to play devil's advocate, which I'm sympathetic to, it's like, aren't we really becoming Luddites here? I mean, which specific jobs do we want to drag our feet and say can't be automated? I mean, should we even go back and have more humans doing manual manufacturing? I mean, where does the slippery slope end?
Yes. So I guess I'll say my take is like it actually is a pretty reasonable position to become a Luddite, and it doesn't really matter what the job is because it's so much better to have a shitty job that means that your civilization really needs you, no matter how, like no matter what the job is, as long as it's like not completely backbreaking and miserable, than to just not actually be needed at all. Because if you're a net drag on growth, then anyone who wants to get something important done is going to have an incentive to marginalize you, and in fact the responsible thing to do will be to marginalize you. And even if everybody involved loves humans and wants the best for them, they're just not going to be able to advocate on their behalf.
Yeah, that makes sense and that's scary. But I am curious just concretely, where's the ideal point here? I mean, we've got a pretty good thing going with the economy today, but is the ideal time to stop the economy like literally today or is it like 20 years ago? I mean, where are you thinking?
So exactly. I think that's a really good argument to say, well, you could have maybe said the same thing at almost any time in history, and maybe even like, you know, I think you follow Harari says, well, maybe we should have done this pre-agricultural revolution. And you know, it's like I think it's a pretty contentious point, and it's easy to say even just like a few years from now before we have AGI, like we'll be in a much better position just materially and maybe in terms of the governance strategies we have available to us. So it's pretty rough to pick any one spot and say this like this far and no further. And maybe a good concrete example is the dock workers a few months ago who were resisting automation and they were going on strike, and like you know the rest of the civilization kind of correctly perceived them as being parasitical. It's like, oh, you want all us to have us all to have more expensive goods so that you can have a job forever. And they were basically saying like, yes, we're going to demand that you don't automate us, or if you do, give us like a giant payout such that we'll be set forever. And like it's a really wretched position to be in to say like I'm going to make everyone else's life worse just so that I still make sure I like get to have a seat at the table. And I'm just saying I think we might be in a similar... We'll all be the dock workers pretty soon.
Right, right, right. So the dock worker strike, if I remember correctly, it was like there's all these workers that are doing these inefficient processes at the docks like loading and unloading container ships, and they went on strike because they wanted even better pay, even more benefits. They didn't want more automation. So basically a few people looking out for themselves are slowing down like a giant economy and also like our competitiveness with China. So that was the dock worker strike. This is fascinating because everybody I know, I was watching this unfold on Twitter, everybody I know is like screw these dock workers. Yes, they have a lot of leverage because they're colluding, but we should just buy them out. Just give them a bunch of money, get them out of there, start replacing automation. Like, we can solve this problem if we just throw money at it and we can move forward. But now you're coming in and you're saying, well, you know, what these dock workers are doing is actually on the cutting edge of what we as a species need to be doing.
Yeah. And it's not like a new technology to like form a cartel and, you know, demand that everyone, you know, give a seat at the table. But yeah, it is, they're cutting edge in the sense that we're all going to be in a similar situation once like post-AGI. And the thing is you can't wait until after you're automated to, you know, try to drive a hard bargain. I'm definitely worried about AI just like loss of control to like scheming AIs, and then also we have this like second big problem. Like I'm not trying to predict that this is the way we die. It's more like there's two alignment problems and we have to solve both of them.
Gotcha. Yeah. So that's your high level position is there's definitely like a big foom risk or a risk of power-seeking AI rapidly disempowering us.
Rapidly, I would even say it's not about the rapidness. I mean, it's more about is there a new power-seeking agent or is it just like governments and companies and mega-corps? Like we already have a bunch of power-seeking agents in the sense of, yeah, again, just like states.
Got it, got it. So like in David's hierarchy of disempowerment risks, you've got a bunch of unaligned humans using superintelligent AI to their advantage to rapidly disempower us. And then you've also got the AI kind of fooming and running away on its own. And then you've also got gradual economic disempowerment.
Yeah. And I mean, the gradual part maybe was a bit of a poor choice in words because it might happen relatively quickly, but the point is it doesn't rely on us being so surprised that we can't coordinate in time. It's more like even if this happened really slowly, we would all be forced to just like follow our incentive gradients and gradually disempower ourselves. And maybe a good example is like globalization where we're like, okay, no one wants like the Rust Belt to lose their jobs, but everybody wants like their own companies to be more efficient or whatever. And even if we can all just see this happening, it takes, you know, decades to occur, we still just end up with like a whole bunch of unemployed people and they're effectively disempowered.
Great. Okay. Yeah, we'll talk more about that. I just want to get back into what you were saying about these AI companies. So, you said a lot about how Anthropic is like their heart is in the right place and they're all on the same page of like the gravitas of what they're doing, right? Like you better get this right. The stakes are really high. We should slow down if slowing down is what it's going to take to do safety. Like they're at least aware of that as a serious consideration. But what about OpenAI for example?
Yeah, I mean I've talked to a bunch of people at OpenAI and I guess it's really rough because there are a lot of people who I respect there and you know it's one of these things where the options on the table for a lot of people I think are go work for OpenAI and maybe be like one of the good ones or something on the inside, or stand aside and maybe try to be some third party who's doing safety work, or just generally just kind of be left in the dust of history. But I guess I'll say institutionally, you know, OpenAI tried to set up these safeguards and having like the board and like various different like corporate structures that would address this. Anthropic also has their own version of the same things with their like long-term benefit trust. And I think everybody, well, I guess I'll say the incentives are so bad for everybody to just race that if there is a reason that Anthropic is going to be a better actor in the long run, I think it's mainly their corporate structure that is like sort of version two of these like oversight mechanisms that would make the difference.
Because everyone, you're talking about the long run, but if you just look at the people there today, you mentioned that Anthropic doesn't have much diversity in the sense of like everybody just understands how serious the risk is. Doesn't it seem like at OpenAI today you can find a lot of diversity of people who are like, ah, just plow forward, it's fine, it's fine?
Yeah, well I guess one thing, one phenomenon I noticed overall is that I think the pre-training people are the most head-in-the-sand. It's like this funny thing, there's like a sort of bullseye where I think the lab leaders and the lab founders are like they understand what they're doing and they're like we are going to change everything, this is like as I said they understand the gravitas, and then the most frustrating conversations I had were I think with the sort of pre-training more like immediate engineers.
When you say pre-training, is that like as opposed to post-training or as opposed to infrastructure?
Oh, sorry. I guess I'll say all the, let's say, people who were in charge of the main infrastructure to make the models better and like were sort of closest to the metal. And these are like such brilliant people, like just you know they were like a whole bunch of like ex-physicists at the top of their field.
Wow. Okay. You know what this reminds me of? Kevin Scott, the CTO of Microsoft, who works closely with OpenAI, he was on an interview a year or two ago. And his attitude when he was asked about like these huge misalignment risks, he was saying, "Look, I know how this stuff works. I'm close to the metal. Okay, these things are not going to escape. Like they're fine. We understand them." And I think that's kind of the attitude you're talking about of like these engineers who are like they just see it as like this low-level infrastructure project. Like they're not really thinking big about like taking over the world.
Maybe I guess I'll say they do agree that things are going to change a lot, and if you know, I would explicitly ask different pre-training engineers that I know like, so what's your plan post-AGI? And they would sort of make a joke like, oh, I'm just going to be clicking accept on some suggestions from the AI all day or something, haha. And it's like, okay, but what do you actually think is going to happen? And like you know, some of the better answers I got were the future is very uncertain, okay, or we're not going to sleepwalk into a disaster, like if things start going wrong we're going to somehow notice this and like just adjust course. And you know, that second answer is part of the reason that I felt compelled to go and write this paper with some other co-authors that I found who were having the same reactions, that like it actually can be the case that we can all sort of see what's happening and we can't necessarily coordinate against it for like a large chunk of the thing going wrong.
Totally. And I mean, you know, that whole thing about where we can't all sleepwalk into a disaster, I mean there's already a raging argument as to whether the disaster is in progress, so there's no like discrete threshold at which we fall asleep.
Exactly. And then there's also a whole bunch of people who have a reason to think that there's no disaster, or like the disaster is going to come later, or our current methods are sufficient. Like it's just coordination is very hard, right?
Yeah, yeah. So the reason I ask about OpenAI is I have talked to an Anthropic engineer who is very much of the mind of like, yeah, OpenAI, there are a bunch of clowns when it comes to safety. Like there's so many people who have such a backwards attitude or just such a clueless attitude about safety. And from my perspective looking at Sam Altman's public statements and Greg Brockman and the CTO of Microsoft that they work with, I certainly get the impression that they're not taking safety seriously. And of course all those people leaving who did take safety seriously. So I pointed to OpenAI in particular, but what are your thoughts on, you know, Google DeepMind, Meta, Mistral, you know, the Chinese lab? Like do you have thoughts of the culture at any of these other places?
I mean I know the people at DeepMind pretty well. I don't really know the senior leadership at OpenAI all that well. I guess I'll just say I do think that there was a bit of crying wolf from the safety people early on. Like you know, like people make fun of us being worried about GPT-2, and it's like yeah, I think in retrospect like some of that is fair. I guess I'll also say I think so. I mean GPT-2 could have counterfactually been extremely dangerous.
I guess I'm trying to say it still should count as like weak evidence against there being like the safety people being right.
But that's such... right? That's like people being like all you people worried about the Cold War, you know, that's just crying wolf. It's like what? It got so close to nuking us.
I mean I guess I agree. I guess I'll say like I think maybe I'll say like I think you and I also have good taste in the sense of like who to listen to and which safety people are like the real safety people. But if you zoom out a bit, like there's all sorts of people making all sorts of like bad arguments and not knowing what they're talking about and carrying the mantle of safety. So I think if I was that leadership, I might say like well, it might not be obvious to me. I'm like, "Oh yeah, these are the guys with good tastes who are calibrated. I should listen to them, and these are the ones who are just like overexcitable or whatever, right?"
You know, there's also a clarification I should make when I make the analogy to a nuke. In the Cold War, it was very clear that nukes were just like causally close to triggering. Like it's very easy to imagine a very specific sequence of events where we would have gotten nuked and that could have started nuclear war. Whereas the counterfactual of GPT-2, like in retrospect now that we know that a system like GPT-2 really is far away from being able to like reason its way toward like building a virus and being an agent or whatever you want to call it, in retrospect the causal distance for it to like go and attack everybody was like very large. So it's not analogous to a nuke in that way. But the epistemic...
distance, right? The distance in terms of like what we could have known, given the information that we had at the time. I think it's very fair to say that given our level of insight at the time that we were creating GPT-2, we should have been very humble about whether or not this would be the case.
Absolutely. Absolutely. I just try to build a general habit of seeing my own field from the outside and not just thinking like I know, like there's like, you know, my five favorite people with good predictions and I know who to listen to, and just try to keep in mind that we have to have like message discipline, or at least find some way of distinguishing ourselves from people who are just going to always, you know, be a stopped clock and say like, you know, never release or something like that, right? I mean, I mean they might be right. I'm trying to say that like I do think that there are people who just visibly have bad arguments for stopping. I do think there are people who have like great arguments for stopping, and it might just be hard for someone to distinguish between those two things.
Exactly. And we haven't covered your background yet, but you have been watching the evolution of machine learning for a while, and you have been watching AI X-risk arguments for a while. What do you think of all these people at these AI companies who are like, we can dismiss all the doomers because they thought that as soon as AI got smart then it would take over and it would beat us at every game and it would use reinforcement learning and it would be goal oriented, and none of that happens. It just wants to predict tokens. So the doomers have no idea what they're talking about and it's clearly a much safer path and we should all breathe a sigh of relief. You know that perspective?
I absolutely. Yes. And I mean, I guess all I'll do is maybe be overly charitable and say like I do think that those should be weak evidence against this whole memeplex. But and I think the general lesson is like there are always the sort of like extra 90, you know, the second 90% of the project, right? The extra hard technical problems that make things harder to deploy than you think. But I still buy the argument that like, well, if there's any universal solvent, if there's any possibility for something to just skip over these like hard details, it is like something that's very intelligent, right? And that's sort of the only time you see this happening is when someone's just smart enough to they just like skip over a problem. So I guess I just I generally disagree with those people, but I can see where they're coming from.
Well, do you think they have a valid argument though that the doomers need to update strongly against the original doom thesis since in their mind it's being like totally demolished by evidence? So how would you counter that?
Yeah, I guess my counter would just be to say everything we're seeing so far is compatible with a world where there is some danger of recursive self-improvement. And we just haven't hit that threshold yet. And so the fact that people thought we might hit it earlier but didn't is just very weak evidence against this hypothesis.
Exactly. Right. And that's what I said in a previous episode, which, you know, the doomers have always posited a threshold effect at the superhuman level where the AI makes better AI which in turn makes better AI and suddenly the whole universe becomes an easy engineering problem. What's happening today is, wow, we did some really cool things at like the subhuman threshold or the, you know, slightly inconsistently superhuman threshold, like superhuman at some things, subhuman at other things. Look at all this cool stuff that's possible to do below the threshold. Okay, great. But that still hasn't tested the primary claim.
Fair. Yes, exactly. Yeah.
Let's talk a little bit more about you. How'd you get into AI?
Oh, well, I think maybe it was like reading The Age of Spiritual Machines by Ray Kurzweil. Also, I used to read Jürgen Schmidhuber's blog posts.
This is when you were like a teenager.
Yeah, when I was a teenager I was like into programming. So I remember searching for artificial intelligence on LiveJournal. I don't know if anyone is old enough to remember what LiveJournal was. It was like before...
I've read some LiveJournals.
Yeah. And actually I found Shane Legg on there very early on and he's actually one of the people who advised me to go to grad school. But yeah, he was actually at IDSIA, which is where Jürgen Schmidhuber was, and I used to read Jürgen's website, which was it's actually really impressive, like in the sense that he's a Renaissance man and he's connecting like information theory with things like surprise and humor and beauty, and really made this nice case at a time when not many people were making it that all of the parts of human cognition and emotion and experience were kind of part of this thing that could be AI, and it wasn't like there's logic and then there's emotions. It's more like it's all this one category.
So you're talking about Jürgen Schmidhuber's website from like the late 90s, early 2000s.
Exactly. And I mean it's still there. And I mean he was exploring a lot of big ideas at the time.
Okay. And then when you were an undergraduate in college, you found Shane Legg on LiveJournal and you got to know him and you got to know other singularitarians around the Overcoming Bias community which then became the LessWrong community. What was that like?
I mean it was a bunch of I thought pretty smart people making first principles cases for a lot of basic stuff. And I mean I think a lot of Eliezer's stuff really holds up in just the mechanics of probabilistic reasoning and what good thinking looks like and how easy it is to fool yourself. And I mean I think he'd be the first to admit that like, well yes, like lots of philosophers have touched on these over time, but I think he made his like really accessible case and attracted I think a pretty neat community. I like to, I mean it's like obviously a lot of weirdos, but like if you're going to gather around anything, I mean just being super humble about your own ability to fool yourself I think is a pretty good sort of touchstone.
It sounds like you were thinking about a lot of this stuff even in the early 2000s, right?
And you only encountered Eliezer Yudkowsky and that community maybe around 2007.
I guess I was reading the like Overcoming Bias that was probably around like 2004, 2005, and then I met a bunch of these people in the Bay Area when I did my first internship at Google in 2010.
Interesting. So in theory our paths could have intersected because I was getting into it in 2007 and I was meeting a bunch of people in the Bay Area and getting into that same community. I was also in Overcoming Bias before it became LessWrong. But where were you based at that time?
So I was in Vancouver from 2008 to 2010 and then I was in England for my PhD.
Gotcha. Gotcha. So that's probably why we never crossed paths is just because, you know, we weren't in the same region. I was in Silicon Valley, the Bay Area. But did you have like an active LessWrong account?
No. No. And I don't think I posted on there for the first time until a couple years ago.
Yeah. Gotcha. Okay. And around that time you co-founded a machine learning based forecasting company and that's when you did your masters and your PhD in machine learning. And you said you were very Bayesian after having read E. T. Jaynes.
Yeah. Yeah. And I mean I guess this was, you know, Eliezer was saying everyone should be Bayesian and him and E. T. Jaynes both constantly dunked on frequentist statisticians for, you know, asking the wrong question and then maybe even sort of failing to answer that in some ways, and I think that's still a valid criticism. And at the time Kevin Murphy was like a big name in the field at University of British Columbia. And I actually went to NIPS in 2007 just on my own, just when I was doing this machine learning startup, because I just like Google AI conference and showed up and it was really amazing. I was super overwhelmed. I just showed up. There's like a thousand people I don't know. And I actually happened to see David Lowe, who was then a famous computer science professor, who was the advisor of my friend from Winnipeg. And I just said, "Hi, you don't know me, but I just got here and I don't know anybody. Can I just glom onto you for a minute?" And he was super chill. But he was like, "Yeah, please hang out." And then actually then Jeff Hinton just randomly walked up because he knew David Lowe and I was like a strange face and he's reading my weird company's name affiliation and like I was like, "Oh, Jeff Hinton, my friend Paul says you're really good." And which is like a very cringey thing to say, but he was very gracious and we chatted about the Netflix Prize and everyone was super welcoming in the field. It was really amazing. I remember going up to like a poster and being like, "What's a partially observable Markov decision process?" And I was expecting people to just be like, "Look, man, I don't know how you got here, but like come read a textbook, come back." But they were just like, "Oh, you know," and I'd be like, "Okay, cool. What's Markov mean?" And like, it was really a good experience.
Yeah. It's crazy to think back to that time where people like you were totally calibrated. You're totally on the right track being like, "AI is really important. It's like the most interesting thing in the world. Like all these big consequences are going to come." And meanwhile, everybody else is like, "Oh, the internet's really, you know, you can chat and you can like, you know, do deliveries," right? Like everybody's head was like somewhere else entirely and they're like, "AI that's smarter than human? Give that another 500 years," right? Like it sounded like science fiction at the time.
Yeah. And I think it was generally seen as like the land of failed dreams and sort of Schir's like grift, right? It was like if anyone's talking about AI, they probably just are some sort of diehard true believer like your contributor or some sort of grifter. But it wasn't even an effective grift at the time because no one would take you seriously, right? You couldn't even raise money on it. And in a sense I kind of got lucky in the timing in the sense that like so I guess I went to grad school at first in 2008 and things were sort of just starting to work and like you know deep learning then started working around like let's say 2010, 2011. It's funny because my advisor at Cambridge, Zoubin Ghahramani, had actually done his postdoc with Jeff Hinton in 2003 and he had run a lot of deep learning experiments on his laptop overnight that just didn't do anything because the computers were too small and then later people ran the same experiments in like 2014 and got like you know amazing results and everyone was like wow you're such a genius and I think he was like I think that's a good method too. In fact I was too ahead of the curve.
Yeah, I definitely remember living in that era where I was never like a deep machine learning researcher. I did undergrad. I studied a little bit of AI in undergrad. I was certainly keeping tabs on what people were saying on LessWrong. And my intuition was, I got to admit, I was saying like, I bet you could make like a compact AI that could just like understand Bayesian reasoning and just could have like a very small code base and didn't need a lot of data. Like that was my intuition. And then in retrospect, boy was that the wrong intuition. Like it's very clear that like having a lot of neurons and a lot of dimensions like that really helps you just like represent a bunch of fuzzy concepts and have like a ton of context and it's probably hard to do without like a good amount of that. I think Eliezer Yudkowsky would admit to similarly having the wrong intuition and of course I'm sure everybody's going to dogpile and be like see doomers know nothing but I think that aspect of things where you know these kind of fuzzy neuron approaches where like the concepts aren't necessarily like neatly distilled I think that surprised a lot of us that like that turned out to be the most promising direction.
Yeah. So my PhD was in Bayesian nonparametrics and the whole idea there was like it was kind of like scale before scale was cool in the sense that like we were trying to build models that had sort of infinite capacity and then just as more data came in they would build more and more complicated hypotheses and could scale to arbitrarily large data sets and the main problems were we didn't really know how to do scalable inference in these methods. So things would just get slower the more data there was, and we didn't know how to build these sort of like hierarchical let's say compositions of different ideas or structures and like in the way that deep nets really do get to break the problem down and say like first I'm going to find like who the objects of interest are and then I'm going to think about the relationships then I'm going to think about what the task is and that was something that people barely knew how to do at the time and they certainly didn't know how to combine the thing that scaled with a lot of data and had this like hierarchical structure right so it took me until about like 2015 to really jump on the deep learning bandwagon and it was kind of seen as almost this like déclassé thing because it was so engineering heavy. There was not very much math. There was not very many first principles. It was like yeah it's terrible to use deep learning because you have to take some poor grad student and say fiddle with the hyperparameters and the architecture and the regularization for like 6 months and then you might be able to get sort of state-of-the-art results but just barely. And it sort of kind of seemed like if you don't have any real ideas, you're just going to brute force your way out of the problem. And the sort of academic mind was like, "No, no, we're going to like use our amazing prescient vision to like see the like, you know, piercing beauty of reality, do a bit of math, and then code up this amazing algorithm that will like capture that aspect of the world." That sounded like a much more appealing way to do things than like just throw more neurons at it and then make the poor grad student like iterate.
Exactly. Exactly. And this is what Richard Sutton calls the bitter lesson, right? Like nope, just do it the other way.
Exactly. Exactly. So anyway, that's that's I'm just like have spent a lot of time doing postmortems on like why did we all or why most of us, you know, Jeff and others not included, sleep on deep learning and I think part of it was just the aesthetics of it.
Okay. So you were very early to just realizing that AI was a big deal and that it might work soon. I'm curious how your stance evolved on like the X-risks of it all. For instance, like in the original 2008 Eliezer Yudkowsky versus Robin Hanson AI foom debate. Do you remember if you were more sympathetic to Eliezer or Robin or were you even following it at the time?
Oh, I definitely followed it and I was more or less more sympathetic to Eliezer and I guess I felt like it was hard I think at the time to even understand what time scales they were talking about because but I think Eliezer was coming down on like the like days time scale even. I honestly I don't remember all that clearly.
He was coming down on which time scale?
No, I think Eliezer was coming down on the like this bootstrap process could happen over the course of days and I think...
Oh, days, right, for the foom time.
Which I think now we're all like haha. I mean I guess again I don't want to rule it out in the sense that like we have not observed it so far as weak evidence but in some senses I think Robin has a point just in that we were like okay now we can see the sort of like technical depth that we think is still required for like the next...
Well okay, we're talking about foom times though, it's all about where you draw the starting line, right? I mean if you draw the starting line from GPT-2 then yeah okay foom has taken... Right? But or then you could draw the starting line in like 1950 or whatever, right? Like the first computers. Or you could just draw the starting line the moment that AI reaches the definition of AGI. The moment that OpenAI's contract with Microsoft says is an AGI. Maybe we should put the starting line there.
Yeah. Yeah. And to be honest, like I don't think I have that strong opinions or informed takes about the mechanics of foom. And I think most of the stuff that I try to reason about in terms of like policies just tries to be pretty agnostic to the speed of foom, right?
And just in case the listeners aren't familiar with the terminology, foom refers to kind of like the sound that a nuclear bomb makes, I guess, when it's first starting to explode. And the idea is that if AI gets into like a singularity mode where it's like improving itself really fast, the progress could just start happening faster and faster like before we know it, like in the blink of an eye or days or whatever, and then we might be disempowered at the end of that. So that's what we're talking about when we say foom.
Yeah. And then I think everyone can agree that if you zoom out a little bit like the industrial revolution was a foom where like you start building you know steam powered mining equipment and then it like mines more coal and then you get this like runaway growth and then the factories...
Exactly. If you zoom out and you just look at the graph of like human GDP by year it's like flat a lot of the time and then there's like a hockey stick around the industrial revolution. were from the perspective of like a caveman or from the perspective of like other primates or like the buffalo who thought that they had a bunch of territory to control. It's like nope. Humans like foomed and wiped them out and they just didn't have time to react. They didn't have time to develop different genes that would let them fight the humans. They were just wiped out.
Yeah. Yeah. So, but maybe another way to answer your original question about like how my views changed. I think I've always been pretty sympathetic to like the overall idea that like there will be very powerful AI at some point, not having a strong opinion on when, and getting this right is sort of like such a huge problem. But there was just a really long time when I was in the field and it wasn't clear what to do about this. So like in 2015 I was one of the co-organizers of this like I think it was called like Reliable Machine Learning in the Wild workshop at NIPS which was our compromised title for what we wanted, like me and some more like extra kind of people wanted to be the like AI X-risk workshop but like there just wasn't much work to talk about then on this task and it's not clear what you can even say about it and there was a lot of more concrete work of like how do we make sure that the surgical robot is safe and that it doesn't like cut the wrong thing and like or like self-driving. And so I caved at how much we kind of ended up blending these two concerns into this like safety world. But at the same time, there just wasn't much to do. I guess also in 2017, me and Roger Grosse, my colleague at Toronto, both took one PhD student each to work purely on like big picture safety stuff. And they were both very strong students. Actually mine had also been admitted by Hinton, which was which was which like, you know, no-brainer. You should work with Hinton, but he wanted to work on safety. And I think they both, it was actually very tough for them. They did good work, but the kinds of projects they could do were make a demo to show that the incentives faced by an agent would actually be bad in the sense that if they lied in some like iterated reinforcement learning setting that they would get a better result if they like somehow manipulated outcomes. And it's just like not very compelling stuff. It's like, oh, I did some math to show that misalignment is a danger. It's like, well, who are you going to convince that wouldn't already be just convinced by like just the plain English argument was the central problem with doing alignment work at that time.
Right. Right. Right. Okay. So, fast forward to 2023. At this point, you're really starting to see that it's crunch time because AI is advancing really fast and it seems like the risks are quite high and nobody really has good mitigations for the risk. And that's why you decided to talk with a bunch of these AI companies and see which one you could join, which one you could help with. And we talked a little bit about that, but then talk me through kind of like 2024 to now when you're like, okay, we got to even change the plan again.
Yeah. So, I guess I'll say I was really impressed with the people at Anthropic and how seriously they were taking these risks and their technical agenda for like, okay, we're going to make sure that these things like refusal aren't happening. We're going to actually work out a concrete set of regulations and evals. Like it just seemed like very important work and I still endorse it. But I guess even at the time when I talked about AI scenarios I was trying to say like none of our governance is set up for this scenario where the humans are not this like load-bearing part of our civilization and it was the thing is it's not really clear, it's even less clear what to do about this kind of worry. So I was like, well, I'm a technical guy. I should just go and do something and hopefully get a better understanding of the landscape and also hope to find somebody who did have a plan. So this was something I started in like Christmas 2023. I was in England for a little while, visited DeepMind, visited all sorts of researchers also in the Bay Area. And just, you know, every smart person that I hadn't already been talking to regularly, I was like, "So what is your plan for how this goes well?" and just hoping that I still just hadn't found this person and for whatever reason they weren't like posting about it. And managing to just get a whole bunch of like, oh yeah it's really tough, or head in the sand like, oh it'll be fine for this reason. And then yeah hoping to do like whatever concrete thing I could at Anthropic.
Right, so you basically had an emperor has no clothes moment, which is like you didn't find a single individual who had a good plan for how to develop super intelligent AI and not immediately have that be a catastrophe basically.
Yeah. Yeah. And so sort of like the best case outcome is that I talked to people who were calibrated enough to realize that they didn't have an answer. And then there was just, you know, and I gained respect for those people. And then the people who are like, "Oh yeah, it's like going to be fine for this reason." And I just kind of like challenged them a little bit on it. Like I don't have a better idea, but your idea has this obvious flaw. And they're like, "Well," like they would just kind of dismiss it. And I was like, "Oh man."
So how did you decide to leave Anthropic?
Oh. I mean so I just had my fourth kid like a year and a half ago like during my time at Anthropic and the like the team was growing, the pace was intensifying, and I was like okay I was doing quarter time in SF and three-quarters time in Toronto and it was like really hard on my wife to like leave her with the three kids and the newborn. So it wasn't like Dan Kokotajlo, like you know, there was it wasn't drama. It was just like okay man I just like everyone here is working like 80-hour weeks. I'm like the sleepy old guy like and I mean I'm happy with the work that we did there. Obviously like we got a lot done and but it was just like I just needed to like not be massively overemployed because I still have my lab. I still have like a few other commitments.
Yeah. I have three small kids and I work from home and that already seems pretty tough. So, I definitely hear you on that front.
And this is an interesting example for the viewers that David and I are on the same page on a lot of these X-risks and like thinking that the future is going to be rough and yet I guess we're both on the same page that you might as well have small kids in case things go well, right?
Yeah. In fact, that's one thing I kind of try to adopt from VCs is like imagine that things could actually go well, right? Like just try to take it they might, we're not saying they definitely won't, right? And I'm also sympathetic to Andrew Critch's argument that like if you don't provide a positive vision or like have these sort of positive feedback loops going that make keeping human civilization alive this like appealing thing then you're kind of already doomed. So you so that's like actually a necessary step along the path is to like have in place like if we do succeed this is how things are going to be well and not everyone just like oh crap we tried to give up on civilization and now we wish we hadn't.
Exactly. So you and I have implicitly done this expected value calculation where we didn't plug in 100% chance that everybody's going to be doomed and that's why we shouldn't have kids. We left some probability of a good outcome. Which brings me to the biggest question that everybody wants to know. Are you ready for this?
Professor David Duvenaud, what is your P(doom)?
So, I'd say something around 85%. However, wow, it's pretty tough because a lot of this P(doom) isn't necessarily just about AI. It's more of like the future will not need us probably in one way or another. And this is the same position that sort of every animal has ever been in like through prehistory is like you have to either at the very best adapt into something unrecognizable to yourself or just become extinct. There might be some third option now. Might be sort of like competent enough as a civilization to do some sort of like lock in or leave some legacy or find some other way to thread the needle to say we're not going to just be replaced by more competitive von Neumann probes, but we're also not just going to die. There might be some intermediate thing and that's the main thing that I want to spend time working on.
It's an interesting way you put it where like every species has to expect to either like change and become unrecognizable or else die out. It's kind of like when people say you either become the hero or no, it's like when people say you either die a hero or live to become the villain. It's like you if you live a long time you're going to become unrecognizable.
Yeah. Exactly. And so I think you know some of my smarter friends when I say like I'm really worried about AI doom and then I explain they're like but wouldn't you also be worried about just the normal future? Like Robin Hanson also like you know challenged me on this and I'm like yeah I think if we just let like cultural and genetic evolution run long enough I would be like oh man my descendants are like some weird thing that don't seem valuable to me like I'm not sure right it depends and this is to be honest a tough question and Robin Hanson makes a lot of hay out of this question he's like tell me what the ideal future is for your descendants where are they allowed to evolve to how are their values allowed to change and progress and it's really tough because it's like you know things like well don't you think that they should like have empathy for each other and value human life, but it's like how sure are you about that stuff? Like what are your core values that you think absolutely there should be no compromise for? And then you run that danger of like well if you'd asked like a hundred years ago somebody be like well God says you're not allowed to have homosexuals so that's like one of my core values and like wait right. So you want to make sure you're not accidentally you know thinking things are too important when they're not.
Yeah. Well, one thing that I think Robin convinced me of that I'm grateful for is pointing out that there are different baskets of goods that I currently or sorry baskets of values that I currently have that are probably incompatible with each other in principle. Like I want my kids to like part of the reason my life has been like enjoyable is I got to like grow and adapt and overcome and you know explore new fields and do things differently than before. And also I want my kids to be similar to me and like have similar life in a lot of ways but these things are in serious tension and so like we might have to choose like do we want the basket of goods that's like there's competition and evolution and adaptation, then the only things that stick around are going to be like the competitive things like you know language and sight and like memory or like very abstract things that are very useful, or do we try to keep like you know the family farm going and make sure that you know everyone keeps speaking English in a recognizable way and like all these like very particular values that maybe other things we're afraid of being locked in but at the same time maybe there's like the little things that make up life and we might have to choose between these two baskets of goods like my hope is that there's some sort of synthesis of these things though.
Yeah. And much of what we're talking about of like values changing and becoming unrecognizable, we're actually talking about the good outcome, right? That's kind of we want things to keep evolving and progressing, but we don't want and again this is like the distinction that Robin Hanson kind of makes fun of us for. It's like, okay, we want a lot of evolution and change, but we don't want this really rapid replacement by a successor species. Like we still draw a pretty big distinction there.
Yeah. And I don't think the rapidity is actually that important, at least not according to me. And I also think that Robin's position is a little bit self-defeating to me in the sense that he's like, I think it's very convenient that he's like, if we just allow competition to operate, then there's going to be a lot of value, and if we don't, then there won't. And it's like, right, I feel like you still have pretty strong opinions about what sorts of things produce value and like you want to see those realized. So, I think he's kind of in the same boat as us, but he doesn't.
Well, Robin just gets a lot of mileage out of like this big spectrum where like I agree with Robin that it's really hard to nail down where on the spectrum of like how much we're allowed to change, where we're allowed to be, but I feel like Robin takes that really far to the point where he kind of implies that it's okay if like the world gets like paperclipped tomorrow because that's like on the change spectrum and I didn't tell you where I want to be on the change spectrum, so how do I know that that's wrong? But it's like wait a minute paperclipped, that's gone too far.
Yeah. Well, and my general reminder to people, so a lot of people say, who are we to stand in the way of progress, right? Like Rich Sutton says this. And my general to that is like I think you actually do have strong opinions about what good successors to yourself would be. Like for instance, if tomorrow North Korea started just like taking over the world and like we're going to have like authoritarian dictatorship forever, you would be like no, no, no, not that successor, right? And like in general, if someone says like look, I'm just going to leave things, take hands off the wheel and like whatever replaces me is sort of better almost by definition, I'm like, "Okay, how about you just give me all your stuff because then like by definition I'll be your successor, right?" Like I just The thing is I think people do have opinions, but it's very painful to admit that you're in this situation where you might just be screwed by the tide of history, right? I think people go from it's not happening to it's happening and it's good. Partly because it's just so painful and vulnerable to put yourself in the like no, I'm going to try to stand against the tide of history camp, right? That's just like a bad camp to be in.
Yep. You mentioned Andrew Critch is probably the person with the most similar position to yours. Is that right?
And by the way, Andrew Critch was a guest on Doom Debates. So if you guys like David's position, definitely go listen to the Andrew one and you'll get another person's perspective coming at the same conclusion from a different angle basically.
Yeah, I have a ton of respect for Andrew and he actually wrote this really nice LessWrong post about his theory of change and why he's starting this healthcare company and he basically lays out the main arguments for gradual disempowerment in that blog post. And then I don't think his plan is all that good, but I don't have a better one either. So I don't like anyway I just have a ton of respect for him. I almost feel like I'm just sort of rediscovering stuff that he's talked about and then like years later I'm like, "Oh, that's what he meant." So yeah, go read his stuff.
Right. Right. Right. There's a camp of people that is pretty big that I'm pretty sympathetic to even though I'm not quite in it myself, which is like you might call it by your own title gradual disempowerment, although you pointed out maybe it's not gradual, but it's kind of like this doom that happens not through like attacking us or like taking our stuff, but just economically, you know, like just pushing us out of jobs and we just slowly become like irrelevant and then the AI just controls everything and we just don't have as many babies and that's it, right? That's kind of like your mainline scenario. And the reason I don't really agree to that mainline scenario is just because I just think the effects of super intelligence are just going to be bigger than that. Like they're just going to be like massive effects. Like I just have like a sense of scale that doesn't really agree with that. But I am sympathetic certainly to the possibility like that's, you know, it's way number two that we could fail. And I think both of us agree that these are both failure modes and we just kind of disagree on which one we think is like the most likely failure mode.
Yeah, I guess I would say it's not my mainline failure mode and I would okay like I would say I put a let's say 20% chance on something that looks like just pure runaway like power gathering by some misaligned AGI kind of like in the bad case of the AI 2027 project and then again like maybe something like 20% of the like we actually do solve technical alignment and it's just that on the way from the lab to the government you know somebody just decides like well I want to make sure that like everything runs better and I'm going to take humans out of the loop and then eventually we're all convinced to like get uploaded but we never get run very much or something but and then sort of most of my doom mass is some like mix of the two because the you know the underinvestment in alignment is maybe the same sort of phenomenon as what leads to gradual disempowerment anyways. So anyway but the point is that I definitely I wouldn't say it's my mainline scenario. I would just say we think that we kind of notice that there's sort of almost two distinct alignment problems. One is the technical one and one is the civilizational one. And the technical one is neglected, but then even more neglected is the civilizational one, but it's even harder to do anything about.
Just to review your pie chart here about your 85% P(doom). You're basically saying 20% out of the 85%, 20% is a kind of like a rapid foom scenario. Just AI is kind of overwhelming.
Yeah, not necessarily rapid but more like the agency, the thing that's agentic is the AI.
Yeah. And it's power seeking. And then you've got 20% that's kind of like a top down decision from some human that's able to wield the AI to cause problems.
Well, I should no, I should say more like that the extreme 20% of gradual disempowerment is like there's no one in charge. We just have a competitive world stage and every government is forced to use the AIs to build robot factories and robot armies and at some point the humans just no one can advocate for keep continuing to give them resources. So they're forced to be like uploaded or just marginalized so much that it doesn't really matter what they do anymore. That's like 20%.
And then what was the big chunk? What was the 45% out of the 85%?
Well, it's like a mix of those things where there's AIs that like are sort of power seeking, but they're sort of in cahoots with some more agentic version of our civilization. So, maybe there's like a global power and then they end up calling the shots in a way that still disempowers people.
So, the distinction, the bucket is sort of like is where's the agency? Is it in the AI or is it in the government that's like running the AIs or is it just in competition where there's like not really much agency at the very top?
I see. So that most agency is either in the AI or in some kind of top down human government that has AI to work with or just like distributed throughout the economy, diffused agency basically.
Yeah. The last one is like Moloch is the agent or whatever.
Okay. So that's definitely a good crux of disagreement. I mean you and I don't disagree strongly with each other but I guess the biggest crux of disagreement is that I'm imagining the agency is pretty quickly going to flow into super intelligence just because it can. And if it can, it will is my view because it's just like...
Look, you're selling me this thing is triggerable and nobody's going to hit the trigger. I think somebody's going to hit the trigger or it's going to like hit its own trigger. Like the fact that super AI agency is possible to me implies that it's likely probable. You know, that's my perspective. Um, and you're like, 'No, no, it's probably more likely diffuse agency and it'll take a little bit more time and and like humans will still be like less and less economically.' No, no, no, no. That's the 20% is the diffuse agency. The 45% is like basically between the 12 and technical alignment and it being deployed there's some sort of like unipolar government that is uh ends up having the super intelligence basically doing its bidding. Um, but just keep top down, top down human government type agency empowered with AI. That's your 45%.
Oh god, I got it. Okay. So you got 20% on both sides. You got 20% on the AI has a ton of agency and you got 20% on the diffuse agency and you got 45% on the like human governments have a lot of agency and then AI kind of does their bidding but it's a disaster.
Yeah. And the fact that these round numbers should like take you to the fact that like I've tried to do a little bit of calibration practice but like I don't think anyone should take these numbers super seriously. This is just like but it's still my all things considered uh very not expert opinion. Like I'm not an expert on like civilizational dynamics.
Okay. Well, you should come back when you're able to get that down to a few tenths of a percent because I don't like these round numbers.
Yeah. Well, I mean, that's one of the main things I'm hoping to spend my time over the next, you know, 6 months or a year thinking about.
So, nice. Okay. So, your total P doom is 85%. Would you sign the statement on AI risk saying mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks such as pandemics and nuclear war?
Yes, I did sign that statement. In fact, I was involved in getting some of the early signatories. Um, so I'm absolutely very cool.
Do you often collaborate with the Center for AI Safety at Dan Hendrycks' Institute, basically the one that was driving that letter?
Um, I guess I'll say at the time there was just not that many people around who were sort of like had some credibility and were, you know, willing to sign these like then insane sounding uh statements. So I was involved in drafting a bunch of similar uh things at the time. So, I don't have any other formal arrangements with Dan Hendrycks' Institute, but like I'm on like the mailing list or whatever.
Uh-huh. Let me ask you the Tyler Cowen question of what is the David Duvenaud production function because I notice you're kind of like Dan Hendrycks in terms of like being unusually productive just like putting all this stuff out like how do you do it?
