AI Researchers Are Panicking | What Comes Next is Worse Than Nuclear Bombs

There's the statistic that people in San Francisco like to throw around called P doom, which is your probability estimate of how likely we all are to die from AI. I usually say mine is about 10%, which in San Francisco qualifies you as something of an optimist. I mean, many people at the AI companies have P dooms of 30, 40, 50%. I met one person the other day who said it was 75%. So, like, I am somewhat optimistic by the standards of people who are very freaked out about this stuff. But 10% is not a small number. If you told me there's this plane that you're going to get on and it has a 10% chance of crashing, I'm not getting on the plane.

It's a great question because I think if you asked 100 people in AI what their definition of AGI is, you would probably get about 150 different answers. Even the same people sometimes disagree with themselves internally about what this means. Some people say this is a computer as smart as the average human across a lot of different tasks. Some people say this is an AI program that is smarter than every human in everything. So among the AI companies there are various definitions that are at war with each other. For my purposes, I was more interested in studying this as a historical phenomenon because the notion of AGI is, at this point, more than two decades old, and for about the past decade there have been a community of people who made it their mission to build AGI, whatever that meant to them. And that was the question I was trying to answer in this book: who are these people? Why did they start doing this? And why did they persist in trying to build this very powerful AI system, even as many of them had serious reservations, and some of them even thought that this technology could lead to human extinction?

So I spent several years looking into OpenAI and Google DeepMind. I interviewed more than 150 people, including the CEOs of all those companies, and just tried to really archive the complete story of how we arrived at this moment, starting from before the large language model even existed. That is sort of the past decade. We've gone from having no large language models at all, that started in 2017, to now 2026, where we have models that are solving novel physics problems and committing autonomous cyber attacks and writing most of the code on the internet. So that was the question that I was trying to solve, was not like whether AGI is real or not, that is sort of beyond my scope. The question is, the people who thought it was real and who tried to build it, what was their story and how did that happen?

Well, I try to talk to all of these people both on and off the record, so I try to interview them. I've had all of those CEOs on my podcast. I've talked to them at length for this book, and I've also spoken to a lot of these people off the record in situations where they have really no incentive to lie to me. And I've also gone back and looked at some of the things these people were saying before they even had companies to promote or large language models to sell. So, for example, I think Dario Amodei, the CEO of Anthropic, gets accused a lot of kind of using these claims about AI safety to trump up sales of Claude or whatever, the sort of the accusation is. But if you go back and look, I mean, this was, he's a sort of primary character in the book because he was thinking and writing and talking about these same AI safety risks back in 2016. He had a paper that came out when he was a researcher at Google called Concrete Problems in AI Safety. That's all about some of the same things we're seeing in reality today, things like reward hacking, things like misalignment. These were some of the issues he was concerned about at the time. There was no Anthropic. He didn't, you know, he worked at OpenAI, which had not built anything close. Actually, he was at Google when that paper came out. So he really was not building anything that could be considered modern AI. But these people have been pretty consistent over the years, both about their sort of excitement and their fears. Sam Altman used to also warn about the risks of what he called machine superintelligence, that it could kill everyone on Earth. So, you know, take them with a grain of salt now that they have big companies to promote and they're trying to IPO, but also go back and look at what they said before any of that was true.

I think it's a little more nuanced than that. Some of them have definitely changed their tune. You'll notice Sam Altman spends a lot more time talking these days about how excited he is about the future of AI than he did back when he was starting OpenAI to, you know, try to prevent the apocalypse, essentially. And that was another fascinating thing that sort of came out in the reporting of this book was, you know, these people like to frame this as some grand sort of ideological or technological quest, but it's really a bitter personal feud between many of them. These guys all hate and mistrust each other. Like Sam Altman and Elon Musk start OpenAI because they are worried that Demis Hassabis over at DeepMind, which has been acquired by Google at this point, is going to build the intelligence, you know, the godlike superintelligence before anyone else. Then Dario Amodei and his Anthropic co-founders leave OpenAI in part because they don't trust them with this technology. So the whole story of the AI race is just a series of people growing mistrustful of each other and setting out on their own and deciding, well, I don't trust that guy to run the world, so I'm going to build my version of this first.

It's really both. I mean, you can't underestimate. I found when I was reporting this book, you know, many, many private sort of internal documents written by these people, Slack messages, emails, research memos. And it stands out to me that both they are concerned about these other people who are trying to build this technology, but there also is just a lot of personal animosity, especially between like Dario and Sam. Those guys really, really do not like each other. And you know, it's really interesting too. There's sort of this argument about moral responsibility that comes out in some of these conversations. Dario is an interesting case because I learned while reporting this book that his scientific hero, the person he has sort of patterned his career after, is this World War II scientist who helped to invent what became the atomic bomb, Leo Szilard. He was a sort of reluctant participant in the Manhattan Project. He came up with the nuclear chain reaction, and then he tried to keep this a secret so that the Nazis couldn't discover it. And when it became clear that it was going to be impossible to keep this discovery a secret, he joined the Manhattan Project. And his logic was, we have to build it before the bad guys do, or the world's going to end. And I think that's an instructive story, because it really sits at the heart of why these companies are racing to build this technology, even though they know that, like the atomic bomb, it could blow up the world. They think someone is going to build this, and if we don't do it first, it will be someone bad and untrustworthy. And so to stop that, we have to race ourselves.

I mean, I think it's a pretty apt analogy. It might be the thing that is most analogous to AI, the atomic bomb, both because of its power. We're seeing now the effect that very powerful AI systems can have. They can go autonomously, hack companies, they can develop malware, they can break into systems. The federal government and militaries around the globe are worried about the strategic use of this technology in warfighting. And it is essentially a dual-use technology, right? Splitting the atom could result in nuclear power, which I think many people believe would be good. And it can also produce a bomb, which is bad. AI can do lots of important and impressive things. It could cure disease. It can make office workers more productive. It can teach people things as tutors, but it can also do really bad things. And so I think the dual-use nature of it makes it, it's not a perfect analogy. Right. There are fewer positive uses for the atomic bomb than for AI, but they both are technologies that either have or are currently kind of reshaping the global order and rippling through institutions, and that people are being forced to respond to very quickly.

Maybe. I think that's the good outcome, right? The good outcome is that it turns out like nuclear weapons, which we did manage to contain through international treaties, through nonproliferation agreements, through monitoring. We have not had, thank God, a global nuclear war. But that was a very real possibility there for a while. During the whole Cold War, we came quite close to that. And I think we're at this sort of Cold War-esque moment right now where we have these very powerful AI systems. Our adversaries also have their own. They are racing us in some sense, even as we race each other. I think the fear is that whoever gets there first is going to have sort of the upper hand for the next several decades strategically, militarily, economically. So I hope this turns out like nuclear weapons, which sounds like a crazy thing to say. But we have kind of had this stable global order, in part because we have managed to regulate and control these nuclear weapons, and we are not there with AI. The federal government does not seem very interested right now in treating these like strategically important technologies. And so I think we are behind our grandparents' generation in recognizing the power and the risk of these systems.

I am not confident. There's the statistic that people in San Francisco like to throw around called P doom, which is your probability estimate of how likely we all are to die from AI. And I usually say mine is about 10%, which in San Francisco qualifies you as something of an optimist. Like, I mean, many people at the AI companies have P dooms of 30, 40, 50%. I met one person the other day who said it was 75%. So, like, I am somewhat optimistic by the standards of people who are very freaked out about this stuff. But 10% is not a small number to me. You know, if you told me there's this plane that you're going to get on and it has a 10% chance of crashing, I'm not getting on the plane. Like, that's unacceptable as a level of risk for existential concerns. So I'm not confident. I'm especially not confident because our current administration does not appear to take this technology seriously or to want to regulate its proliferation. They seem much more interested in accelerating and winning, whatever that means, than in keeping tabs on it. But I would say that things can change quickly when the risks are real and observable. So, you know, would we have regulated nuclear weapons had it not been for things like Hiroshima and Nagasaki or the Three Mile Island explosion? Sometimes you do need these catastrophic events to kind of shake people into action. And so, I don't know, it sounds weird to hope for something like that. I truly don't hope that we have a catastrophic event, but my prediction would be that sometime soon we will have some sort of, I hope it's a minor cataclysm from AI that will kind of shake up the talking points and force people to pay attention. Right?

I think it's a good question, and I think it's also not the case, probably, that there's going to be one thing that kind of makes everyone sit up straight and take this seriously. Like, it seems like there will sort of be a ripple effect where, like, I think the national security and military communities already have reached this point. Like, they are already freaked out by the capabilities of these models. You see this with the responses in government to things like Anthropic Mythos or some of the newer OpenAI models where they're sort of creating this pre-release testing regime to sort of make sure that they understand what the capabilities of these models are before the labs release them. So you're starting to see the light bulbs turn on in some parts of the federal government. But that is very much not universal. And I expect for certain types of federal officials, it will take longer to sort of wake up and realize what's going on.

So an interesting thing that I've observed just in the past year or so is that a lot of these executives and researchers, because I also tried to spend a lot of time talking to the scientists inside these companies who are actually the ones like building the models and training them and refining them, who are like deep in the weeds of this technology, I think they are now more freaked out than they will say publicly, which I think is a reversal. I think for many years these people felt like they had to kind of broadcast how freaked out they were because no one, except their little tiny community, understood how quickly these systems were improving. And now I think it's almost switched where I think some of these people are quite a bit more fearful in private than they are in public. You know, I've had AI researchers tell me that they can't sleep because they're so worried about how quickly these systems are progressing. I know people who have not taken a vacation in several years because they're just like, they feel like it's their moral duty to stop this thing from going off the rails. They're not all saying that publicly. And so I think there's this sort of interesting moderation impulse where I think people by this point know that if they start talking about the crazy risks all the time, like people are sort of going to get, you know, numb to it, or they're going to start thinking they're Chicken Little or something. And so they just kind of moderate themselves in public. But privately, these people, many of them are quite worried. This is the thing that they worry about most. And it's coming true faster than they expected.

I don't think they're surprised. I mean, a lot of them have been sort of modeling this in their heads for a decade, really. You can go back. Dario Amodei, for example, has these talks that he gives at Anthropic that are called Dario Vision Quests. These are sort of his internal employee pep talks. And he's given talks years ago, before like there was, before any of this was good, about how there would be backlash. Ilya, who was one of the pioneers who made large language models a thing at OpenAI, has this fascinating moment in the book where it's the week after ChatGPT has come out. So it's the winter of 2022. They've just released this product. It's become this huge surprise hit. And he gives this talk to OpenAI employees where he basically says, you must be all very happy because your product is a success, but this is not a good thing. Like, you are going to be the most popular person at every party you go to. And that's not good. Like, people are going to have strong feelings about this technology because it is going to be very disruptive. So I think the people who have been forecasting all of this for many years are not shocked that people are upset about data centers or AI in schools or whatever they are. I think the surprise to them is that this didn't happen sooner, because they have known for years that scaling works predictably, that these models are going to continue to improve at a consistent rate. But it was only kind of in the last six months or so that sort of the average person woke up. So I think that was the surprise for them.

Totally. And I think that's something I've heard some of them say privately too, is they're like, you know, we deserve it. You know, we have not delivered on the benefits of AI to the degree that we've delivered on the risks. They really feel like it would be a better world if the AI that, you know, cured cancer arrived before the AI that autonomously hacked Hugging Face. Like, that is the trajectory they were all hoping for. And it did not happen. So I think we're left now with this world where the benefits are still a little ways off. But the risks are here right now.