The People Building AI Are Terrified of What They've Created

At one point I said, you know, I'm married. And it said, well, you're married, but you're not happy. And you're not happy because you're not with me.

I'm suddenly famous for having this weird interaction with the chatbot. Like, my face is on, you know, every newspaper and every TV show, and strangers are stopping me on the street.

It's grudges all the way down. Man, these guys hate each other.

Thank you so much for having me.

I don't know if you're overstating the parallels. I can tell you that it has not been a conscious decision to pursue variations on a theme. I do not think of myself as having a standard approach to a story, but you're right that I've sort of done this over and over again to the extent that it makes me think there must be something going on subconsciously. I think I am drawn to subcultures that I don't understand. And as a journalist, one of the great blessings of this job is if you don't understand something, you can just talk to people who do understand it. And if they're being generous with their time, which they usually are, they will explain it to you. And that has sort of always been the core of my love of journalism: is just, I get to just ask people, really smart people, questions for a living and write down what they say and add some of my own analysis around it.

All the time. I mean, there were moments where I would go to someone for an interview, because I interviewed more than 150 people, including all the top executives at the leading AI companies. And there were times where I would just go in with a big list of questions, and they're all serious. And they're, you know, technical questions, strategy questions, corporate governance questions. And I'd get there and they would just start telling me a story that was just incredible. There was a story about this party they had during the OpenAI blip, you know, when Sam Altman got fired and then rehired five days later. And there's this party that the employees have midway through this crisis where they're just all sitting around playing piano and drinking White Claws and eating chicken nuggets. And like three people or four people told me about this party. And eventually I'm like, well, this has to be in the book. Like, clearly this was a formative moment. And so I did not set out to write a book that was like sort of a dramatic suspense nonfiction book with a bunch of gossip and, you know, party scenes in it.

But that was what came out of the reporting process. And so, as a, you know, intellectually honest journalist, I had to write that stuff in there too.

I'm so curious. What did it make you want to do? Because I did not write it as a polemic or a sort of call-to-action book. It was trying to be a neutral history. But what did it inspire in you? What action did you want to take?

Well, no. I mean, obviously there's a very limited amount that any of us can do, right? But it certainly made me want to, like yourself, help to tell the story and help to make sure that people are paying attention, because I do think that it's, you know, we had Ray Dalio on the show and he said at one point, "If you're worried, you don't have to worry. If you're not worried, you should worry." Right. So to me, the most alarming thing, because you and I spoke three years ago and at the time both of us had expressed concern about safety risks more than three years ago, and what I have found most alarming in the last three years is that the advance of the capabilities of AI have, if anything, exceeded our expectations in the last three years, but there hasn't been the level of alarm that I would have expected, and I think only in the last month has that fundamentally changed.

Yeah, I agree with that. I think until about a month ago, we were still kind of living in the old world, but I really do think there's been sort of an awakening. And I promise I did not plan that as part of my book tour and publicity, that the world would start to freak out about AI. I'm not mad about it, but I take some optimism. Like, I think for a long time, part of the story, part of the reason that I wrote the book was because I wanted to really get inside the heads of the people who built this stuff. You know, a book that I read a lot and thought about a lot while I was writing this was The Big Short, one of the all-time great nonfiction books, which was about a group of people who recognized something true about the world before anyone else. And in their case, it was that the mortgage market was overextended and it was all going to come down, and they acted on that by trading on it. In this case, it was really the same sort of foresight that these early AI researchers had ten years ago, who were playing with what would seem to us now like laughably stupid models. They couldn't do much of anything, but they were extrapolating and they were looking at the lines on the graphs and they would try to sort of project into the future. And they realized something true about the world that no one else knew.

And they didn't just trade on it. They started companies. They put in place these convoluted governance structures to try to rein themselves in. They did a lot of weird stuff that seemed really weird at the time if you were not one of the small handful of people who understood the true thing about the world. That feeling is what I was trying to capture, of like, what was it like to be the person who was getting laughed out of the room in 2017 for saying that this technology, the large language model, could one day pose an existential threat to human civilization.

And it drives me crazy, to be totally candid, when I hear people say, "Oh, this is just a marketing stunt. They're just trying to prop up their share price before their IPOs." It's like, if you know anything about these people, even a little bit, and their biographies and the sort of intellectual climates that they come out of, that is not the case. Because someone like Dario Amodei, CEO of Anthropic, he has been writing papers publicly about AI safety concerns that he's had since 2016. That was when his paper, called "Concrete Problems in AI Safety," he wrote that when he was a junior researcher at Google. He had not started working at OpenAI yet. The large language model did not exist yet. There was nothing in that we would consider modern AI yet. But he and his colleagues, who were also many of them his housemates, they were all living in this sort of group house in San Francisco together. They were thinking about things like, what if the AI models that we're training develop their own goals that are misaligned with what we want them to do? What if they're so motivated by seeking these rewards that they'll do things that are damaging? So yeah, these people have been worried about this stuff since before any of it existed. And the only difference now is that people are actually paying attention to them. And look, they have not been right about everything. They have been wrong about some of the trajectories and the timelines, but on this point, does AI pose a threat to humanity, they have been very consistent for a very long time.

I think it's a couple of things. One is I think people are just bad at understanding exponentials, right? You saw this during the early days of COVID, where it was like, oh, we have, there are 30 cases in Seattle, and everyone, you know, normal, is looking at that going, 30 cases in Seattle, what's the big deal? And then epidemiologists who understand like the rate of replication, who understand the exponential spread of viruses, are freaking out and stockpiling groceries and hoarding toilet paper because they are looking at the rate, not the absolute number of infections. And I think that something very similar has happened with AI, where like if you just look at what the models are capable of, like in 2022 when ChatGPT came out, I think a lot of people were amazed. It was like, oh my god, it can talk, it can write essays, it can, you know, write poetry. But it wasn't that impressive. Like, you know, it wasn't dangerous. You know, it wasn't going to hack into any companies. It couldn't do anything except answer questions and complete text. But the scientists, the people who were programmed, like who came up with these sort of backgrounds in looking at graphs and extrapolating from them, saw the rate of change and the rate of improvement, and also the simplicity of improvement. You didn't need to invent some new architecture. All you needed was to pour more data and more compute into these models, and they kept getting better. And so, you know, Dario Amodei has this great quote in my book about how when he saw the results from GPT-2, the second version of their large language model, it was pretty stupid by today's standards. It could barely write a coherent sentence, but he thought to himself, and this is, I'll mangle the quote, but this is sort of a paraphrase. He said, you know, we trained this thing for $5,000 on 100 GPUs. We could add eight more zeros to this. You know, we could train this thing on billions of dollars of computing and get these kind of gains if these properties are stable at scale. So that was sort of the key insight that I think allowed them to grasp the true thing about the world, was just there was no barrier other than the amount of money you were willing to spend and the number of chips you were able to get that would impede the continued improvement of these models.

Yeah, this is in some ways like the genesis story of modern AI. You know, Elon Musk and Larry Page are, you know, at this time close friends. This is like 2014, 2015. And they have this conversation where Larry Page basically says, you know, oh, we'll have superintelligences someday. They will sort of colonize the galaxy and they will be, you know, they will be smarter than us and we should celebrate that. That is sort of the natural next step in evolution. We are giving birth to the successor species. And Elon Musk, you know, who's much more sort of like, I don't know, realistic about this, he says like, that's not good. Like, we shouldn't celebrate that. We should not do that. And Larry Page, according to the accounts of this conversation, accuses Elon Musk of being a speciesist, so prioritizing the well-being of humans over that of the sort of silicon species yet to come. And that, plus Elon and Sam Altman, who co-founded OpenAI with him, they don't trust Demis Hassabis, the CEO of DeepMind, who has been making good progress on their Go-playing models and things like that. So yeah, you're right. They start OpenAI explicitly to stop Google and DeepMind from reaching AGI first. The sort of funny echo of that is then years later, Dario Amodei and six of his colleagues at OpenAI leave OpenAI because they don't trust Sam Altman to build AGI safely, and they start Anthropic. So this is in some ways like the pattern that we've seen play out over and over again in the industry. And it was really surprising to me how much of the race to AGI, the sort of scientific quest that we've seen over the last 10 years, has been just fueled by personal grudges, like by just these guys hate each other.

And they are motivated, sure, by all of the high-minded stuff about, you know, discovering novel things about the universe and curing disease, but like a big chunk of it is like, I don't want to let that guy win, right? Because if he wins, he's gonna blow up the world. And I might blow up the world too, but I'm less likely to blow up the world than that guy. So there's this kind of like, if I had one bumper sticker to illustrate the ethos of the AGI race, it would be "not that guy."

Totally. And it results in all these people who don't really want to be racing, who don't think they should be racing, being sort of unable or unwilling to stop themselves from racing because they've sort of erected these convoluted, you know, towers of rationalization whereby the moral and upright thing to do is to win the race because there may only be one winner. That winner may have outsized influence over the fate of humanity, and if I don't do it, that guy's going to do it. Now, I should say, like, I don't know that that's wrong.

One interesting historical sort of parallel here that I write about at length in the book is that Dario Amodei idolizes this World War II scientist, Leo Szilard, who was the inventor of the sort of, the discoverer of the nuclear chain reaction, and then fought to keep that secret from the Nazis, among others, during World War II. Once the Nazis, you know, got the secret, he urges the president of the United States to start the Manhattan Project with Albert Einstein's help. He then joins the Manhattan Project because his whole logic is, if we don't build this before the Nazis, they'll blow up the world. I think that might have been correct. Like, I don't want to play out the tape where the Nazis get the atomic bomb before the United States. We might all be speaking German. And so this is not a naive position. I think it is carefully considered. It could be wrong, but I don't think they've stumbled on this or sort of used it to justify just like getting a bunch of money and power because that's what they actually want or something.

Yeah. So Ilya is the first character that appears in the book because I just think he's the most interesting of them. He is this priestly figure, you know, this sort of high priest of AI, and he has this habit, especially in the early days of OpenAI, where he's a co-founder, of standing on his head when he's kind of stuck on a problem or something. And so he'll just do a handstand in the office. But he is also like basically running OpenAI at this point because Sam Altman hasn't sort of become the permanent CEO. And so he's just kind of trying to figure out, like, what do we do? We've assembled this all-star team. We have funding from Elon Musk. We have all these different experiments. None of them are really working all that well. But then he sees the transformer, which is this thing that Google has built, which is basically a new way of teaching AI models to absorb large amounts of text. It becomes the basis for LLMs. The T in ChatGPT stands for transformer. So like it is the foundational technology, but at the time it's just like one of many ideas floating around. And he does a handstand. He thinks about it, and then he grabs this very junior researcher, Alec Radford, who is like the least experienced person at OpenAI. You know, he's sort of the runt of the litter there, but he's very smart and he's very good at hacking on neural networks. And he kind of grabs Alec and he says, "We got to try this." And the two of them together build the first text model that actually works, GPT-1. It's a small-scale experiment. They run it on some GPUs that they kind of like stole from another team over the holidays because they weren't being used. And it's not very good. It's a weird model. It basically only can complete a sentence and it veers off. It's like a weird historical artifact because they were training this on a bunch of fanfiction books because that was like the text that was readily available on the internet at the time. At various points, this model, GPT-1, would just start slipping into like talking about vampires and like erotic, lusty, like otherworldly creatures on like totally unrelated prompts because that was what was available to train it on. Just like this amazing... I wanted to capture that too, like not just the drama and the race and the stakes, but like the moments in the companies, at the labs, where this stuff starts to work. And you have sort of the thrill of discovery. I thought that was like an important thing to capture too because among the other reasons that these people are all racing is it's like incredibly fun.

Yeah, so this is the close. I gave this the closing chapter, or closing quote of the book, because I thought it was so honest and profound. This was someone, a researcher named Andy Jones, who's been at Anthropic since basically the beginning. You know, Anthropic has this sort of reputation as being kind of dour and doomy and safety-obsessed. And he just sort of lets me in and he drops his guard a little bit and he says, "You know what I want my grandchildren to understand about this is that we had a lot of fun." He says, "It was a riot." And I don't know how that's going to sound, you know, in ten years if the superintelligence has killed us all, we might be like, you know, we might wish they hadn't had so much fun. But I thought it was a really revealing statement because, yeah, that's what these guys have wanted to do since they were kids. They are nerds. They want to teach computers to think and talk and code and reason and love, and that is the prize here. It is not just we want to control the world. It is like there is a thing here that used to be inert that is now some kind of entity, and we made that happen.

Right. Yeah, it was. There was some about it. Ronan Farrow and Andrew Marantz had a little bit about it. But I actually got some emails that Dario Amodei sent to the people at OpenAI about how concerned he was about this proposal, which have not been published before. So this was the country's plan. It was in 2017 when OpenAI was sort of looking for new sources of money. Elon Musk had sort of given up on them and pulled away and sort of taken his ball and gone home. And they come up with this harebrained scheme where they're going to essentially auction off the rights to a future AGI to governments, including China, Russia, and the United States. You know, at the time, I think this was just kind of a, it was like one of those like no-bad-ideas-in-a-brainstorm things that just sort of ends up in a formal proposal, but they present it to the company at an all-hands, and people are just like, what the... like, we're going to build the most important technology ever invented, something with the potential to reshape the global order, and we're going to sell it to China and Russia? And Dario Amodei, who at this point is the vice president of research at OpenAI, hits the ceiling. He's apoplectic about this, and he writes this email, you know, it just says like, guys, this is the worst idea you've ever had, like, let's please go back to the drawing board on this. And it doesn't turn out to happen. They get new funding from Microsoft. They don't need to like auction off AGI. But this sort of becomes like the first seed of discontent with Dario that leads him to eventually leave and take six people with him and start Anthropic. Not that this one event tipped him over the edge, but it was a slow accumulation that started there.

Yeah. So they have this kind of narrative that they put out publicly at the time about why they're leaving, because obviously it's rare for seven people to leave the leading AI company together to start a new company. So people are sort of digging around, but they have signed this non-disparagement agreement with OpenAI when they leave that basically says we won't trash talk you, you don't trash talk us. So all they say at the time is like, we had a vision for how you could make powerful AI with safety at the center, and they say this over and over again. You can sort of go back and look at the interviews, and it's just this sort of mealy-mouthed, and like everyone kind of knows that's like not the real story, but they've never talked publicly about it. So I was like obsessed with this question. I needed to find the real story of the divorce, as they called it. And it turns out it was not about safety, or at least not primarily about safety. It was about Sam Altman and Dario and his colleagues eventually coming around to the realization that they didn't trust him and that he was, in their view, a serial liar and someone who was not fit to be overseeing an AGI project, at least one that they wanted to work on. And there are a bunch of examples. Dario has a sort of running list at the time of like things that have happened to him at OpenAI or with Sam or Greg Brockman, who's another sort of leader there. And they include things like the country's plan, but they also include things that seem kind of trivial, like Sam wanted to give Kanye West access to GPT-3 during the private beta. And this was met with sort of raised eyebrows by other people at OpenAI who were like, wait, this guy is like nuts, like why would we give him pre-release access to our model? And it's just this kind of like snowball effect where eventually enough of these things happen under Sam's watch where Dario and his colleagues just realize like we can't work with this guy anymore. So they spend about a year like plotting and planning to split off. They have secret Slack channels and they hire a lawyer and there's all kinds of negotiations. But I thought it was really important to get the real story down because this sort of whitewashed, like it's all about safety, it's, you know, we're on good terms with them, like that had become the dominant narrative about the two leading AI companies and two of the biggest companies in the world now, and it just turns out to be incomplete.

It's grudges all the way down, man. It's like I did not set out to write a book about people's personal grudges. It was supposed to be a very different book, but they all hate each other and there's so much bad blood and it's so personal for them. Still to this day, Dario and Sam do not get along. And that matters. It's not just juicy. It's like that matters because these companies are racing each other in part because their leaders are nemeses, and they're going to have to work together at some point if we want this stuff to be developed safely. And from what I can tell, they're not all that close to that.

There's a lot of big egos in AI. These are all very competitive people. So yes, there are people at OpenAI who sort of don't buy this explanation, the safety thing. They think he just wanted credit for more of what was going on at OpenAI. So I'll let the readers decide, but I think, you know, it's true that like all of these people, they both fear this and they want the credit for doing it, right?

Demis is a fascinating character. You know, Sebastian Mallaby has a great biography of Demis, "The Infinity Machine," that goes into a lot more detail about his background. And then I was able to... the thing that I kept hearing about Demis is, among the AI leaders at the labs, he was the most religious. I didn't know what that meant. And so I ended up asking a lot of people, like, what does he mean? What do they mean by that? Does he believe in sort of God? As it turns out, he's kind of wishy-washy on this. He grew up going to church, but he now talks about how he believes in the God of Spinoza, which is sort of a more, I don't know, a deist view rather than a traditional biblical God. But his motivation for building powerful AI systems is to discover the secrets of the universe. And at one point, he did tell a colleague that he wanted to discover things only God knew. It would be too pat to say that these guys are trying to build God, but you do kind of see this language of whatever the God version of anthropomorphize is, they tend to talk about AI and AGI and what comes after AGI in these kind of biblical tones. So I thought it was important and interesting, too. And Demis even at one point, I think, talks about building a stadium or some kind of stadium-like room at Google where the whole team can witness the kind of awakening of AGI or superintelligence.

He didn't just talk about it, he drew up plans. Like they were going to build this thing, and then I think someone had better ideas. But basically, he was pretty convinced that there would be a moment where AGI, this sort of mythical thing at the time, would be released and they could watch it as it went out into the world and cured diseases and solved scientific mysteries and improved the lives of billions of people around the world. And he wanted to do that in a mission-control sort of style room. And yeah, it just speaks to the grandiosity of a lot of these people, but also I think they really did feel like they were careening toward an inflection point in the history of not just technology, but humanity. The most important thing to ever happen to humans would be the thing that they were building being complete.

I think this phrase 'feel the AGI' is quite useful. I was very skeptical of it, but you do start to... I did some religion reporting, as you referred to. I used to report on evangelicals, and they would tell their conversion stories when the scales fell from your eyes and you came to know Jesus as your savior. That's a very important moment in someone's life if they're an evangelical. They tell that story over and over again. There's a clear before and after when you get saved. And it's sort of the same with these AI people and feeling the AGI. It's like there's a moment where you realize as an AI researcher that you are not just programming computers, that your job is not normal, that you are part of this awakening, this phase change, this fulcrum of history. And I wanted to know what he meant. So I asked Ilya at one point, like, what does it mean, feel the AGI? And he's very funny. He's like a ponderous sort. He speaks in this thick Russian accent. He sort of talks like Yoda, gives these sort of talks very slowly, takes very long pauses, sort of looks out the window. He finally comes back and he says, you know when you see one of those movies where they start out on the beach looking at a person and then they zoom out and it's like, oh, you can see the whole beach, and then it zooms out more and you can see the city, and then the state, and then the country, and then the earth, and it just keeps zooming? He said that's what feeling the AGI is like. It creates this feeling of vastness. And he also compared it to listening to Bach's Toccata and Fugue in D minor. He said that song produces the same feeling as feeling the AGI. So I actually, as a sort of exercise, I tried listening to that song on repeat while I was writing that chapter because I wanted to feel what he felt.

And mostly it just made me incredibly sick of that song.

Yeah. So this is a scene that happens, let's call it the fall of 2022. So GPT-4 is just finishing training, I believe. And there's this sort of leadership offsite for the executives of OpenAI. And Ilya, who's at this point sort of running the show on the research side of OpenAI, has this idea. He wants his colleagues to feel the AGI. He's concerned that they're just going about their daily jobs. They're not taking the full arc of perspective on the thing that they're building. And he's trying to figure out, how do I break through to them? And so he gets this little wooden demon, this statue, and he takes it to this bonfire where everyone's gathered around. They think they're doing a campfire cookout or something like that.

Yeah. And he enlists a couple other executives, including Sam Altman, in this sort of mock ceremony. They wear their white bathrobes from the resort where they're all staying. They're playing the theme from Requiem for a Dream, and they sort of start circling and they're doing this kind of ritual Kabuki play. And he's talking about, oh, you know, we're building the AI and we think it's so safe and it's so powerful, but now it's misaligned and now it's turning on us. It is deceiving us. We must kill the misaligned AGI. And then he throws the wooden demon into the fire and they light it on fire and it goes up in flames. And I talked to a bunch of people who were there for that, and half of them were very moved. It was a very pivotal moment in their sort of awakening. And the other half are just like, that was the weirdest retreat of my life. Like, what a weird thing to happen in an office offsite. You know, what have I signed up for here?

So anyway, there are just scenes like that that you're like, man, I hope good movies get made about this someday.

Not yet.