Dario Amodei, Sam Altman, Demis Hassabis, Elon Musk, even... they're not besties.
No, they hate each other. They all... Yeah, right.
I mean, that's like one of the primary things that I learned while reporting this book is like it is personal with these guys. I think it was remarkable that they all came out in support of this proposal by Dario to pace the frontier. I think it's a signal that this is really serious. I believe that their fear is sincere.
There are people who believe that these are just CEOs trying to pump up their valuations so that they can go public. These leaders have been talking about existential threats from AI for longer than they have had companies. The idea that all these people just came up with the doom stuff recently to like scare people into, I don't know, investing in their IPOs or whatever just does not hold water in my view.
Kevin Roose is a journalist racing to cover the AI race. After almost a decade as a tech columnist for the New York Times, Kevin recently announced that he's leaving to start his own media company. Kevin also has a new book out, The AGI Chronicles. The book documents the sprint to create artificial super intelligence and the handful of minds at OpenAI, Google DeepMind, and Anthropic behind it all with all their messy humanity on display. Today I'm sitting down with Kevin to dive into his book, to talk about our concerns around AI safety, and we kind of debate whether or not AGI is in fact here.
Hi Kevin, good to see you again. Welcome to Pioneers of AI again.
Great to be back. So, the last time you and I spoke was back in 2024, which seems like a lifetime ago in AI years. That's like a few decades.
We're going to dig into AI in a second. But after, what, nine years working at the New York Times as a tech columnist and then four years co-hosting Hard Fork with Casey Newton, you guys have decided to leave and start a new media company.
Yes. Exciting. We have. It's an occupational hazard of living in San Francisco, which is that if you spend enough time there, you start thinking about building a company. And so this is a dangerous affliction. We tried our best to protect ourselves, but unfortunately, we succumbed.
So we started this company. We're starting a new podcast called Machine Gods. We're partnering with NPR. So we're very excited about it.
So, in addition to starting the new podcast and the new media company, you wrote a book, AGI Chronicles, and I will note that we are recording this two weeks before the book launches. And I'm kind of nervous because I feel like anything could happen in the next two weeks and we'll be like, darn it, we wish we had Kevin to opine on it. So, how do you deal with all the FOMO around the news and how do you ensure that the book is not outdated?
So the first step is just admitting that you've already failed. If you're trying to produce a book about AI that is not outdated by the time it comes out, I know that that is a losing proposition. I have done that. I had a book about AI that came out in 2021, which was one year before ChatGPT changed sort of everything about that conversation. So for this one, I really wasn't trying to be current. I wanted it to feel like a history, like a time capsule of this decade in AI research, starting in the summer of 2017 with the transformer paper from Google that sort of set us off on this path toward LLMs, and ending roughly this spring. So, I just decided I'm going to lean into that and really try to make this a durable history so that you're looking at it as a way to find out how we got here, not necessarily where we are right this minute.
You know, I was envisioning, you know, in a hundred years if AI doesn't kill all of us, I think students will be studying this book like this could be the history textbook of what happened in this decade of building AI. That would be super cool.
I mean, that's sort of what I was thinking as I was writing. I was really trying to write for future humans and honestly for future AIs that will want to understand how they got here, who built them, what were the companies thinking, what were the key decisions. Every future AI system is going to be trained on probably this book and others like it. And I want those AIs to have good information about themselves and their origin story as well. So you could think of it as sort of having two audiences, human and AI.
You just want to be on the right side of AI.
Look, I've got a lot to make up for. We got off to a rocky start.
I feel like it's important to take a beat and define what we mean by AGI. So what's your definition of AGI?
I don't have a canonical definition of AGI. Basically, the way it's been used is sort of human-level intelligence. Now, people have different definitions of that. Some people say to qualify as AGI, it only has to be as good as the average human at most things. Some people say it has to be better than all humans at everything. Some people say it only has to do computer-based work. Some people say it also has to do manual labor as well as humans. So, there are lots of different definitions and I'd be curious to know which one you favor.
I feel like my definition is kind of like the Supreme Court's definition of pornography in that I'll know it when I see it. So, I felt like for me, AGI arrived the moment that I stopped treating AI models like they were dumber than me. It wasn't this conscious decision or some release that came out and I was like, this is AGI. It was just sort of this gradual realization that I was relating to these models as intellectual equals, and that happened for me last year sometime.
So interesting. I don't know if I subscribe to that definition of AGI. Yeah. What's yours?
For me, it will be no surprise, right? Given my background building artificial emotional intelligence and thinking about emotional and social intelligence in machines, for an intelligence to truly be a general intelligence, it has to integrate all forms of human intelligence. So, it needs to be cognitively intelligent, which I think we're seeing with all these models, but I also want it and expect it to be emotionally and socially intelligent and perhaps even have embodied intelligence. And so, with that definition, I feel like we're a ways away. I don't know. Does anybody else you've spoken to share that definition?
For sure. There are people who think it has to be able to control a humanoid or it has to be better at therapy than human therapists or something. I think everyone's going to have their own definition. But I'll also just say not all humans are intelligent in the ways that you imagine true AGI. So not all humans have social and emotional intelligence. Not all humans are able to make full use of motion. So people talk a lot about jagged intelligence in AI, how the models are vastly superhuman at some things like math and coding and vastly subhuman at other things. And they talk about that as sort of a weakness of the models, but I actually think it's part of what makes them compelling. They are jagged in the same ways that humans are jagged. I think it's a continuum rather than a single point in time.
That is sort of, I think, the part of the AGI discourse that has been, in my view, wrong. I think a lot of people who started talking about AGI thought it would be this one point that we arrived at when someone would get up on stage, Sam Altman would get up on stage and say we have built AGI and everyone would go, ah, AGI has arrived. And that has never been the possible outcome here. Like we were always going to argue about it after someone claimed that they had built AGI. I think everyone has their own subjective bar for AGI and all kinds of incentives to raise or lower that bar over time.
It seems like we are in this race to get to AGI, but do you even think that's the right race and are we stopping to ask that question?
I don't think it is necessarily the right or wrong race, but it certainly is the race that we are in. This book is not an attempt to tell people how to feel about AI or whether the people who are building it are good people or bad people. It is really a descriptive narrative history. I am just trying to say here is what is going on. I've talked to more than 150 people at the labs. I've spent a lot of time with Dario and Demis and Sam and Greg and all the headline figures in this world. And I'm just trying to get a sense of why they are racing and what they are racing toward. And for them, this is really the race. People think, oh, it's just about money or it's just about power or it's just about control. Like no, they want to build AGI, all of them, very badly. Now their reasons for wanting to build AGI differ based on the person, but this is the race that they are running genuinely and earnestly. They are not just trying to build huge software companies. They want to be the first to build the machine super intelligence.
How would you articulate the different archetypes for each of these main characters?
So in the book I talk about there being three kinds of people pushing toward AGI. There are the scientists, the seekers and the survivalists. So the scientists would be people like Demis, people who really are into what can advanced AI help us solve in the physical and scientific world. So how can it unlock cures for disease? How can it improve material science? How can it speed up the overall rate of scientific progress? And they approach it as sort of a scientific problem. The seekers are the sort of more futurist types, the people who think it is sort of our destiny as intelligent beings to create the successor species that is going to replace us at the top of the intelligence pyramid, the sort of more Kurzweilian of AI futurism. Those are the seekers. And then there are the survivalists who are basically like Dario Amodei and other folks on the safety side of things who are like, well, it's not that complicated to build powerful AI systems. Someone is going to build AGI eventually and I don't trust those guys to do it. So we have to build it ourselves first.
Exactly. And you can quibble with that motivation. Plenty of people have, but that is honestly why they are doing it. It's not just because they think the science is cool. It's because they worry that someone with worse safety practices will get there first.
What would you say are some of the biggest promises of AGI and the biggest perils of AGI?
Yeah. I mean, the promise is look at all the hard problems we face as a society. I'm not just talking about disease and climate change, but war, poverty, all of the things that can make human life miserable. What if you just had several billion hyper-intelligent AIs working alongside humans to solve those problems? It's a cliché now to talk about AI curing cancer, but it's not a cliché. It's really something that many researchers think is possible either soon or in the distant future and that would be amazing. We know that technology generally improves people's lives in the long term. The downside is it could blow up the world. I mean, that's a pretty severe downside. The funny thing about writing this book was like I was talking to people, many of them who had been thinking about AI risk and existential risk for like a decade or 15 years, back when that was just crazy people who were talking about it, because the AI systems of the day sucked and there was no way they were going to turn into rogue AI swarms that would go out and conduct cyber attacks.
But now the technology is here. We have that, that's happening right now with Hugging Face and incidents like that. So I take really seriously the arguments for existential risk from AI, even if it's not literally all humans get killed. I think that just having more and more autonomous long-acting AI systems out in the world pursuing their own goals would just lead to a lot of really unpleasant outcomes if we're not careful. I'm also worried about things like job loss, but to me it's like if all we were worried about was job loss, that would actually be pretty good, right?
Right. Let's kind of reconcile what's happening in the AGI race and then the headlines, right? And you alluded to some of them. The frontier lab CEOs are coming together and saying we need to slow this down.
Yeah. I was very surprised by the fact that all of the major AI leaders, Dario Amodei, Sam Altman, Demis Hassabis, Elon Musk even, and these are all guys who are not like, they're not besties.
No, they hate each other. They all... Right.
I mean, that's one of the primary things that I learned while reporting this book is like it is personal with these guys. This is not Coke versus Pepsi. They have deep personal grudges with each other that go back in some cases more than a decade. So yes, I think it was remarkable that they all came out in support of this proposal by Dario to pace the frontier, to basically slow down the cutting-edge AI development. And I think it's a signal that this is really serious. These guys are not people who just agree with each other because it's fun to do. They're not fair-weather friends. They are bitter enemies and rivals. And that they would come together and say this is evidence in my view that things have gotten quite scary inside the labs. Because keep in mind what we see with things like the Hugging Face incident is old, right? That stuff happened a lot of it this spring. The systems have already advanced several generations beyond that. And it is the tip of the iceberg. We do not know yet what all internal deployments are doing at OpenAI and Anthropic that maybe haven't made it out yet or maybe the researchers at those companies haven't even found them doing things yet. I think that's what concerns me is like we don't know what we don't know.
You've interviewed more than 150 people for this book, including the CEOs, right? What are they saying behind closed doors that we're not seeing in the headlines?
I mean, they talk to each other constantly. And they're not doing that in public. I think they generally try to keep a pretty convivial attitude in public, but what I'll say is that I believe that their fear is sincere. There are people who believe that these are just CEOs trying to pump up their valuations so that they can go public and you know...
