In the span of a decade, we've gone from AI not really working and not doing much of consequence to now we have these systems that are out there solving Millennium Prize problems and conducting automated cyber attacks.
So, how many people are deciding our fate at this point?
I think it's probably like 50.
I am here with Kevin Roose. Kevin, thanks for joining me.
Thank you so much for having me, Sam. Well, you have written what I think is probably the first proper history of the birth of AGI. The book is 'The AGI Chronicles: The Inside Story of the Race to Create Artificial Super Intelligence.' It's a very fun read, very behind the scenes. And maybe before we jump into the book and its various players, can you just summarize how you came to focus on this topic? Because you were not always on the tech beat. So how did you get into AGI, and when did you first convince yourself that this was worth paying attention to above seemingly everything else at this moment?
See, I've had several fields in my career as a journalist. I was first a religion reporter, where I first came across your work, Sam. I was spending a semester undercover at Liberty University with a bunch of evangelical college students who were, by the way, obsessed with you. They were watching your YouTube videos up late at night in the dorms, because you were challenging their faith. Anyway, that was my first career.
When was that? What year was that?
That would have been 2007. So, this was sort of at the apex of the new atheist YouTube scene. And it was very interesting. I remember sitting around watching you and Richard Dawkins and Christopher Hitchens on YouTube with these evangelical college students. I was not evangelical, but I was there to investigate and report this book. And that was very interesting. So then I had a short career as a finance reporter, and then for about the past decade I've been covering tech. And you know, when I started covering tech, no one was really talking about this idea of AGI, artificial general intelligence, and the idea that you could create a human-level AI system. This was something that not even the AI experts at the time thought was possible. But there was this little community, this little crew of technologists who were sort of considered fringe weirdos within their community, who believed in AGI, who believed that it was possible to build something like a digital brain and endow it with the powers of the human brain. And they're now some of the most famous, powerful people in the world. So the question I was trying to answer with this book was, how did we get here? In the span of a decade, we've gone from AI not really working and not doing much of consequence to now we have these systems that are out there solving Millennium Prize problems and conducting automated cyber attacks. And so I really thought this era deserved a proper history. I think this is something akin to the Manhattan Project, and we have lots of books about the Manhattan Project, but we didn't really have anything that was an attempt to document what's been going on in AI for the last decade.
Yeah, we should come back to the analogy to the Manhattan Project because it's fitting in some ways, but in other ways I think it understates how strange this moment is. How real is the risk to you? I mean, obviously there's a continuum of concern on this topic, from people who are quite cynical about it who think that any expression of risk, certainly existential risk, is nothing more than marketing or an attempt to produce regulatory capture. You've got these arch-accelerationists, some of whom are very close to the government and were even in it until five minutes ago. People like David Sacks, Marc Andreessen, who's been on the podcast, is somebody who seems to have very few concerns about risk of any sort. And then there are the people on the further end of the continuum, many of whom have also been on the podcast, people like Eliezer Yudkowsky. Nick Bostrom is probably not as shrill as Eliezer at this point, but he's the person who wrote the book 'Superintelligence' that got so many people worried about this. And many of the people we're going to talk about, who are actually doing the work, started as being quite radicalized with respect to the risk, and now they are racing in this tangle of perverse incentives to hear them justify it, but nonetheless racing toward building AGI as quickly as possible, and in some cases they seem to have forgotten many of their safety concerns. So before we jump into the dramatis personae here, what's your sense of the reality of risk, and how much does that motivate you in covering this topic?
I think the risks are real. I think that the people who have been saying for years that there's nothing to worry about from AI, that it's all going to go smoothly, 'What are you talking about? They're just stochastic parrots, they're just language models, they're just fancy autocomplete'—those people have been saying the same thing for the better part of a decade, and they have been wrong every time with their predictions that deep learning was about to hit a wall, that capabilities were going to plateau. What we see instead is this very smooth progression. You know, if you go back to the early 2010s when guys like Dario Amodei, who was then at OpenAI, were starting to come up with what became known as the scaling laws. These are these sort of mathematical equations that dictate the performance of a model or predict the performance of an AI model at a given scale. They saw a very smooth upward curve. They described it to me as feeling almost like it was a simulation, like it was just too smooth. The resulting intelligence when you scale up the size of a model, the amount of data it's fed, the amount of compute it's run on, you just get this kind of smooth curve. And they were predicting that the curve would just continue, that there was no obvious asymptote, no place at which it would stop being true that scaling worked. And so far, across many orders of magnitude, they've been correct. That's why we have models today that are so much more capable than they were just a few years ago. It's not because we came up with some amazing new algorithm; it's just because we've been scaling these things now for many orders of magnitude and they keep improving.
Yeah. So more compute, more data, more parameters. There's a linear relationship between those three variables and the intelligence that comes out on the other side. And again, we're talking about LLMs. Of course, there were many people who have been skeptical that the LLM would be the final platform and that we would hit some wall here. People like, you know, who you discuss in the book, Gary Marcus and Yann LeCun. You know, safety aside, what's the state of the skepticism at this point about LLMs? I mean, where—if we had Gary Marcus and Yann here, would they modify their skepticism or are they singing precisely the same tune they were four years ago?
It's mostly the same tune they were singing four, or six, or ten years ago. But I think there are fewer of them now. I think a lot of the skeptics have kind of capitulated and they've said, 'Okay, well, they used to be stochastic parrots. They used to just be predicting the next word in a sentence, but now they're doing something more than that.' And to be fair, the LLMs that we think of today—Claude, ChatGPT, Gemini—they are not pure LLMs anymore. They also have this reinforcement learning architecture kind of bolted onto them. And so they are a different thing than they were ten years ago. But the basic idea still holds. This idea that if you train a model to be very, very good at generating text and predicting what comes next in a sequence of text, you can scale that up until they've basically learned not just the words but the underlying principles. That was the bet that was not at all intuitive to people back in the early days of what we now think of as modern AI. But those were the days when I wanted to go back to those days in the book because I thought, how else to tell the story of this technology than to look back at the origin and the birth of it and the moments where the researchers are sitting around their computers in 2017 kind of watching the first outputs from these kind of weird language models, these GPTs, and they're starting to generate coherent strings of text. And to sort of be there, metaphorically in the room, when these breakthroughs were made. That was one of the reasons I wanted to write the book, because I just thought those are important moments. That is the first, you know, the lab notes from Watson and Crick or from the Manhattan Project, where you get to see this sort of scientific breakthrough in real time.
Yeah. Yeah. What was the most important inflection point from your point of view? Was there one moment or were there several where we passed a point of no return and people recognized that the path to AGI might in fact be wide open just if we keep going in this direction?
