I'm in DC now. It is definitely the second most AI-pilled city in America.
Demis and Dario and Sam, they are some of the most competitive people who have ever lived.
The thing about the Manhattan Project is once they build the bomb, the scientists don't matter.
The generals really take over. They're like, 'Thank you for your work. We'll take it from here.'
Question of like, why is Dario such a China hawk? Dario's first job in tech was at Baidu.
We have no explanation for why this works. Who closes an academic machine learning paper that way?
Do you think people actually hate AI as much as they say they do?
If unemployment goes up by two points, that turns into an extremely real hate. And so...
But they're not going to stop using it. It sounds like you're not sufficiently AGI-pilled.
Okay. What could possibly be more important than teaching sand to think?
Well, maybe I'm a little too sane. I think that's my feed. I think that's the feedback I really got from this episode.
Kevin Roose, formerly of Hard Fork, author of the frankly brilliant AGI Chronicles, with an assist from Jasmine Sun, who was a researcher on the book, joins — I guess we're calling this Model Talk because we're mostly talking about models, not about China — today to go through his, you know, recent history of OpenAI, Anthropic, DeepMind, and the community of people who think and worry too much about artificial intelligence. Kevin has left the New York Times mothership, is launching his own show called Machine Gods later in October. You know, you don't need me to tell you to subscribe to that. Kevin, congratulations. What a...
Thanks so much, Jordan. First time, long time. Have I been on China Talk before?
No, you've been on Hard Fork before.
I made it on Hard Fork in the...
But I have not gotten the honor of getting on to one of your esteemed shows, so this is a real pleasure.
So AGI Chronicles, I didn't really know what to expect. You know, I expected I would end up skimming it, and I didn't. I read every single page, and I read it over the course of 36 hours. It had life. It had humor. It had journalism and factoids. It had brilliant observations. It had brilliant writing as well. You just got these like great analogies, great quotes, and you weaved it all together in something that I, as someone who has lived this for the past five years, learned something from, but also as something that I am going to buy for my family for the holidays to kind of like get them caught up on my universe. So, I want to thank you and congratulate you because books are hard, and you know, you could have just been messing around like me podcasting all the time, but you like, you know, took it on yourself to do the thing. So, thank you so much, Kevin.
Thanks so much. That's very kind of you to say. And I should say like this was my idea. Like someone like you, and maybe actually literally you, was in my head as I was writing because I did want to write the kind of book that a total outsider could come in from a cold start and pick it up and know nothing about AI and come away with a working knowledge. But I also really thought it was important for like people who are on the inside in the bubble to learn new things and to get some perspective on it. So I think of you as sort of a consummate insider, and I was trying to make it so that people like you would learn something. So that means a lot to me. That is exactly what I was going for.
And to be clear, it wasn't something. It was like every single page I was like, 'Huh, I guess that's what happened.' Or, 'Huh, that's like an interesting characterization of this person that I've already spent, you know, dozens of hours of my life consuming them having talked already.' So, you know, these are not like the Axios for wrinkles is the end of it. It's really something where, you know, page by page and then the entire narrative I think brings you some new stuff. So let me like workshop some takes that I have had after reading your book. So, this idea of believers, and believers having what they believe come true, and then kind of like what happens to you, what happens to your worldview, what happens to the world? And because we're on the China Talk feed, the first person that came to mind was Vladimir Lenin. So look, this idea that like you have Marx and Marxism and these like abstract things that you know, like historical inevitability, you think are going to end up manifesting. But at the same time, look, Lenin is not like sitting there in, you know, the equivalent of church just like waiting for God to do it. He's, you know, running, he's organizing, he's writing articles, he's sneaking back into Moscow, he's like creating the revolution, and then all of a sudden the revolution happens, and then all of a sudden he wins, and he's like running the country. And it's just like the sort of the before and after of the sort of not just Lenin but any kind of like political revolutionary of believing in a thing that no one else believes in, and all of a sudden you're righter than you could have ever imagined like within the span of a year. I don't know. There's something that sort of spoke to me about that analogy when thinking about a lot of the characters in your...
Yeah, I buy that. There's a little flavor of kind of dog who caught the car to it. It's like these people are so used to — because, you know, it's important to remember when these people started talking and writing and going to conferences about AGI in like the, you know, early to mid-2010s, like this was a crazy thing to believe. It was kind of disqualifying to believe if you were like part of an academic research institution. It was like, you know, showing up to your department meeting and saying like, 'I'm really into astrology this week, guys.' It was like that. It had that effect on your credibility.
And so, as a result, like I think these people just were unusually persistent and immune to social pressure on this point. Like their conviction had to be so strong that it could overcome people that they respected telling them like, 'This is crazy. It's never going to work.' You know, there's this scene in the book where Bill Gates is coming to OpenAI, and it's the early days of the language modeling thing, and he basically scoffs at it. He says, you know, 'This isn't going to work. You guys need to stop building language models and start doing more neurosymbolic AI.' It's like I'm sure for the researchers there that was like an incredibly deflating thing to have like this, you know, icon of computer science tell you that you're on the wrong path, but they like they stuck with it, and now they're kind of in this dog that caught the car position of like, 'Well, what do we do now that people actually believe us?'
Yeah. You have this great arc of like Alec — Rashford is the soccer player — Alec Radford, who, you know, is 23 years old. Like all these guys are brilliant, but he's just like doing things and it keeps not working, and then he's depressed and he's like, 'Should I even do this?' He shows it to Elon. Elon's like, 'Your work is crap. Why are you wasting my time?' And that's rough, but they stick with it. And I think there's like a — there's an aspect of — okay, so we have communism, now we're doing California — to believe as deeply in this like, I think it's fair to say like ideology or science, which, you know, felt much more like an ideology 15 or 20 years ago than it does today. You have to be like just really odd...
...and really, and like having the courage of your convictions in a way that, I don't know, speaks to the Bay Area in some like mystical way. It's like there's something that lets you be this weird for this long, where nowhere else in the world can kind of allow set you up for, you know, continuing down that path.
Well, and totally. And I think you're right to put your finger on the role that belief plays in this story because something that I realized kind of midway into the reporting process is like this was not a book about technology. It's about this belief. And the thing that actually differentiates the kind of AI labs and the companies that are winning versus the ones that aren't is not a matter of who has the most money or GPUs or even like research talent. It's like, do you believe or not? Because only by believing and then only by selling that belief to others can you amass the resources and the focus to get to AGI. And there's this like interesting sub-thread about what happened to Google, which is, you know, the inventor of most of the modern AI stack. You know, they come up with the transformer, they come up with, you know, mesh TensorFlow, like all these things that just become very important pieces of the puzzle, and then they just lose it. Like they don't believe. And there's these famous like memo battles inside Google where like one person writes a memo called like 'We should say AGI more,' and then another person writes a memo being like 'We should never say AGI. Like that's a terrible word.' And so they just — they have all the pieces except for the belief, and it doesn't work for them.
I also think about China and people talk about like, you know, what does — you know, can China innovate? Of course they can innovate. Can China build companies? Of course they can build companies. But does China have the sort of scientific intellectual ecosystem to allow something as weird as thinking, you know, AGI was a thing in 2015 and then be able to sort of seed enough money to kind of do the proof of concept? No. American exceptionalism has had a pretty rough decade, but there is something that did seem like very deeply Californian and very deeply American about these like weirdos just being able to keep at it for long enough to finally end up, you know, striking reality.
Totally. I thought about that a lot and like the sort of overcoming resistance piece of it because there just weren't many success signals for a long time. Like if you were a person who believed in AGI in like 2010 or 2012, like there just weren't a lot of incremental things happening that you could point to and be like, 'Okay, now we are closer to achieving the goal.' Like it really was a kind of mini AI winter. And I think about this a lot in terms of Demis and DeepMind because unlike the other labs, they were not in California. They're in London where like it's even crazier to believe in this stuff, and you have even more like respectable academics telling you that you're full of shit, and you kind of have to keep going anyway. And so I think for them it was like even harder to overcome the voices of skepticism.
Yeah. And where does the belief come from? Right. I mean, I asked ChatGPT to tell me how many sci-fi books you cited over the course of your narrative. We have the Foundation series. We have 1984. We have Ender's Game. I mean, you know, Iain Banks wasn't something that you shouted out, but in The Melancholy of... I mean, Arthur C. Clarke, it's like — that was the scripture, right? That was the training data.
And people are citing it as like, 'This is the future we're working towards.' And even if, yeah, like the numbers aren't necessarily going up in 2009, 2010, 2011, like it's such a compelling narrative and vision. And it's such a compelling thing as a nerd to get to think that you are going to be Leo Szilard or, you know, Oppenheimer or, you know, the Player of Games, that like you're just working towards that, and like that's — and that's enough.
Totally. They were all pattern matching onto the science fiction that they had read as kids. But they were all also influenced by the sort of empiricism of someone like Ray Kurzweil and how, you know, his big predictions about, you know, superintelligence or the singularity arriving by 2029 or whatever were all just like rooted in straight lines on graphs. It was like, oh, if the biggest supercomputer in the world is this big in 1999, then it's going to be this big in 2001, and it's going to be this big in 2012, and by, you know, 2029 it'll be big enough to like, you know, carry out the singularity. And I think a lot of these guys have a similar faith in the power of straight lines on graphs. And that is the scaling laws, that is like all of this. It is downstream of like, what if this just keeps going forever.
Anyone can be a believer, but being a believer plus being a scientist means this ends up happening, right? Like you also kind of like need to be able to run the experiments and, you know, manifest it into existence. You're not just sitting there waiting for the rapture.
Totally. Yeah. You have to do something with that belief. And, you know, various people do different things, but they kind of all converge on this set of actions like, let's assemble these like, you know, these giant blobs of GPUs and let's train on this kind of model. But yeah, it's not enough to just have the belief, but belief seems to be a necessary ingredient.
So you wrote a — you spent a semester at Fuller University. Sorry, we're just — we're gonna hit this belief stuff for a while now. Where does religion play a role in all of this? Like, how do you compare these guys to the evangelicals you spent time with way back when?
I don't want to strain the comparison because I think they're very different groups, but I think there is something similar in the social cohesion and the dynamics of both scenes, right? Like the Fuller folks were very insular, you know, many of them only had other friends who were evangelicals. And as a result, like they developed these incredibly strong cohesive social bonds. Some of the closest, you know, bonds I've ever seen and felt toward the people sort of in their group. And I think among the AI crowd, they also are like very bonded to each other. Even the people who hate each other, which is like small, you know, but important groups of or sort of pairings of people who hate each other like Sam and Dario. They're all sort of bonded together in this first the sort of, you know, the sort of contrarian bet that any of this was possible, and then the sort of like secret knowledge that it was starting to work, and now they're like, you know, they're the most powerful people in the world. And so I think there is a kind of kinship that comes from that. And, you know, I hope that's what brings them to the table to like collaborate on safety and stuff, but who knows?
You know, this idea of like the schisms. We're coming back to Lenin, right? We got the Bolsheviks, we got the Mensheviks. But yeah, it is this idea of like a revolutionary vanguard of like, only we see what the future is and we're going to bring it to the world like because it's inevitable, but also because it's really fun to be like a revolutionary or a leader or a, you know, titan of industry. The idea that these people are the most powerful people on the planet. Like look, you kind of going down the list, right? We have Sam Altman who's like talking to his friends about wanting to run for president before all this. We have Dario who like I think in a different timeline probably would have been happy just being a really excellent scientist in a university, but like there's a piece of him and it comes out which is maybe 30% of his character which is this like, you know, 'I need to win for humanity,' and also he's like good at winning. So here we are running a two trillion dollar company. Demis, like he's got a bit of the fantasy of 'I want to be the hero of the sci-fi novel,' but he also just kind of wants to be the scientist too, which is like a bit of a step back from being like, you know, enjoying the power games that he would have had to play to like have him end up being the CEO of Google instead of Sundar, for instance. I don't know, thoughts on these folks and the relationship they have to the power that they currently possess.
I mean, I think I have no way of proving this scientifically, but I envision that if you took everyone on Earth, all eight billion of us, and lined us up by least competitive to most competitive, like I would be somewhere in the middle, and Demis and Dario and Sam would be like in the top 10. Like they are some of the most competitive people who have ever lived. And this is not just like in AI. It's this way with everything. You know, people close to Demis would tell me like, oh, you know, he would like fly off for the weekend and like no one would know where he was going, and then it would turn out that he was like entering a pro poker tournament and like won it, you know, like and he doesn't even really play poker.
Yeah. This idea of him like doing Mensa competitions — like he is so too cool to do Mensa competitions, but like the fact that he needs to have his intellect validated by like, you know, doing crosswords in a room in a conference room in Ottawa, like I just — yeah, there's something like that's like very deep in your soul. And like I don't know, you didn't get into the relationship, but like he was a child chess prodigy.
And so from a very early age, you know, he was playing against people who were much older than him and beating them. And that had to feel really good to be sort of a, you know, a world-recognized chess prodigy from a very young age, and also to create some kind of crazy psychological bar for yourself that you then spend the rest of your life trying to meet. He's a very competitive person. Dario in his own way also a super competitive person. And then Sam, I think, is just, you know, he is unique among these people in that he's not a researcher. He's not a technical person who actually contributes to the models, but what he is is a world-class fundraiser and salesman and kind of person who brings the research into the world in an attractive package.
So, in Malaby's book, he kind of goes into Demis' relationship with his father and how he like pushed him really hard, and it's this like immigrant-made-good story. There's nothing in the book about Dario's parents or Sam's parents. Did you go down that route at all?
Not really. I mean, I asked people who knew them as children what they were like. And the stories were all fairly similar. You know, Dario, it was like, oh, he was like the smartest guy any of us had ever met. Like super bookish and also like very principled and, you know, kind of stubborn in the way that he is today. Sam's upbringing, you know, he grew up in St. Louis, came out as gay in high school, like sort of was kind of an early — even by the standards of Silicon Valley where like 19-year-olds are constantly like succeeding in fantastic ways, he was like unusually gifted at a very early age at kind of this like drawing powerful people into his orbit, selling them on whatever his project was, and then, you know, just kind of expanding his circle. So, I think these people all had like different gifts, but they were all useful once they sort of settled on AI and AGI as the thing that they wanted to go after.
I want to do the Manhattan Project analogy for a bit. I mean, Demis, he would literally talk about DeepMind as 'we're doing the Manhattan Project.' Dario had this fantasy of him being Leo Szilard, the guy who, you know, was trying to get the physicists of the world not to publish the recipe for a nuclear reaction. And then once it got out, he helps get Einstein to send the letter to FDR and then builds the bomb. You know, like the thing about the Manhattan Project is once they build the bomb, the scientists don't matter anymore.
And this is like — this is like not...
Yeah. The generals really take over. They're like, 'Thank you for your work. We'll take it from here.'
'We'll see you later.' It's like there's no trillion dollar company at the end of it. You don't get to be John D. Rockefeller. You're just like a guy that someone writes a history book about, and maybe you get a job for, you know, 100 grand a year in Los Alamos in the 50s to build even bigger bombs or something. But this is not where we're at yet, right? It hasn't been taken away. Probably not going to be taken away yet. Yet. Question mark, question mark.
I mean, look, I think like it's just different in that, you know, the Manhattan Project started as a bunch of government employees and contractors doing something under the auspices of the government. So the capacity — and it was sort of like the bomb was different in that once you build it, you have it. You don't need to keep improving it. You don't need to do version updates and patch bugs and stuff. So, you really could kind of tap the scientists on the shoulder and say like, 'Thank you for your service. We'll take it from here.' In a way that I don't think you can with AI. Like, I'm just imagining like, you know, Anthropic sort of handing over a checkpoint of, you know, Claude Fable or something to the government saying like, 'Hey, could you guys finish post-training this?' Like who in the government can do that? I don't think there's a single person that I know of in the government who would have the technical capacity to do that.
Yeah. I mean, I guess I'm not like — 'Are we going to be nationalized?' isn't really the question. It's more just like it is, you know, the creators of the Manhattan Project, once they were done, they didn't have these like instruments of power with them, right? They just all they were left with was like a little bit of moral authority which like was rapidly kind of like drowned out into nothingness because we had a Cold War to fight and win. Right. There are parts of your book where, you know, TBD, like who knows how ingenuous this is or not, the lab creators are like, 'Yeah, we don't know what should happen with society. Like please regulate us, like set up UBI, like political world, like figure this out.' And then there's also the aspect of them, you know, publishing all these papers being like, 'Okay, we're creating this world, like here's how you all need to respond to this thing that we're unleashing,' and because we are all billionaires and run the most important companies for the US economy, like we have some leverage here.
I totally, and I think that is where the AI leaders have learned from the Manhattan Project leaders and actually tried to improve upon. I mean, like Leo Szilard, that story did not have a happy ending. Like he sort of, you know, lost his influence after the war and then sort of ended up writing like sad science fiction, and he was like just sort of — he was lamenting for the rest of his life this thing that he had helped to create. I think Dario especially, who's, you know, Szilard's scientific hero, sort of saw stories like that and thought like, 'Well, I'm not going to like do that. I'm going to win the race because then they'll be forced to listen to me. Like they cannot ignore me if we have the best model.' And I think that's still a lot of what drives them today.
The question I have is like if you're thinking about the — you know, if the analogy isn't quite the Manhattan Project, but it's like Standard Oil, right? Or like Carnegie Mellon. These people like were able to shape politics for decades because they had more money than God. They had all the politicians in their pocket, and they were kind of like, you know, willing and able to play the game. I don't think oil pulled at like 7% back in 1880. You kind of conclude your book with this of like the fact that kind of no one wants it at some weird level. Like this community thinks that they're creating, you know, they're like bringing us towards the rapture, but, you know, we have a society which is like not necessarily crying out for it, and having that kind of bounce off against the political system I feel like is going to be the tension we'll be watching for years to come. Right?
Yeah. I want to ask you about this actually because it's really interesting. Do you think people actually hate AI as much as they say they do? And like how do you square that with the like absolutely insane trajectory of adoption among both consumers and businesses? Like do they hate it in theory but like it in practice?
I think there's like an aesthetic hate, but what if unemployment goes up by two points, that turns into an extremely real hate, and so...
But they're not going to stop using it, right? Like even if unemployment was at 10%. Like coders are not going back to like writing their Python scripts by hand.
I mean, I think if unemployment goes to 10%, then the government has a really dramatic reaction and lots of industries like figure out ways to kick it out of their industry. So, the jobs...
You think — I just do not think this is possible at this point. I think that you basically have like industries in which AI has become such a load-bearing technology that like — you think all those lawyers are going back to doing contracts without ChatGPT and Claude?
I mean, maybe not back, but I think the only universe in which a version of a pause happens is that unemployment goes up. Look, like I don't think this political system is like — when's the last time we had unemployment, right? Like it's been a really long time, 7, 10 years. And, you know, this is the thing that politicians are most responsive to. I think there's one thing that you can say for, you know, money and politics and the people that are gaining from it. But, you know, the idea that you can sort of just like have sustained social disruption and no backlash to it.
No, I think there will be riots in the street if unemployment reaches 10%. Like I think that would be horrible for social unrest. And there would probably be violence attached to it. I just don't think it means that like anyone's going to give up their Claude and ChatGPT and Gemini. Because I think that has now become so fundamental to the way that many, many people go about their lives. Look, like Mom Donnie said no AI in schools, right? You can see like insofar as there are like organized corners of professions that like want to keep them around, that want to like keep their membership around, like there will be guilds for sure.
Yeah, I think and there are guilds in some ways. Like you have to go to law school for three years. You don't need to go to law school for three years, but you have to go to law school for three years. Like, you know, insofar as you — the same with the sort of medical industry I think, and it may end up emerging, right, if there's enough that happens dramatically and fast enough that like people feel like they have to kind of get together and push back. But I feel like that is the only break I see. And then maybe to stay on that in a second, but what about China as a relevant argument I think just like falls away completely.
Yeah. I know you're the one asking the questions here, but can I ask you a question about China because I feel like I'm in the room with the expert. So, I'm getting asked a lot on this book tour about China and whether I paid enough attention to China, whether they're really, you know, three months behind or six months behind or a year behind, whether it's all based on, you know, distillation and theft or whether they actually have their own cracked teams of engineers. Like I'm sure listeners to your shows will know this already. So, this is more for my own edification, but like what is the correct response to that question?
I mean, I think you had the right amount of China in your book. I think the story you're telling is an American one. And it's one that these people created. And, you know, the story of the next 10 years of AI may not be an American one, but, you know, it's like the ChatGPT moment hit China extremely hard because they were all just like, 'What the heck, why didn't we see this coming?' Like we had compute, we had engineers, we had data. I kind of talked about this earlier, like they were kind of stuck in the same mindset as Google and DeepMind were, like, 'Look, we're just doing our thing. We are going to do our little explorations and we're going to try to optimize the businesses we already have, not kind of invest time and talent into a new paradigm.' So like I think that's a totally fine response. And then like the...
What would it take for China to get like fully AGI-pilled, and can that even happen in an authoritarian country?
I mean, it's a really good question. I think there are — it depends how you define it, right? I mean, I think that you're seeing more and more Chinese executives start to talk in ways that would kind of resonate with the characters of your book. I mean, like the — I forget if it was the CEO or like the head technologist of Alibaba basically said like, 'We're on a path to AGI.' And we want to get — Liang Wenfeng, like the most visionary of all these guys. Like he's the one character that would kind of fit in your book because he's just — I mean, of course you can't interview him, but he's just like — he's a total weirdo. He's totally self-made. He was like off on, you know, down in Hangzhou with his like hedge fund team doing his own thing.
Had like a secret Less Wrong account, right? Like an alt on Less Wrong.
Absolutely. I mean, he basically followed the path that Demis wasn't able to because he needed money in the 2010s, right? Like he was just able to have his small team and do cool research. So it seems he's going to continue to do that. Like he's just taking on like quote-unquote patient capital. You know, he's not trying to build a big business off of this, but he just wants to like, you know, do his little version of the Manhattan Project. But why don't you talk about China from the perspective of your main characters? Like how do you feel like it plays in the back of all of their heads?
I think it looms largest for Dario. And this was something that I ran at a little bit because this question of like why is Dario such a China hawk was super interesting to me. I mean, he is much more of a China hawk than either Sam or Demis. I think, you know, people have told me like that the percentage of Chinese nationals working at Anthropic is like much, much lower than at some of the other companies.
And I tried to figure out why that is. And all I could really come up with was, you know, there were people I talked to who pointed out that Dario's first job in tech was at Baidu. He got recruited by Andrew Ng to join the Silicon Valley outpost. This is when he was like, you know, pretty fresh out of his PhD program. So he'd done like a career pivot, and he gets hired at Baidu. And I wasn't able to figure it out completely, but something changed in him through that experience where he started to believe and to talk about the necessity for democracies, or in particular the US, to win the race. Like something about that experience spooked him and made him want to get to this critical threshold of AGI before anyone in China did. And he's never talked about it. He's written a little bit about his feelings about what it would mean for an authoritarian country to have AGI before a democracy. But I think this is still a big unanswered question is like what happened with Dario and China? The others, I think, sort of use it as like a tactical shield or like...
A way to convince people in Washington to like take AI progress seriously, but for Dario, it really does seem personal.
