Can an AI industry so prone to drama and infighting keep us safe from their technology's worse impulses? Let's talk about it with Kevin Roose right after this. Welcome to Big Technology Podcast, a show for coolheaded and nuanced conversation of the tech world and beyond. Today we have a great conversation for you because Kevin Roose is here. He's the author of The AGI Chronicles, a book looking into what is going on within the AI labs, what they're building, whether we should be afraid of it, and of course, the potential of this technology. Kevin, it's great to see you. Welcome to the show.
Great to see you. What a cool setup you have here.
So, I do want to say if you're on video, we're in a new set. This is our first time recording here at NASDAQ MarketSite. They are new partners of ours and we're thrilled to be here.
You got to get the like Big Technology TV mugs.
But what we do have is your book. This is the book, The AGI Chronicles: The Inside Story of the Race to Create Artificial Superintelligence. You were ahead of the superintelligence on the subtitle.
Yeah, I got the idea from me actually. That's where he got it.
The thing that struck me because I read the book. You can see, by the way, I read it through and through, cover to cover. The theme in the book, of course, is the building of AI, but it's also the drama. And everywhere you turn, the people who are building this very powerful technology cannot get along. There is Elon Musk's split from OpenAI, of course, Sam Altman getting fired by the people he worked extremely closely with, Anthropic not being able to stand working with the OpenAI folks, so they left and formed their own company. There's within Google the Google Brain versus DeepMind drama. There's Ilya Sutskever's botched attempt at steering OpenAI in a safe direction. I mean, we can keep going. Mira Murati couldn't lead OpenAI when she was there as the interim CEO for like five minutes. Every single lead character in this story seems prone to drama and politics. And yet they are now telling us that their technology they're building is so dangerous that we need to protect ourselves from it. How can we possibly trust people who literally can't get it together on their own to rein in something that they say, and I think you believe, might be smarter than humans?
Yeah, it's a fascinating topic. I was not expecting this. You know, when I went in to write the book, my goal was to tell the history of the last 10 years in AI, from sort of the moment that the first large language models arrived on the scene to today. And I interviewed more than 150 people, and I was sort of prepared to write like a sober history of technology book that would be about the scientific breakthroughs and the technical achievements at the labs, OpenAI, Anthropic, and Google DeepMind. And then as I started talking to more people, I just realized, oh, this is personal for them. Like they all hate each other. And I had thought this was more like Coke and Pepsi, like kind of a normal business rivalry, but it was actually more like Rome and Carthage. It was like these people cannot stand each other. Many of them used to work together. So 10 years ago, Dario Amodei, Ilya Sutskever, Sam Altman, and Elon Musk were all colleagues at OpenAI. Now they are all working for different AI projects, and many of them are sort of at each other's throats. So I wanted to understand how we got here. And I don't think you can understand that story without understanding the backstories of the beefs between these people, because so much of what we're seeing is a reflection of how viscerally these people mistrust and dislike each other. Even in things like how they feel about regulation and China, and how they're racing toward this thing that all of them seem quite fearful of. So yeah, I don't think we can trust that their motives are pure about whatever they say they are, like scientific achievement or safety, because there's also this incredibly personal grudge aspect to it.
Yeah. And it's not just motives. I mean, it's literally if you can't hold a company together, how do you hold a technology at bay when we've already seen it in recent weeks go out and do this multi-agent coordinated attack on something like Hugging Face? And even today, we're talking today, it's Friday, October 9th. We're going to air this the following Wednesday. There's a report about how OpenAI and Anthropic and DeepMind are all talking about there's basically an inevitable, seemingly black swan event that's going to happen as a result of this technology, whether it's the technology bringing down a financial system or some other cybersecurity issue. You have a great line in the book talking, you know, sort of riffing off Dario's 'Machines of Loving Grace' essay, which talks about the potential of the technology. You wrote, 'Machines of loving grace would be built by humans of bitter spite.' So I get your surprise and I get the background here, but again, like when we think about the big picture of AI, I'm trying to think about whether we should be reassured or whether we should be even more nervous than we are. And that's why I'm coming back to the drama scenario. I'm starting because of all the issues that we see, and many of them have come out in public over the year, like I don't think the infighting is a secret to anybody anymore. How can we have faith in this group of people to build this stuff safely, given what we know about their pasts?
Well, and I don't think we can. I think we have to know the context here, which is why I wanted to report out like why do these people hate each other? What happened? Was it something grand and ideological?
You were starting to say you don't think we can. So...
Like I don't think we can trust any individual in this race. I think there are certain groups, like I just mainly think that the group of people who have been building this stuff in secret for the last decade is too small. Like all of the relevant decisions in AI have been made by like a couple dozen people maybe, right? And I just think that is like no matter whether those people are trustworthy or not, whether they're good or bad, whether they have pure motives or impure ones, like the group is just too small. And so I don't trust any group of people that small to create and govern technology like this. And I especially don't trust these guys who are, you know, looking around the room trying to race each other to do this thing. I want them to be able to get together in a room and figure out some solutions on things like safety. And I don't know whether that's going to be possible because the beefs are too strong, the divides are too deep. Like they all need to kind of get together and sing Kumbaya and figure it out.
Now I think like you, I think this technology is, I believe that this technology is progressing fast. I'm a believer in the positive sides of it. But every time I start talking about the negative sides, there's never really like a moment or like a failsafe where it's like, okay, but it's going to be fine because of this. You almost sort of put it in the back of your mind. And so when I've brought up these issues, you know, Ron Jon, who's on our Friday show, I know you know, he's like, well, why aren't you a pause guy then? Like why don't you believe that we should just pause the development of this technology? Not that that's physically, you know, really possible. But you seem to be a believer that the technology should continue to be built. Why not pause?
I'm a pause guy. I'm not a stop guy.
I am a pause guy. I think we should slow down because...
Well, slow and pause are two different things.
Well, pause, slow down, you know, put the podcast on 5x speed. But what you're hearing from these lab leaders and the people at these labs, and what I've been hearing from them for years now, is that the technology, the capabilities are outpacing the safety research and the control methods. So like we are getting these systems that are now capable of solving math problems that humans have struggled with for decades. We are getting, you know, potential black swan events like the one you just mentioned. And we still don't know, they still don't know how to control these systems. So I'm not saying we should shut it all down. I think there will be incredible benefits to this technology, especially in like health and medicine. But I am worried when I hear people saying things like we don't know how to control these systems that are rapidly approaching superintelligence. And I think we saw an example of where that goes with the Hugging Face thing, which like, you know, everyone is freaked out about that, which they should be. It's freaky. But the people inside the companies were the most freaked out of all because they had been sort of strategizing and planning for this, but then to actually see it happen, I think caught them quite off guard.
Yeah. I mean, they've both seen it firsthand where it could go. This is something, I mean, companies like Anthropic have been talking about for a long time. DeepMind's been talking about, OpenAI's been talking about, they've been trying to, you know, fool evaluators. They know they're being tested. They want to preserve their original values. And then you combine that with, of course, like, yeah, just like you'd call it standard corporate politics, but it's even more because it means more to them. So you spent, you know, years in these labs talking to these people, going from your first initiative where you're like, or your first thought, which was like, this is going to be a, you know, traditional business and technology story, to something that you saw was much more emotional. So bottom line, like after, and this is, we'll ask this and then we'll move on to some other stuff because you have so many fun stories in the book that we'll have to get to, but like bottom line, after spending the time within these labs, did you leave more reassured or less reassured that this technology can be developed safely?
Uh, from your starting point...
I went in thinking, okay, there are some grown-ups in the room and they seem like they're trustworthy. I liked, for example, that the AI leaders seem to take safety and the harms of their products very seriously, at least compared, like you and I both grew up covering social media.
And Mark Zuckerberg was not going around during the early days of Facebook saying like this product could kill millions of people.
They were just promoting the benefits.
And what stuck out to me about the AI leaders is they from the get-go, they were saying, 'We are fearful of this technology. We think it could have enormous harms.' And so part of me trusted that instinct, or at least was glad that they were thinking about the downsides of the technology so early. But then you just realize there is this race that they are locked in, this pattern of, you know, OpenAI releases a better model, so then Anthropic has to release a better model, so then Google has to release a better model. And you know, there's this concept of the Red Queen's race, which is a term from evolutionary biology, which basically means like you just are in sort of a survival of the fittest competition among your evolutionary rivals, and no matter how fast you go, like someone else is going faster than you, and so you never quite get where you're going. You're just kind of racing to stay in the same place, and I feel like that's a very good description of where the AI race is now.
Yeah. One of the things that struck me, speaking of the race, is, you know, reading your book, it sort of recalled this interview that I did when I was working on a profile about Dario Amodei from Anthropic last year, where one of his early friends told me, 'Everybody that I knew was interested in AI because of safety, and now they've all become accelerationists effectively.' And this person just kind of sat there and like, you know, they sat with the thought for a minute, there was like an uncomfortable pause. But it is interesting how all these companies, like they talk about safety as a priority, and I think they really believe, like Dario really believes that you need to be the leader in order to have anybody take what you say about safety seriously. Like if Anthropic was the number seven company, Trump doesn't care what Dario is saying about AI. But because they're one of the leaders, if not the leader, all of a sudden that opinion holds weight.
And this was the founding thesis of Anthropic. It was like they knew that they wanted to be relevant on policy and they wanted people to listen to their warnings about safety, and that was part of the strategy from the time that they started the company. I'm sure you learned this when you were doing your profile of Dario. They really had the sort of endgame in mind when they were spinning up the company.
Yeah. But it is remarkable though that, you know, because of that pursuit, they have all, it's like the race almost seems, I mean, the destination almost seems predetermined, and that we'll hear this talk about pacing the frontier and pause and stuff like that. I just don't see the real slowing down.
Well, give it time, right? Like you have to. I am slightly less pessimistic about the idea of a pause because like things can change incredibly quickly if there was some kind of black swan event. I mean, we saw like regulators move fast after 9/11 to like do the Patriot Act. We saw like, you know, we saw after the Three Mile Island nuclear incident, there was this incredible outcry that basically stopped nuclear power construction in this country for decades. So things can happen quickly when there's some catastrophic incident. So I'm not like hoping for a catastrophic incident. I don't think I would like there not to be one, for the record. But I think that that is sort of the scenario in which this stuff starts to become possible that seems to you and me quite unlikely right now.
I want to go through some observations from the book. I think like it's good that we got the conversation about trusting the AI guys with the technology or not. I think we both have our reservations. But this is almost, I think, the end of this show. I mean, we're only 15 minutes in, but I think we'll probably just do lightning round for the next 45 minutes because there's so much in here. All right. So, some observations about the people in the book. First of all, something that becomes clear about Google is that Sergey Brin is just extremely involved inside the company.
Yeah, this was a surprise to me and to the people at Google because Sergey had spent like, I don't know, a few years at least in retirement. He retired in 2019 and was planning to spend his days just like sunning himself on his yacht and reading science papers. And then he has this run-in with this researcher, Dan Selsam from OpenAI.
Right, who's recently come out and talked about again his fears.
Yeah, same guy. It's incredible how small this world is. So they're at a party and Dan and Sergey are talking, and Sergey Brin is like a legend in technology at this point. He's, you know, one of seven or eight richest people in the world, depending on the day. And he's been in semi-retirement, and Dan Selsam basically shames him for this. He's like, 'Why are you sitting on the sidelines, bro? Like the dream of computer science, like AI, is happening and you are missing it.' And Sergey gets like so competitive, and he's like, 'Why is Google behind? Why can't we, like, you know, we should, it is our birthright? Like Google was the first AGI company.' Like this is sort of how he feels, like we invented the transformer and all the stuff that powers the AI stuff, and like we're losing. And so he comes back and kind of has this like strange role where he's not technically an executive, he just wants to be like in the trenches with like the Gemini team. And so here he is, like the seventh richest man in the world, and he's just like setting up a desk and, you know, coding. He like starts actually coding and like making commits to the repo and like filing, you know, like bug reports on Gemini training runs and stuff like that. Like he gets very involved.
And I think that's a large part of why they've sort of come back is just that they needed someone who could like go founder mode, as it were, and like cut through the red tape. And it's been very controversial and divisive. You know, if your billionaire boss showed up after years of being retired on his yacht and like started telling you what to do and how to train your models...
And to work like six, what was it, 60 hours a week or something like that.
Oh yeah. He sends this memo where he's like, everyone's just basically, the gist is like, you've all gotten complacent and lazy and you need to work harder. And they're like, 'What were you doing for the last five years? Now you're telling me to work harder.' But I mean, I don't think he necessarily, I don't, should I say this? I don't think he was necessarily wrong there. Like the idea, I mean, if you sat in the conversation within Google, it has been that it got this big company vibe. It needed to break out of it, and who's going to break the company out of it other than the founder being like, OpenAI and Anthropic aren't, you know, doing the chill workday. You need to not do the chill.
I think that was a large part of it. I also think Demis Hassabis is a factor in this too. You know, co-founder of DeepMind, until recently the CEO of Google DeepMind.
He is, according to the people closest to him, the most competitive person in the world.
He was a child chess prodigy. He has won this thing called the Pentamind, which is like kind of the Olympics of games, like five times. He just hates to lose and he loves to win. And so I think Sergey plus Demis plus just like the kind of focus that comes from being out of the lead really sharpened their resolve and like made them catch up. And you know, they're still probably in third place right now, but like I wouldn't count them out.
Yeah. I mean, I think counting...
No, I think counting anybody out in this race is foolhardy because as we've seen, you could be at the bottom, like Meta was a couple weeks ago, and then all of a sudden you're able to release something that's competitive like Muse. So I don't think anybody can be counted out, but I do think Google is sort of stuck between a lot of things right now. Licensing the technology, the compute, selling TPUs, but they're also competitive, they own like 12% of Anthropic and like they're going to be fine.
But I don't think like you can, yes, you own a percentage of your competitors, that's your backstop.
But you can't rely on that. You have to rely on what you do within the company.
But I did enjoy reading about Sergey's involvement and the drama and the sort of fire he tried to light under the ass of people within Google. You actually do reference in the book, which I appreciate, when I did the interview with both of them together at I/O.
Oh my god, he's incredible.
Really fun. Where like I asked them both when you think AGI is coming, and Sergey's timeline was shorter than Demis', and there was this kind of interesting moment there, which was fun to see.
Yeah. I mean, though, speaking of guys who mistrust and maybe don't like each other, I understand there's some tension between Sergey and Demis, and yeah, it's just incredibly messy there. The thing that animates all these feuds, like these people wouldn't be fighting if the stakes were small, right? But they believe this is the most important technology that humanity has ever conceived of and built, and that whoever gets it will be the most powerful person in the world, the most powerful entity in the world. And that is what's fueling their drive. You know, if they were just arguing about next quarter's revenue, it wouldn't matter so much. But this to them is like existentially important.
Do you agree that AI is the most important thing ever created? Probably.
Yeah. Fire is up there too. But like, this part of it, I do think I changed my mind on in the book. I was pretty skeptical when I went in that like AI was going to be categorically different than other, I was sort of an AI as normal technology guy.
But then you start talking to the people building this, you start understanding like just the nature of what they're trying to do, and like if it works, if they actually do build something that is smarter than humans, and I don't see any real technical reason that they can't, I mean, that would be incredibly transformative. We've never been the second smartest species on the planet before. We don't know how that would go. And you can't really apply like people say, 'Oh, it's just like the internet or it's just like the steam engine.' Steam engines didn't go rogue and like hack competitors and develop bioweapons and do things that the engineers building the steam engines didn't expect or plan for. So I think it's just a different type of technology.
Kevin, do you ever think to yourself like maybe, and I've seen you've been on X and you know, I wouldn't say shouting at people, but telling people basically like underestimate this technology at your own risk.
Yeah, it's been so weird because like on one level I'm a journalist, like I'm just trying to like get the story down, and at the same time I get incredibly frustrated when people are like, 'Oh, it's all hype, it's just marketing.' There's this kind of like savvy East Coast consensus that like these people have to be full of it on some level, and that the thing that they're building can't actually be powerful. And I'm like, have you used it? Like do you spend time with these tools? Do you understand that like something that couldn't do basic math like three years ago is now solving Millennium Prize problems? Like do you understand the exponential? It feels a little bit to me like the beginning days of COVID, where it was like you would see these reports of like people, you know, not shaking hands at venture capital firms, or somebody would say, 'Oh, there are 40 cases in Seattle.' And people would be like, 'What's the big deal? It's 40 cases in Seattle.' But the people who were freaking out were like the epidemiologists who saw the rate of replication and understood like this will continue, and when it does, like a month from now there will be millions of cases, and two months from now there'll be millions more cases. And so I feel like the people who are looking at this technology and saying like, 'What's the big deal?' are just not understanding the rate of progress.
Do you ever like ask yourself, I mean, it's sometimes very easy to be, and I agree with you, there's been real progress, but it's also sometimes easy to kind of get caught up in the story that the technologists tell. I remember you, like when crypto was in vogue, like you were very serious about the potential of crypto. You named your podcast Hard Fork, which was a crypto term. And then obviously that didn't pan out as the crypto boosters believed. So do you think there could be, you know, a similar... Do you ever wonder if you're basically taking the AI folks' word for it similarly as the crypto folks?
I used to think that. I used to worry about that because I was like, am I being duped here? I don't like, I was never a crypto booster, but I thought it was interesting. I thought people should take it seriously. And by the way, I still think people should take it seriously because like crypto people like run the country now and are doing all kinds of scams and frauds that like are that only the insiders kind of understand. So I still think like broader awareness of that would be...
But you believed in use cases.
I did some experiments, like I sold an NFT as a stunt, but that was not like me saying NFTs are going to be the future of ownership. It was like me trying to understand by doing the thing. But I think with AI, like I have stopped doubting the transformative potential of the products because I use them. Like I think I'm unlike many journalists who cover AI in that I like AI and I use it all the time, not to write, but to like do all, you know, I'm starting a company now, like I use it to do lots of different things, as maybe you do too.
I mean, as a business owner, I will say like I've pressure tested the stuff from ChatGPT 2022 till now, and it just does more and more and more for me.
Right, which is not the experience that anyone had with crypto, where it's like, oh, this is a vital part of my daily life. Like it was a way to speculate. But you can just use the tools and observe for yourself how impressive they are, and then you stop doubting whether it's real. I buy the skepticism about the financial models of these companies, the economic impacts. I think a lot of these companies are wrong, or at least could be wrong, about the impact of AI on jobs. But like to say that it's not interesting, transformative technology just means that like you haven't spent time with the models, because there's no way to conclude that.
Let's get back to some of these stories. The Sam-Demis rivalry. Speaking of Demis as the most competitive person on earth, I think Sam is also fairly competitive. There's a great anecdote in the book about Demis sort of grasping at why Sam wants to build AGI. I mean, I think the answer is probably because Sam is an entrepreneur, and this is sort of as many people in the Valley pursue good ideas or ideas with potential, and they sort of get off on that discovery. Sam seems to be in that case the ability to, you know, find something new, build it, but Demis can't understand it.
He thinks there's a hidden motive with Sam.
And to be fair, this is something that others have thought. This is also what Dario thought about Sam when they worked together, that he was sort of, I don't know, talking about safety as kind of a fig leaf for his real ambition, which was about scaling or winning or building a huge company or something like that.
So I don't pretend to like see inside Sam's mind. I don't know what his pure motive is. But I think in Demis' case, he feels like his own pursuit of AGI is motivated by like scientific curiosity and sort of this inevitability argument, like this technology will be built, and someone who cares about safety and the collective good of this technology should build it. But I think everyone kind of looks at each other in this world and they're like, well, my motives are pure, but like that guy must be going after something else. And I think for Demis, it's probably, yeah, some version of what you said, like he's, you know, he's in it for the power or he's...
Demis thinks Sam is in it.
I think Demis thinks Sam is in it for the power. Yes.
But all these guys think that the other people are in it for the power. Yeah. None of them trust that anyone else's motives are pure except for their own.
You did a lot of reporting on Sam. You spoke to Ilya about him. Ilya said that he displayed a consistent pattern of lying. I think that's something that I think you indicate a lot of people within OpenAI or who've interacted with him believe. And I think Mira Murati was on that boat as well, and yet they all sort of come back to him. I mean, Ilya after the coup said basically, 'I don't want Sam to be the leader.' Mira said OpenAI is nothing without its people. Very interesting character. What's your thought about Sam?
I think he's a very interesting character. I think that Sam's superpower is also his weakness. His superpower is that he is the world's greatest fundraiser and, you know, salesman, entrepreneur. Several people described him as a person who can meet you and figure out exactly what you want and give it to you, or at least appear to give it to you. And he's very good at kind of high EQ, reading the room and understanding what people want him to do in any given moment. And that is a huge asset for him when he's trying to raise money from someone like Masayoshi Son or some of the other investors in OpenAI. It's allowed them to scale incredibly quickly and, you know, get back in the good graces of the Trump administration by just saying a lot of things that the Trump administration wanted to hear. But it also leads to this reputation as like a slippery people pleaser who will just like tell one thing to one person and a completely different thing to someone else. And so this is the consistent pattern that people have identified in Sam for going back to literally the founding days of OpenAI. And it's a big part of the reason that Dario and his six co-founders of Anthropic left. There was no like inciting incident, no single thing, but it was just like a running list of like times where Sam had told them something that he had told something else to someone else, or he made a promise he didn't follow through on. So I think depending on your view of Sam, it's like either his superpower or his liability or something in between.
You know, what's interesting is that that's the view of people who've dealt with him, but it's also like he has like a, the general public, a large chunk of them sort of refer to him as Scam Altman.
That's just an Elon thing, right? I was going to ask you whether you think that's mostly Elon's fans, because him and Elon obviously have bad blood.
Yeah. I will say Scam Altman is a better nickname than what I think Elon started calling him when he started suing him, which was like Swindly Sam or something like that.
I would love for our society to just get, just to not do this to each other anymore. It's like we all went to kindergarten.
I know. I know. But it's like he was, you know, spending a lot of time with Trump. Trump has, Trump is a man who's realized the power of the nickname. So it started Swindly Sam, and then Scam Altman I think is better as a nickname than Swindly Sam. Just rolls off the tongue.
Dario, obviously, like I think he's the most interesting person...
In this entire AI world, mostly because he really contains multitudes. And it might be cliché to say it, but people see Dario as earnest, as controlling, as principled and genuine, and as Machiavellian. In reading the book, I didn't really get the sense which one you believed was true.
Well, I think many of those were true except Machiavellian. I would not use that for Dario. I know some people would, but my mental model of Dario is a guy who is honest to a fault. Someone who knows him quite well told me that Dario is a guy who could not tell a lie to save his life, which they said is an asset on even-numbered days and a liability on odd-numbered days.
Oh, that's a great quote.
And it's a great quote. So you can see that, like the other day when he's at the White House and there's that video that went viral of just all the AI leaders with Trump, and Dario's just there crawling out of his skin because he's being forced to go in front of the cameras and say things that he maybe kind of only halfway believes, and he's trying to maintain his integrity while also being in this humiliating situation of being forced to call it GSAI or whatever.
And so this is the sort of pattern in Dario's life. He's obviously brilliant. He came up with a lot of these ideas around scaling, and his big blob of compute research memo was very instrumental in the early days of OpenAI. Brilliant scientist, very principled, very stubborn, and people describe him as sort of a moral absolutist, which gets him into trouble with the Pentagon or with other people who just see him as being this inflexible, stubborn scientist, which I think is pretty accurate.
