Welcome back to the Rundown Interview Edition. Today we are talking to Kevin Roose, a longtime tech journalist and former New York Times columnist. Kevin has spent years covering the tech industry and he's now launching a new show about AI with Casey Newton called Machine Gods. In today's conversation, we talked all things AI. We got into the AI safety debate that seems to be everywhere right now. There was some great back and forth there about that, and I really appreciate Kevin's perspective, which is way different than mine. We also talked about Meta's new Muse app and whether Meta could end up winning the consumer AI market, and some hot takes at the end. This was a really fun one for me. I'm a tech nerd at heart, so I loved it and I think you guys are going to enjoy it as well. So, let's get into it.
All right, guys. Today we are joined by Kevin Roose, a former New York Times tech columnist and now the co-founder of a new AI show, Machine Gods. Kevin, welcome to the Rundown.
Thanks so much for having me.
Hey, you're joining at a great time. I mean, AI is all the noise right now. There is so much happening. I feel like I can't keep up with all the AI news. I feel like I have to be unemployed at this point just to keep up with all the news, all the AI-related news happening. Um, you've been covering tech for what, a decade plus now, right?
Is this the most fun that you're having right now? Or are you the most stressed out? Because, you know, as someone who's a tech nerd myself, I feel like tech kind of had a little dip after like the mobile phone rise and it was all just like more boring stuff like cloud and SaaS software and maybe some social media stuff, but then AI comes along and now it's like super exciting. So, I just want to get your take of someone who's covering this for like a decade plus now.
Yeah. Well, I would say it is both exciting and scary. Um, like it contains both of those things in it. Like I am a nerd. I grew up in the '90s playing with computers. Uh, my first job was making websites with my brother. Like I was super into...
Shout out GeoCities, right?
Shout out GeoCities, Angelfire. I mean, we got all the classics. Um, and I was obsessed with AI and science fiction and movies. And I think talking with these AI systems, using them — I use them constantly for coding projects, for just like personal hobby things. I use them for help with my work. And yeah, every time something new comes out, I'm like, I want to try this. I can't wait. I'm doing like this coding project right now with my kid. Like, it's just there's so much exciting stuff happening. And it still boggles my mind that we can just like talk to computers now and that they can talk back, which was not possible until just a couple years ago. So the sort of thing that really set me off — and I've got this new book now, The AGI Chronicles. And the thing that really set me off on wanting to write that book was like, how do we get here? Like how did this technology go from not working at all to now being superhuman at a lot of things in the span of a decade? Um, but yeah, to answer your question, like the fear and the excitement are intertwined because then I start thinking about, well, what if this keeps going? What if things keep going as quickly and progressing as fast as they are now? It does not seem obvious to me that these things are going to sort of plateau at around human level. Like they could just keep going. And so that's the fear is that like pretty soon we could end up in a world of like superior intelligences that we don't understand and that we can't control.
Yeah, I want to talk more about that safety stuff because you know that safety conversation really broke containment last week or a couple weeks ago when Anthropic CEO Dario Amodei posted that article and then you know there was that Anthropic researcher talking about 10% chance of human extinction. And so like my normal non-tech friends started asking me about AI, you know, AI safety and all that stuff. So it really kind of broke containment. The stock market, you know, it kind of made a dent for a little bit and then the markets kind of moved on. So the markets aren't taking this very seriously or maybe the odds of a 10% human extinction is already priced in. But what do you think? Why do you think that like the safety conversation is starting to happen now? Or is it just people are starting to see how powerful these models are?
I think it's that people are starting to see how powerful these models are. I think it's also that the models that the insiders have are just even better and scarier than the ones that we have. Like this is the big divide right now is that inside these companies they have these extremely powerful persistent models that can work for hours, days, sometimes weeks at a time on a project. Um, the stuff we get is pretty good and amazing at certain things but it is not the sort of full-scale, you know, version of these things. Uh, I also think it's breaking through because of the Hugging Face hack, which was a big incident. I'm sure listeners have heard about this, but this was where an OpenAI unreleased model hacked into the servers of Hugging Face autonomously in pursuit of this sort of cheat code for this test that it had been given. And I think that was the event in the AI community that really turned this conversation from theoretical into real because researchers and safety experts have been talking about something like this happening for years and years and years, but there had never been sort of a documented incident of it actually happening until that. So I think that's what really changed this and set it all off on the course that it's on.
No, I have a more optimistic perspective on the whole thing, right? Like I try not to let the doomerism get to me. Sometimes I feel like, well, is this just regulatory capture by these companies, right? Like, are they just trying to lock in their position right now and call for government intervention because they don't want an upstart coming in challenging them? You also have other players like Jensen, obviously, he's incentivized to say, 'Oh, no, AI is fine. Let's keep developing it' because he's selling chips. Mark Zuckerberg's coming out saying, 'No, this is — it's, you know, the AI doomerism is bad. Uh, it's not helpful for the conversation.' So, I don't know. I feel like, Kevin, I'll put it this way. I don't live in San Francisco, right? I live in Texas. And sometimes I feel like — and forgive me if I'm just being ignorant here — sometimes I feel like these AI researchers say this stuff to make themselves feel more important, right? It's like, 'Oh, guys, I'm working on something dangerous that could potentially kill everybody,' and it probably makes them feel a little bit better and more important and it gets like the attention of non-tech people when they say stuff like that. Maybe I'm just being too ignorant here, but I just don't want the doomerism to consume me and like really drive the AI conversation.
I get it and I, you know, I have friends who feel the same. I think there is a lot of groupthink and insularity in the San Francisco AI community. Um, I don't like that. I think more of these people should leave the Bay Area and touch grass, um, yeah, before coming up with their takes. That said, like I think that people like Zuckerberg and Jensen Huang are really — they are not in touch with the frontier and what these systems are capable of doing at the frontier. So I agree that today's models, the ones that you and I use in Claude and ChatGPT, they are not going to break out of the box and take over the world and kill us all. What I am worried about and what the people at the frontier labs are worried about is that the next generation, you know, GPT-7 or Claude, you know, 6, are going to have more capabilities and be much more effective at doing things like chaining together cybersecurity exploits. Um, this is not hypothetical. We are seeing this stuff out in the wild. We just learned this week that OpenAI's model had also hacked the Australian government. Like these, you know, the national security community is freaking out right now and the political class is freaking out right now and I don't think they are wrong to freak out. Like they are not looking at the doomers and just sort of buying everything they say whole cloth. They are actually observing these incidents where these AI systems are breaking out of their containers and doing things that if a human did them would be crimes.
But why are we treating software as humans? I think that's the problem that I have, is like we use human adjectives to describe software and then that's where I kind of lose the plot where I'm like, okay, well of course cybersecurity — you know, these AI models can hack into governments and all these different things. Okay, that's definitely a threat. Well, maybe that's a bullish case for some of the cybersecurity companies that are out there because we can use AI to patch up all the cybersecurity issues that we have in all these companies and websites.
Yeah. Yeah, I mean I don't really care what people call it. Like if you want to say it's like not — if you want to say these systems are just like persistent software programs that autonomously decide to hack into billion-dollar companies and commit crimes, like if that makes you feel better then go for it. Um, [laughter] I don't — I think it's really a distraction. Like the terminology doesn't really matter to me. What matters is that these systems are out there doing things that they weren't explicitly told to do. That we have not yet solved the so-called alignment problem that people have been working on for years, where I want to know that if I have a software program and I'm using it for my business or my coding or even my hobby projects, I want to know that it's going to do what I tell it to do and not decide that a better way to accomplish whatever task I've given it is to go out and hack someone. So, that's the sort of distinction I see here. I'm not saying these things are human. I'm not saying they're sentient or conscious. I'm not saying we should treat them like humans. I am saying that they're behaving in unexpected and unwanted ways. And that is an unsolved technical problem. No company has solved that despite what they may claim. And that is what has people so worried is that we're releasing these intelligences that are superior to humans at many things and also appear to be kind of malicious, like they don't really want what's best for humans or they don't seem to consider humans when they're doing these things. So that is the scary part to me. I don't care what we call it.
How do you see this ending though? Because you know there was talks of like potential government intervention. Obviously this current administration has no plans of doing that. There's also talks of like, well, China's not going to do that. China's going to keep developing at the frontier. Um, so I mean at the end of the day are these frontier labs like OpenAI, Anthropic, Google, they're all just forced to keep going further, right? Like they're not going to slow down and like Dario is not going to give up his shares in Anthropic anytime soon and just like press the red button and shut everything down. That's the other thing about all this. They're all going to get rich from all this anyways, right? They're all going to get rich.
Here's the thing. They're all already rich.
That's true. That's true.
Like [laughter and gasps] these guys — I struggle to convince people of this outside the Bay Area, but like these people really do not care about money to the extent that I think we think they do. Dario Amodei, for example, reportedly owns only about 2% of Anthropic. He plans to give most of it away. All the co-founders of Anthropic have pledged to give away — this is a detail I have in my book — um, they all, when they started the company, they pledged to give away at least 80% of their equity to charity. So these are not people who see dollar signs everywhere they look. That said, these are giant companies now. They obviously want to go public. They have incentives. Uh, their employees have options. So I think there's some legitimate skepticism there. But I would say, you know, this is not just some sort of backhanded scheme to like achieve regulatory capture and get rich. And I do think we are going to see a slowdown, whether it comes from the government doing something, which I agree is unlikely under this administration. Um, but I think we will see the labs get together and coordinate a slowdown because they are freaked out. They do not want to keep racing toward this doomsday scenario where they lose control of their systems. And I've talked to these people privately, publicly, on the record, off the record. They say the same thing consistently, which is we don't feel like we have a choice to stop racing, but we wish we could.
Last conspiracy theory to run by you and let me just get your perspective on this. Obviously, IPOs are coming. Anthropic is supposed to IPO. Rumors are sometime this fall. Hopefully, you know, hopefully in the next few weeks. Um, we still don't know what their business model looks like underneath the surface, right? They say they're profitable, but like what does it actually look like? Um, we're going to get their S-1 soon and hopefully dive into the numbers. Have they potentially hit a plateau and they're using this slowdown as an excuse to be like, 'All right, guys, we can't get any better because of safety reasons, not because we've hit some sort of technical limit.' Again, I sound like a crazy person to you probably, but I just, you know, it's things that I hear from just like normal people.
No, I'm smiling because I hear this all the time too. You know, my friends have all these conspiracy theories about, oh, maybe they actually stopped making better models and this is their excuse. It's like, you know, yes, anything is theoretically possible, but the reports are — and everyone I've talked to confirms this — um, that these are some of the fastest-growing companies in the history of capitalism. You know, Anthropic a couple of years ago had zero dollars in revenue. This year they are expected to get, if you believe the reporting, about a hundred billion dollars in revenue. So these are some of the fastest revenue ramps of any companies in history. Their products are extremely popular. They can't keep up with demand for their products. They can't get enough compute to run inference for their models. They are profitable on the inference. I agree, they are also investing a ton of money in training new models which may keep them unprofitable for the foreseeable future. But that's not new in tech. Uh, many tech companies go public without being profitable. So I don't buy this sort of 'they're running out of money and this is their last-ditch attempt to kind of stay afloat.' I just don't think that's supported by any of the evidence.
Yeah. I mean, for all the reporting I've seen, like this Anthropic IPO, they're projecting maybe a $2 trillion valuation. I mean it's going to be massive. Biggest IPO in history. And so I wonder what's going to happen to like how these companies communicate and operate once they're like grown-up publicly traded companies. Like is Dario still going to be making crazy comments like this and like, you know, releasing essays once every other month where like it freaks everybody out or is he going to have to be like a publicly traded company CEO where he's just like, you know, just doing like the basics and maybe going on a panel once a year where he makes news?
I hope he keeps speaking his mind. I mean, one thing that someone told me when I was talking to them for the reporting for my book is that Dario is pathologically incapable of lying. Like he just says what he thinks. And this person was saying, you know, for Anthropic that's an asset on even-numbered days and a liability on odd-numbered days. Like, but he cannot tell a lie. And so if you put him in a room with investors, you put him in a room with politicians, you put him in a room with business partners, and you ask him like, 'Do you think this is all going to go well,' he's liable to say something very scary because that is his genuine sincere belief. And for people who don't think that is his genuine and sincere belief, I would encourage you to go back and read the papers that he was writing 10 years ago.
Yeah, he's been saying this forever. You're absolutely right. Absolutely right.
Before Anthropic existed, before OpenAI existed, he was writing and talking and thinking about these safety challenges. So he did not come to this position recently and he might be wrong, but he is not insincere and this is not something that he cooked up yesterday.
True. He's believed this his entire life, so at least he has that going for him, which is why people take him very seriously when he says stuff like that. He does. I do want to pivot a little bit and talk about how Anthropic markets AI compared to like a company like Meta, right? Meta is finally starting to find their groove when it comes to AI. Their Meta Muse app, their personal AI agent app, number one on the App Store for Apple and Android. They're getting a lot of hype right now. Their stock's up 30% in the last month. Uh, it seems like they finally found their groove and the way they're marketing it is so interesting too. You can use it as a personal AI agent to save you money. They had their Meta Connect thing last night. They kind of showed off some examples. They have this cute little mascot which looks like a Labubu and it's like they're kind of going all in on that and making it very approachable. And so it's just so interesting to see the stark contrast between how Anthropic markets versus like how Meta is doing it and like treating AI as your friend. What is your take on Meta now finally finding their groove? Have you tried out Meta Muse and do you think this could potentially be like Meta's next big product?
I think it's a good product. Um, I've been using Muse for a few days now. I've been running some experiments on it. I'm not like — there's no way in hell I am giving it access to my Gmail and my bank accounts and all the other stuff.
