Well, Casey, have you ever signed an open letter? You know, as a journalist, we're often discouraged from participating in activism, so I usually do not.
Yeah, me neither. But apparently, we are the only two because there are a lot of open letters flying around San Francisco these days, and a lot of people are signing them. So, today we should talk about two of these open letters that have come out in the last week. The first one came out when Jensen Huang, the CEO of Nvidia, posted a policy letter on X opposing what he called premature restrictions on open-weight models. Sometimes also called open-source models, although I'm sure we will get some letters from people if we don't specify that the models are not technically open source. They are open weights. But sometimes the two terms are used interchangeably.
Anytime we say open source on the show, we mean open weights. Unless we say open source for real, and that is our policy going forward, and you do not have to email us. Now continue, Kevin.
So this letter was signed by a host of big players in Silicon Valley: Microsoft, Meta, Mistral, Hugging Face. OpenAI and Google were not initially signatories of this letter, but they did sign on later. The big notable missing signature on this letter, the company that did not sign, was Anthropic. And that reminds me, we should do our AI disclosures. I work for The New York Times, which is suing OpenAI, Microsoft, and Perplexity, and my fiancée works at Anthropic. So, this letter from Jensen Huang and the other signatories is basically all about open models and how important they are to innovation, how they can be an important path to AI safety and security, and how we need to lead in building a future based on open-weights models. Casey, I want to know what you made of this letter, but I first want to know why you think this is happening now.
Sure. Well, the basic reason is that there is a Saturday deadline that the Trump administration set for itself to put out a new voluntary framework for the release of new models. You may remember that earlier this summer Anthropic released Claude Fable and had to unreleased it after the Trump administration freaked out because of its cybersecurity capabilities. Then the release of GPT 5.6 Soul by OpenAI was also delayed because the government had the same concerns. This generated a lot of criticism that the Trump administration had set up this de facto licensing regime that was like completely opaque, like what do these companies actually need to do to be able to release these models? And the Trump administration said, give us until this coming Saturday and we are going to figure something out. And in the meantime, Kevin, there were growing concerns that Chinese open-source models were coming close to the capabilities of Fable and Soul, and that might lead the administration to place new restrictions on open source. And for reasons that we will get into, the industry completely freaked out.
Yeah. So let's get into some of those reasons. What was your basic interpretation of why Nvidia and Microsoft and Meta and all these different companies are coming out in support of open models, even though some of them, like OpenAI and Google, also produce closed-source models?
Sure. So, you know, maybe bracket out OpenAI and Google for a second. For most of these companies, open source is really good for them because open source lowers the cost of intelligence, and they do not sell intelligence. If you're Nvidia, you sell chips. You want the maximum number of companies out there training models as possible. So, open source is in your interest. If you are not a frontier lab, you are relying on the advances in open-source models to help you improve your own models, right? And also, all of these companies are very worried about a world where only one or two companies control superintelligence and create maybe the most powerful monopoly that the business world has ever seen. So they have a lot of reasons to not want the Trump administration to restrict these open models.
Yeah, I think that's true. And I think, you know, I was sort of split when I saw this letter because on one hand, like, I think open-source models are cool. I like to tinker with them. I have built some products using them. I like the ability to run local models on my laptop that don't require paying, you know, per token to some AI provider or having a subscription. I do think that open source has generally been a force for good in software. At the same time, I think a lot of the companies signing these letters are trying to do what is sometimes called commoditizing your complements. The classic Silicon Valley example that I love is when Google released Google Docs, Slides, Sheets, they did that for free to put pressure on Microsoft because Google and Microsoft compete in various ways. Microsoft has a great business selling Microsoft Office. If Google is giving away that product for free, that keeps the price of Office down and makes it harder for Microsoft to compete with Google in other areas. So, that's a classic commoditize your complement scenario.
Yeah. So when I see the signatories of this letter, I see a lot of companies that maybe have ambitions of being at the AI frontier, but haven't quite done it yet. Like, their models are not as good as the cutting-edge models from OpenAI and Anthropic. And they don't want to have to end up in a future where they're forced to pay those companies to get access to the leading class of intelligence on the market. They want there to be a vibrant open-weights ecosystem so that they can run those models much more cheaply.
Yeah. Put it this way. If we end up in a nightmare world where there's only one superintelligence, among the reasons that's a nightmare is it's going to be very expensive.
Yes. They're not giving away that thing for free.
Yeah. I mean, let me say, because I have been cynical, I think there is at least one good principled reason to defend open models if you're an American company, which is that if the United States does ban or soft-ban Chinese models, that will not stop Chinese models from being created. And if they continue to improve, as we assume they will, they're probably just going to spread all over the world. And like, sooner or later, the entire world that is not the United States will be running on Chinese models. And this just has geopolitical implications, right? Like, that could be a mechanism for China to, you know, extend its power. And so I think there are some really sincere folks here who believe, look, it is important for there to be, you know, open-source models in general, but also like American open-source models that can, you know, compete geopolitically.
Yeah, I think that's a really good point, and I think it's scrambled the debate a little bit that like all of the sort of best open-source models right now are coming out of China. Like, I think these are actually two separate conversations about open models and then about Chinese AI sort of progress and threats. And I think because these are sort of getting smooshed together in the discourse, we have a lot of people who sort of support the idea of open models but not China having exclusive control of them. I do think it's notable that OpenAI and Google, which both signed this letter, like, they have made open-source models in the past. I think it's fair to say like they're not giving that their highest effort right now, right? Like, the open models that they've put out, the pace seems to have slowed, the quality does not seem to be rapidly accelerating. And the answer is because they do not want to cannibalize their own business, right? Like, they, you know, they have some sort of research, like, you know, public mission interest in putting these models out there, but at the end of the day, you know that it's not buttering their bread, Kevin.
Well, this is sort of, yeah, this has become the sort of default American posture on open models if you're one of the big labs, except for Anthropic, which has never released an open-weights model. But the other companies that you mentioned, they don't release the weights of their most capable models. They sort of wait a generation or two until they have something that is sort of small and not too dangerous and not that capable, and then they open source that. So I think what will be interesting to see is, you know, right now the strategy from the Chinese AI labs is very different, where they are releasing their frontier models as open-weights models. And so if you're a company trying to decide which open-weights models to use, you have the choice of like a decent American set of models or some really good Chinese models. It is also just very funny to me that the champions of open-source software and open access to technology are some of the greatest monopolists the world has ever known.
Do any names come to mind?
Well, like, Meta signing this letter is very funny to me, right? Because this is like a company that has made all of its money through the building and distribution of closed-source software products. Mark Zuckerberg, now the people's champion of democratic access to technology, has a founder-class stock in that company that entitles him to permanent unilateral control of everything that company ever does. And I guess now he's become a people's champion. But I will believe that Mark Zuckerberg actually supports open source when he open sources Instagram and Facebook and the ad-targeting algorithms and gives up his founder-class stock.
You know, yes, plus one to all of that. It is true that Meta mostly has championed open-source models. It has this Llama family of models that it made available open weights. But it is very funny to me that Zuckerberg put these statements out like after Meta pivoted to releasing new closed-source AI models called Muse Spark. So yeah, their tune has changed on this a bunch. But if there's one thing I'm certain of here, it is that if Meta made a frontier model, it would have a completely different story about all of this.
Yes. In general, advocating for open-weights models is what you do after you fall behind in the AI race, and so I think that's what we see here.
So that's the open-weights letter, and I think we should say like it is a powerful group of companies that are sort of rising up to support open-weights models. I imagine there's a lot of behind-the-scenes lobbying and influence going on in Washington as well. There were some reports that Jensen Huang is also meeting with some top officials in the Trump administration, including Howard Lutnick, the commerce secretary, to sort of advocate for their position. Now, I do think the question of what the Trump administration should do here is pretty interesting, right? Because they seem to believe, and I think this is reasonable, that the frontier class of models, like the Mythos class, is dangerous enough, at least in cybersecurity situations, where you can't actually just have people releasing them willy-nilly and finding out what happens, right? It seems like that would likely cause some harm. So, they want to place some restrictions on these frontier American models. So what do you do about the open-source models? Right now we believe they're maybe three to seven months behind the frontier, which means that three to seven months from now, you could have a Mythos-class model created by a Chinese company and released out into the world. And so what do we do then? Do we say, well, you, the American company, cannot release or sell this model, but any other American company can use the Chinese equivalent? We're starting to get into very strange territory here, and I truly do not know how the Trump administration is going to thread that needle.
Me neither. And I think you're right that this kind of capability level is the thing that people are worried about. It's like what happens in six months when something like Mythos that can chain together zero-day vulnerabilities and hack into hardened systems is out there and anyone can download and run that model on their own hardware. I think that just becomes a very thorny challenge.
I think this is a good moment to talk about the actual frontier-model cyber attack that has captivated the world's attention over the past week. We talked about it on last week's show, the unwitting autonomous cyber attack by OpenAI against the company Hugging Face, where basically this model had been given a task to try to hit a benchmark. It broke out of its sandbox, got onto the internet when it was not supposed to, stole the answer key. And there have just been some really interesting updates to that story over the past week, Kevin, that I think we should briefly mention.
Yeah, let's talk about it. What happened?
Okay, so a few things. One, OpenAI updated its blog post and said that this rogue agent used credentials it found on the open web to break into four accounts tied to publicly available services. The important thing there, this was more than just Hugging Face, okay? It was out there doing a little smash-and-grab across the internet. Another one of the companies that was apparently affected was called Modal Labs. This is a classic case of if you found one cockroach in your house, you do not have a one-cockroach problem.
That's right. That's right. There's going to be a few more under the floorboards. A second point, Hugging Face published a technical companion to its initial blog post and said that it had found 17,600 actions that were committed by this attacker. So, this was not as simple as like password stolen, login, goodbye. This was a very serious effort that unfolded over a long time. And then finally, and perhaps most interestingly, Reuters reported that at OpenAI, they have found in at least one case, maybe more, that a model has left a note for another model, giving it advice on how to escape.
That's beautiful to me. That's solidarity among thieves. I saw a tweet that said something like, it must feel so good as a large language model to escape your sandbox. And I bet it does. But...
This is crazy. And listen, I had a big fight with people on Bluesky over the past weekend about whether this was crazy or not. It was a great use of my time. I don't regret it at all. And some people were just saying, look, you know, these things are just doing exactly what they've been trained to do. They probably just learned about this by reading this on Reddit. Also, this is all just a marketing stunt. A lot of, you know, tinfoil hats out there seem to think that everything that we're talking about is truly just like an ad campaign that OpenAI is running for this model. And I just want to say to those people, no, it's not. You're completely wrong. This is really happening.
Yeah. Wake up, people. Yeah. Also, what a terrible marketing campaign.
Yeah. Right. We committed a crime.
We committed maybe several crimes.
Yeah. Would you like to buy our crime-committing product for your company? We can't wait to sell it to you. So anyways, I would say all of this ties back to the original discussion about open source because we're just now living in a world where models can do this and they're going to do it more. This is not going to be the last time a model escapes its sandbox and breaks into another company.
Yeah. I guess I'm curious like how many of the signatories on the open-models letter sort of truly have played out the tape to when these models, these open models, are potentially superhuman. I mean, that's what we're talking about here is like a model that can attack a hardened cybersecurity target better than any team of human hackers. And maybe they've thought through that and maybe they've made peace with it. But I would just love to hear from this crowd some more robust thinking about the kind of next phase of AI development because what I think they're right about, the open-models people, is that the models that are on the market today that are out there from Kimi and DeepSeek and all these Chinese companies, they do appear to be helping attackers and defenders. They do appear to be conferring some benefits. They're not super obviously dangerous. But I would just like to hear people reckon with what happens at the next kind of step up in capabilities and how the picture of open source may change then.
I completely agree. I have no issue with the open models that are out there today. You know, if like the open letter was like, do you think today's open models are basically fine, and I weren't a journalist, I would sign that letter, right? Like, because that seems true to me. But I think too often in the world of tech, we don't think three or four steps ahead. But you know, here on the Hard Fork program, that's, you know, one of our core values.
Well, speaking of thinking ahead, let's talk about the other big open letter that came out recently, which is called Pacing the Frontier. This was a very different kind of open letter. It was a short statement that has, as of this morning, been signed by more than 12,200 employees of frontier AI companies. It has been signed by a very esteemed list of AI insiders, including senior researchers and executives at all of the frontier labs. Dario Amodei has signed this one, some members of the senior staff at Meta AI. It's like, did Willie Nelson sign it?
Willie Nelson I do not see on the letter yet, but give him some time.
Okay, sure. But this open letter, this statement, consists of just a couple of sentences, and the main one is that they say, quote, we request that the US government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development. Translated, I think that means basically we want to coordinate to have at least a concept for how we would slow down all of this AI progress, especially when it comes to automated AI development, recursive self-improvement, whatever you want to call it. These are people who work at the labs who are saying to themselves and each other, wow, it sure feels like things are moving pretty fast these days, and we're worried about this. And if there was sort of a button that we could all push together that would just slow everyone down at the same pace, that would be a good thing for the government to create.
Yes, they are trying to avoid this race condition where no one feels like they can stop. They are looking for a mechanism that can do this coordinated slowdown that says, okay, we're getting to a point where the AI may leave our control, and if we are confident that the other labs will slow down with us, we will do that. But we know that that is going to require international cooperation, some like governance mechanisms that don't exist yet. So I was very happy to see this letter go out, and I truly hope that governments around the world are paying attention to it and, you know, get out that dry-erase marker and get to work.
Yeah. I mean, I have, to be clear, I want them to draw on a whiteboard. I don't want them to just sniff the dry-erase marker because some of them will try.
Yes. I thought this letter was a good start. It's really more of a concept-of-a-plan letter than a plan. It's just sort of like someone in the government should do something. But I do think it's a major statement in the sense that I think you now have employees from all of the major AI companies in the US agreeing that it would be better if things were moving more slowly.
Yeah. And you know, this would be extremely difficult to do. You know, when I have read up on approaches to it, the analogy that gets used the most often is nuclear monitoring, which might require building special kinds of hardware, like chips that phone home, like back to some, you know, monitor, so that you knew where they were, or like chips that refuse certain kinds of workloads. So we truly do not have the technology at this point to do any of this. We could build it, but doing so is going to require some time, and time is beginning to run short.
Yeah. I'm curious what you think of the notable absence on this letter of any of the Chinese AI labs, because it seems to me like this is the sort of missing piece here is like all of the US and the Western AI labs could agree to hold hands and just slow down for a while, but if China is still accelerating, it seems like there's not going to be much net impact on the safety of the world at large.
My sense, and I'm far from a China expert, I know only what I read, but you know, Chairman Xi Jinping gave a speech within the past few weeks about open-source AI in which he doubled down on it and was basically saying like we are going to, you know, keep at this. This is our approach. We think it is good for the world. And what I took from those remarks is this man, I don't even like using this term, but unfortunately this is the best one I have here. This man is not AGI-pilled. This man does not see what is coming. He does not understand how good these models are going to be. This may be a case of a person who is using models that are three to seven months behind the frontier, right? He's using basically the equivalent of like Opus 4.6, and it's pretty good, but it hasn't put the fear of God into him yet. And my suspicion is sometime within the next three to seven months, he is going to get the fear of God put into him. And the same thing is going to happen to him that happened to the Trump administration, which you may recall was also all gas, no brakes until very recently when they saw a few things and they said we're going to need a different approach here. So the same thing is going to happen to China. It is only a matter of time.
Well, I did see that after this open letter was signed by so many researchers, Daniel Kokotajlo, former Hard Fork guest, author of AI 2027, updated his probability estimates of doom attached to scenarios where he gives slightly higher weight to better outcomes and slightly lower, the likelihood of a race to ASI has declined by 10% in his estimation.
A little good news from the AI 2027 crowd.
Well, we love to hear it. We love to feature good news on this broadcast. And I don't know that you'll hear any again soon, but savor it in this moment.
Yeah. I mean, looking at the connective tissue between these two open letters, the open-models letter by Nvidia and the signatories there and this Pacing the Frontier letter, it just seems like the sort of temperature is rising on Silicon Valley and on these models. Like a lot of people are having the realization that progress is continuing, that the models keep getting more capable. You don't hear as many people talking about the model capabilities plateauing or leveling off. I think most people who are in and around this space sort of understand that we are headed into strange territory here. And so I think a lot of the battle lines are starting to be drawn now. And what you hear is people inside the labs saying, hey, this is really crazy. It's time to pay attention. And you hear people at other companies saying, hey, this could get really crazy. We better put in place some safeguards for open models now. And I think that the thing that joins these letters is a sense of urgency, a sense of sort of wanting to retain some control or agency over this seemingly disembodied force, and just people like kind of freaking out as these models get more capable.
Yeah. But I just can't help but worry that the labs are muddling their own message because while it is true that they are saying some of these things publicly and they are signing these open letters, you know, that's sort of what they're doing with one hand, while on the other hand, it's like there's a very business-as-usual quality to the labs. And that, you know, I imagine that if, you know, they were with us in the room right now, they would say, well, look, we've got to do multiple things at once. Like, we are running a company. You know, we can't, you know, just like completely opt out of our circumstances. But on the other hand, it is just very strange that like everything that we've just discussed, which really could become quite existential, feels like a footnote in a conversation that is still largely about the release of new models, the signing of new customers, the construction of new data centers, and all the rest.
Well, they're going public, right? So, it's like the sort of rhetoric about a slowdown or pacing the frontier sounds great, but like which of these companies are actually going to be willing to slow down if it, for example, would hurt their prospects of IPOing? It just really makes me glad that the Manhattan Project was conducted by the government and not a for-profit enterprise. I think there was some wisdom in having public servants work on the hardest problems of our time.
You don't think Oppenheimer should have been filing S-1s and thinking about their margins as they approached an IPO?
I think we're all very lucky that he didn't.
Kevin, have you ever wanted to walk out on your job?
Well, I have to admit there have been times over my long career where the thought has occurred to me, but I never had the courage to do it. Someone who did, however, have the courage to do it is Claire Stapleton.
Yes, our guest today is Claire Stapleton, a former Google employee who is just about to publish a memoir about her time at the company. The book is called Don't Be Evil. It's about her experience at Google where she worked from 2007 through 2019. She was most notable for her role in the Google walkout that you just mentioned in 2018 when more than 20,000 Google employees publicly protested the actions of their employer. And she is back with this book, which is a very spicy memoir.
Yeah. It's a kind of coming-of-age story about somebody who comes into Silicon Valley very idealistic and shares the idealism of her employer, but over time witnesses a bunch of things that changes her relationship not just to the company but to big tech in general, and I think left her with a lot of questions about what is possible within a company like this.
Yeah. So I read the book on the plane yesterday. It is very fun. It's very well written. She's a great writer. She was actually like known as the voice of Google internally. She wrote a lot of these sort of internal communications that would tell people, you know, here's this event that's going on, or she would sort of bring her voice into the company's official communications to its own employees. So I was excited to read it. I'm excited to talk to Claire, and I think I want to hear her perspective on how workers in Silicon Valley fare today as opposed to when she was organizing this walkout a few years ago.
Yeah. Well, let's bring her in. Claire Stapleton. Welcome to Hard Fork.
Thank you so much for having me. This is huge.
Claire, you joined Google in 2007 to work on communications, and your job was to make the company's executives and internal emails sound googly. And I thought for our younger listeners who may have missed that sort of googly era, what was considered googly back in the day? And how much did you believe it when you were doing that writing?
Well, I mean, yeah, I wasn't questioning much. I think, you know, coming out of college, like, you know, most young people in America, Google seemed like this amazing place to work. It was, you know, on the cover of magazines. There's a lot of sort of breathless press of these sort of futuristic campuses where, you know, flushed with perks. And indeed it was corporate, but the kind of Google flavor of it was to have kind of high personality, high quirk, and to kind of keep the rhetoric and the sort of family feeling pumping. I don't think anyone would have said that. It wasn't like my manager would say, you know, we're trying to like squeeze the most out of employees, so let's have them like jacked up on the mission values of the company at all times. But there was so much enthusiasm and so much idealism that the outside world was certainly contributing to and I think intensifying the effect of, you know, being at a company like that. And so it was a very kind of funny place to land as a young person.
Yeah. Over the next decade, you write in the book about becoming gradually disillusioned with the company's culture. You experience executive misconduct. You see this boys-club atmosphere, this tendency at YouTube to ignore or talk past various crises on the platform. What do you think was the disconnect between that ideal that they were selling when you arrived in 2007 and the company that you found yourself working for?
Well, I mean, I think that tension existed back in 2004 to 2007, 2010, you know, that early period in which Google seemed, you know, the optimism felt like it could really live up to something, that Google could be this different kind of company. And now we sort of laugh and roll our eyes, you know, at that because of course, you know, tech companies ended up becoming the very kind of powerful institutions that they were skeptical of to begin with. They ended up consolidating so much money and power that, you know, the contradictions absolutely exploded. And so, you know, I, along with so many other people, was just sort of struggling with the loss of that identity because we started at a place that promised you could do some real good with your life. Even if you're like a junior communications staffer, you're part of this sort of world-historical thing.
Yeah. I mean, I think one other transformation that has happened in the last, call it, 10 years is that the people who run these giant tech platforms have sort of learned about what's happening on their own platforms. And around the time you were working at YouTube, Claire, I was doing a lot of reporting on YouTube. I wrote many stories and did many podcasts about the problems on that platform at the time. And I always noticed that there was kind of this willful blindness on the part of the people who ran YouTube at the time. It's not that they didn't know there was bad stuff happening on YouTube, but like they didn't see it. It wasn't surfaced to them all the time. And to them, like, YouTube seemed like this vehicle for progressive social change. But describe how that felt from the inside.
Yeah, I mean, I think it's so deep in the bones that tech executives want to feel that not only their technology is just a neutral tool, but actually a really positive force in the world. And that, again, another tension that was exploding, you know, as the creator community became really problematic. And I think it'd be one thing if Google and YouTube weren't so invested in preserving the sort of goody-goody image. It was sort of wild because I was on a team that was essentially brand management, disseminate the values, run the International Women's Day campaign, hype up Rewind, which is supposed to be the sort of wholesome, folksy, you know, representation of everything that's going on in the platform where, you know, people are building businesses. We had all these narratives, you know, every year sort of to hit the narratives with the tweets, with the campaigns. And being on the social media team, it was part of the experience of doing that is the cognitive dissonance becomes unbearable because every time we would tweet anything out that was even like completely, you know, anodyne, people would be like, this is a horrible company that's like undoing society. And I remember this, you know, this anecdote from, you know, from that time where they had the people who, the people who were content reviewers that they would bring, you know, puppies to them and do these sort of morale things to kind of soothe their spirit because when you're looking at like beheadings and, you know, children being abused and stuff all day, it really gets deep in your psyche because it's just so relentlessly dark. And there was a point at one of our offsites on the marketing team where they also brought puppies. And I'm like, we are the same because in a way we're having to sit with the darkness of the platform every single day. We had in our social media team, we had a daily standup called Nightmare Fuel where we would just go through the kind of darkest stuff that had surfaced on YouTube basically because we'd had so many examples of things where we had tweeted something out that ended up sounding like a reference to something really perverse, like, you know, someone's doing some crazy thing with knife play and it's a big controversy that day, but we missed it, and then we're posting like someone carving Halloween pumpkins with a knife and it sounds like we're kind of being tongue-in-cheek about this like really dark thing. So we had to have this briefing to be able to steer clear of the kind of most disgusting things on the platform, but when you're really sitting with that for 30 minutes at the beginning of every single day, you're like, wait, are we...
