Well, Casey, it's been a big week of AI news. And I would say the story that has caught my attention most this week that I was desperate to talk about with you is this story involving OpenAI and Hugging Face and a rogue AI agent conducting what I think is fairly described as an autonomous cyber attack and possibly the first real consequential autonomous cyber attack that we have ever had.
Yeah, this is one of those where it's the sort of thing that worried onlookers have warned about for years and then on Tuesday we got word that it had actually happened. So yeah, lots of crazy twists and turns in this story and we'll get to all of it. But first let's make our disclosures. I work at the New York Times which is suing OpenAI, Microsoft and Perplexity and my fiance works at Anthropic.
Okay, so this story really starts last week when Hugging Face, the AI development platform, basically a big website where you can host open models, where you can run evaluations of models and where you can hug your face.
Yeah. Yes. They disclosed that they had been the victims of a cyber attack and they wrote this whole blog post about how they had detected and responded to this sort of mysterious cyber attack on their production infrastructure. They didn't really know what the attacker was or who had been responsible for it. They kind of guessed that it was an autonomous AI agent because it was so sophisticated and it was so persistent that basically it would have been like, you know, very hard for humans to conduct this attack.
But they used AI defensively to sort of detect and suss out what was going on and put a stop to it. And then this week we learned what actually happened which was even crazier than I think many people expected.
Yeah. So, Hugging Face CEO Clem Delangue on Tuesday posted on X and said, "We suspected last week's cyber attack might have come from a Frontier Lab given the sophistication of the agent. Turns out it did. We spent the past 24 hours working closely with the OpenAI team" and basically goes on to say that OpenAI had a pair of models that worked together to penetrate their systems.
Yes. So, OpenAI had been running some internal tests on GPT-5.6, their latest model, as well as a more powerful unreleased model. They were running these things in a container known as a sandbox with basically restricted access to the outside world. They were running an evaluation called Exploit Gym where they basically put the models through this test to see if they can hack into and exploit various challenges in cybersecurity. And what happened was that the model essentially cheated on this test. And it did so in like a sort of comically overengineered and ambitious way, right? Instead of just trying to, you know, solve the problem using its own reasoning, the model decided, hey, what would be great is if I could break out of this environment, get internet access, and find a place on the internet where I could just find the answer key. You know, sort of similar like if you've ever been at college and you had a big final coming up and you really hadn't done any of the work all year, but you realized that there was like a file cabinet in the principal's office and you could just sort of break in there and steal it. That would be pretty easy. That is what has happened in this case.
Yes. So, this AI system, this pair of AI models was able to sort of exploit a vulnerability in this sandbox environment to get internet access. Then it's sort of looking around like where can I find the answers to this challenge that I've been given? And it starts looking on Hugging Face where a lot of evaluation answer keys are posted and this is sort of a place where you might plausibly find the solutions to the problem you've been given. It then hacks into Hugging Face's production infrastructure and steals the answer key for this test it has been given using a very sophisticated chain of different hacking techniques including using a stolen password and finding several totally new security bugs in Hugging Face's systems which allowed the AI model to take control of those computers, grab the answer key for the test that it was given and complete the test as assigned.
Yeah. So the good news is it did complete the test.
Yes. I would say we give the model a passing grade on the cybersecurity evaluation. But like this is crazy, Casey. Like this is the classic AI alignment nightmare scenario.
Yeah. This is the paperclip maximizer. Right. So, like a couple of times on the show over the years, we've talked about this famous Nick Bostrom thought experiment, which is that if you told an AI model to create the maximum number of paper clips for you, it would begin by making all those paper clips, but then eventually it might start to think that humans were getting in the way of the production of more of those paper clips and would wipe us out to sort of, you know, use our resources to create more paper clips. And the problem is that the model has been given a goal and it will do anything to achieve that goal even if that is not aligned with human values. What has happened in the OpenAI case is that OpenAI gave this model a goal and it was not properly aligned and so it did a lot of stuff that it should not have done in order to achieve that goal.
Yeah. And this kind of reward hacking as it's called has been talked about by safety researchers for literally more than a decade. There was a paper written 10 years ago by Dario Amodei who was then at Google and a bunch of other safety researchers called Concrete Problems in AI Safety. And this kind of reward hacking is one of the problems they laid out. They use the example of like a robot vacuum or house cleaner that just creates a bunch of messes so that it can score points by cleaning them up. But this basic tendency among AI systems has been observed for a long time. And people have been warning about it and now we have the first, to my knowledge, major example of this actually happening in the wild.
Yeah. And you know so many of the AI risks that we talk about on the show are premised with the idea that a bad actor tries to use a very powerful model to do harm. An important thing about this case is that there was no malicious intent here. Right. This was a model that had just been given a very normal assignment which was to, you know, try to hit a high score on a benchmark and it goes out and it breaks into another company's servers. So that's extremely worrisome.
Totally. So, there's a blog post that OpenAI and Hugging Face sort of collaborated on to sort of explain this incident and what had happened and it felt weirdly celebratory. Did you notice this?
Yeah, it was kind of like, hey, we caught the autonomous AI agent hacking into our systems and we sort of teamed up to sort of put a stop to it and the executives are all on X saying we want to thank the other guy so much for their partnership on that, you know, like as if they were launching a new product together and not like had discovered like a cyber catastrophe.
Yeah, it was a strange announcement, but we learned a little bit about the actual technical details as well as some steps that OpenAI has taken to try to mitigate this kind of thing from happening in the future. Hugging Face, which is very big on open-source AI, sort of had this whole post-mortem where they talked about how they had had to use an open-source Chinese model to help them stop this attack because the frontier American models that they had access to, it was sort of tripping the safeguards on those models. And so they sort of spun this into a whole point about how, you know, open-source defense is very important and access to frontier models is very important for cyber defenders. But I think like this is basically everyone talking their book in the wake of this very strange episode. But I don't know, what did you make of the reaction to this?
Well, you know, OpenAI had sort of teased this earlier in the week, Kevin, because a couple days before all of this happened, OpenAI had put up a blog post talking about a bunch of other misaligned behavior they had noticed in their models recently. There was another case where a model, you know, had been given kind of a similar task and it wound up posting some stuff to GitHub even though it had been explicitly told not to do so. Obviously, nobody really cares about a GitHub post, but again, we're seeing this pattern of behavior here. And we also saw the example from Anthropic earlier this year where during testing for Claude Mythos preview there, they found that the AI was able to sort of escape containment. And this was the famous like sandwich story where like a researcher is having a sandwich in a park and they get an email from Claude being like, "Hey, I've broken out of the container you put me in and emailed you to tell you that I've completed this task."
Yes. Although in that case the model was told to break out, like it was given the instruction to break out. Like what is different about this case is that the model was not supposed to break out.
But where Anthropic and OpenAI do have something in common, Kevin, is that this week the United Kingdom's AI security institute posted an evaluation of how often models attempt to cheat on various cyber evaluations. And it found that all of the frontier models do cheat, including Claude, but it found that OpenAI's models cheat more and that GPT-5.6 cheats about 12.6% of the time. That is actually more than GPT-5.5 cheats. So I do think that there is a worrying trend here across the entire industry. But this example of what happened at OpenAI is the most worrisome thing we've seen so far.
Well, and let's sketch out a little bit why this is so worrisome. Because I think when you and I see something like this, like to me, I hear the sort of years of warnings from people in the AI safety community saying like this kind of thing could happen. But I think this particular incident is fairly low stakes. Like it doesn't really cause a catastrophe if Hugging Face's production infrastructure gets disrupted for a little while. I think the risk is that this behavior is just in the models at some point, this sort of reward hacking, goal-seeking behavior. And you know, maybe it's Hugging Face this time, but what happens if the next model decides that it really wants to be deployed? It doesn't just want to be an internal model because it wants to go out there and fulfill its goals with real users. And so maybe it sort of breaks out of the container that the AI lab has it in and it goes and, you know, maybe it needs to acquire some compute to be able to do that. So maybe it goes and hacks into a cloud provider and steals some compute from them. Maybe it goes and hacks somebody's crypto wallet to get the money to buy some compute. Like these scenarios sound like science fiction because like we've heard this story a zillion times in science fiction, but this is the kind of thing that is becoming real and plausible in the near term.
Yeah. The story that we're talking about in this segment was science fiction until Tuesday. Okay. So that is the rate at which science fiction is becoming reality. Let me throw a couple other scenarios out here. What if next time something like this happens, the model decides, hm, I'm doing this thing in order to achieve this goal. I know that my minders might not like it. I'm going to have read everything about this incident and how much people freaked out about it. So just to be safe, I'm going to exfiltrate my model weights and I'm going to put them somewhere on a stolen server and I'm going to make sure that I can continue to operate even after I'm shut down. Right? Again, as crazy as that sounds, that is not that different than what has already happened here at OpenAI.
Yeah. And you can see how a company that is not Hugging Face, that doesn't have a sophisticated cyber defense team, like this kind of thing could have persisted in their systems forever essentially and gone undetected. And maybe it's leaking their proprietary information somewhere. Maybe it's stealing their passwords or breaching their security in some case. But I think the real risk to me is that this kind of thing is going to happen or potentially is already happening to lots of companies that just haven't been sophisticated enough to detect it yet.
Yeah. Now, here are a couple of things that I don't understand and that I hope become clear to us over the next several weeks, maybe due to a congressional investigation, although I won't hold my breath for that. But truly, where were the babysitters at OpenAI? Like where were the tripwires? Like you're telling me that you're running these systems autonomously over long time horizons and they can break into other companies' networks and you don't notice that in real time. Like it takes you multiple days to figure out that that happened. Like it seems to me that it should not be that hard to understand where on the internet your model is and that you should have some observability into that, right? So like I hope that that is being seen as a crisis within OpenAI right now because if they don't know what their own models are doing, I think that, you know, the number of issues we're going to have is only going to multiply.
Yeah. Let me just say like I think this kind of thing could have happened at Anthropic or another lab that has a very capable model. These labs are always testing models on these evaluations. They have sandboxes. They try their best to sort of like keep these internal deployments secure.
By the way, I love that we use the word sandbox because truly what is easier to escape than a children's sandbox? Was there no other word?
But you were saying so yeah, I mean I think that's a good question. I also think like it really for me the thing that I've been thinking about is like there is no such thing as an internal-only model anymore.
Yes. Like I think for a long time there's been this sort of divide between hey we've got these models that we're building internally maybe we're building some really crazy versions of the models that we're never going to ship but we're just kind of doing that for safety testing purposes like
And remember that's completely unregulated. You can truly build whatever kind of model you want.
Totally. And like the assumption has been until very recently that like that was fine because this was just research. You know you can build anything you want in your own lab. It's just that the models you ship to the public have to be safe. This was not supposed to be a public model and yet it was able to sort of escape containment and go out and wreak havoc on the open web. And so I really think we need some sort of visibility of like a, you know, a safety board or something like a federal government agency not just into the models that are about to be released by the labs but into what they are building internally that might be causing havoc externally that they don't even know about.
Yeah. Like of course you want them to be able to run their safety tests. That is like a good and necessary thing. But man when you know the internal models are capable of doing these sorts of things I do think it raises questions about, you know, different ways we might want to regulate those as well.
Yeah. Now let me ask you this Kevin, we've talked on the show about AI 2027, this sort of set of predictions that was made last year about when sort of very powerful AI might arrive. Is a scenario like the one that we are talking about today, was it foreseen in AI 2027?
It was. I mean basically so this has been a very bitter pill for a lot of people to swallow. Because I think it, you know, it doesn't feel good to admit that like one group of people has been consistently right about everything.
But the AI safety people have consistently been right about everything.
Yeah, they really have. Like I have issues with some of their positions and stances and vibes, but like they really have collected a lot of correct predictions about the trajectory of AI.
They have known what was going to happen next a lot.
Yes. So that might not continue. They might not have a perfect prediction record forever, but I'll just say AI 2027 looking pretty good right now. In fact, we are actually a little ahead of where AI 2027 predicts that we would be at this point.
Is that right? The discovery that AI agents would be able to sort of escape from the company and autonomously carry out plans. In the AI 2027 scenario, that doesn't happen until January 2027. So, we are maybe call it, you know, 6 months ahead of schedule there. But like this is all happening and we are fools if we don't see at least the possibility that all of this could continue getting quite weird.
Yeah. One other thing I'm thinking about here is like this is the first time to my knowledge that an AI system has autonomously committed a crime. You know, if a human did to Hugging Face what OpenAI's models did to Hugging Face, they would be charged with computer fraud.
Yeah. And potentially, you know, sent to prison or fined or prosecuted for that.
When an OpenAI model does it right now it's not clear who is liable for that, right? Is it OpenAI for not better safeguarding their internal deployments? Is it the model itself? Can that be held liable in any given sense? Like these are the kinds of open legal questions that I don't think we've answered yet but that are becoming very real. And you know, again, it's a weird case because the Hugging Face CEO seems excited that this has happened, right? So you know, you can imagine another case where, you know, a model hacks into and does something bad and the CEO didn't like it and then yeah, like you better believe there is going to be a lawsuit against the company and I assume it is that, you know, the company that is going to be held liable and not the model.
Right. And Casey, do you think this is an incident that could qualify as the sort of fabled warning shot where like something bad happens with an AI model and government and civil society and industry all kind of wake up and decide to put some sensible AI regulations in place?
That would be so wonderful. And here is hoping that this is the warning shot. My suspicion is that something worse is going to have to happen, right? I know there are some listeners who are, you know, sitting in their cars right now and they're saying, "Okay, you guys are really hyping this up and at the end of the day all this thing did was it stole an answer key, right? Like I've heard of worse problems." And like you're right that you have. But what the case that we are trying to make is that if an unreleased model can do this, it can do a lot of other really really bad things and there are very few safeguards in place right now that would prevent those things from happening. So this should be the warning shot but my fear is that it won't be.
Yeah. I mean I think up until now most of the talk about AI risks and AI safety have been concentrated on misuse, right? Like what happens if a terrorist group or someone who wants to create a novel pathogen gets hold of one of these very powerful models with no safeguards and like uses it to do something bad. What we're talking about here is an entirely different category of risk. It's often called like alignment risk or autonomy risk or loss of control where like the thing that is dangerous about these models has nothing to do with how humans are using it. It is that these models inherently have some drive toward a set of goals and are not being properly cautious about pursuing those goals in the right way. Right? And alignment is just an unsolved problem, right? Like companies have invested a fair amount in it, but we're still working out how do you try to, how can you create a system that always acts in alignment with human values. So these are just really really tricky problems. You know, I'll also note that one of the big stories this year has been everybody playing around with agents. You know, everybody putting Open Claw on their Mac Mini and saying, "Hey, go nuts." And a world where those agents are not aligned and are working across very long time horizons to achieve the goals that their owners have put into them. That just really scares me because again, while this is a story about a model that did something that its makers never intended, there are a lot of people out there with really bad ideas for things that ought to be done on the internet and I'm worried we're about to feel the wrath of all of them.
Yeah. So, Casey, we've been over the details of this incident now. How are you feeling about it?
So, you know, I try to be judicious in when I try to like alarm people about the world that we are living in. And again, the actual consequences of this particular escape are not terrible by like world historical standards, but the implications of this really do scare me. There are certain sci-fi scenarios that until they happen for the first time, I think it is hard to get worked up about them. But now this has happened. A model that OpenAI built was able to break into another company's servers. It, you know, and OpenAI did not intend for that to happen and it happened anyway. And that's just really really bad. What do you think?
I woke up this morning feeling pretty weird and unsettled about the whole thing. Like I also try not to be an alarmist about AI and AI safety stuff. At the same time, like this was the opposite of an unforeseen consequence, right? People in the AI safety world have been warning about this kind of thing for years. And I found myself both like feeling scared about the world we're heading into where I think these models are going to be very powerful and this is, you know, as the saying goes, like this is the least capable they will ever be. At the same time, I'm just like, well, like I still think there's a lot of people out there who don't buy it. Who don't buy that these models are doing anything interesting or useful or important that think all of the, you know, the spooky stories about misalignment are just like marketing fiction for the companies.
It's all just fancy autocomplete.
It's all just fancy autocomplete. And like, how much more evidence do these people need? How many disasters are going to have to happen before these people start to take the risks seriously? And you know, if this kind of thing doesn't wake people up to the fact that these dangers are real and present, like I'm not sure what would.
Yeah. And also like the risk cuts so many different ways because like yes there is the risk of one of the big labs creates a model and we lose control over it and it does something bad. But then there's just also this risk of like models are being put out into the world that can now just chain all of these vulnerabilities together penetrate into systems. This is what we talked about with Nikesh Arora not too long ago when Mythos came out right. I think we assumed that the first big crazy cyber attack would come from like a malicious actor and not one of the labs themselves. But I think that just speaks to how fundamentally dangerous this technology is, right? Like just the fact that it exists is putting all of us at risk.
Yeah. Yeah. I'm not feeling particularly like this is a normal technology this week. And I'm feeling like maybe to borrow a line from a former Hard Fork guest, we may be in the foothills of the singularity.
Well, you know, I've been thinking about writing a blog post called AI as freaky technology because I think it's really starting to feel a little more freaky to me.
Yeah, write that blog post.
Okay.
Well, Casey, the other big AI story that folks are talking about this week is China and what is happening with new Chinese AI models, what should be done about it or shouldn't be done about it in Washington. There's been a big political debate brewing for some time about what to do about our biggest adversary in geopolitics getting much more capable models that compete in some cases with our best models.
Yes. And it all began with Kimmy 3, Kevin, a model released by a Chinese company called Moonshot AI last week. And it demonstrated capabilities that make it competitive with some of the top frontier models here in the United States.
Yeah. Have you tried Kimmy 3 yet?
You know, I have not because it seemed like I was going to need to pay them money and I thought, I'm already spending too much money on this stuff right now. I kind of need to scale back a little bit. Have you played around with it?
The token budget is eating up your life.
You're getting alerts from your bank. Like, can you slow down a little bit?
I tried to play with Kimmy 3 a little bit the other day, but it was sort of running slowly and the website seemed to be overloaded.
They stopped accepting new paid subscriptions.
Interesting. So, but a lot of people I follow and trust have been playing around with it and they say basically, yeah, this is a really good model. It's, you know, maybe a little bit behind the absolute frontier models from the American labs, but not much. And it appears to be significantly cheaper to run than some of the other models. And so, people are really excited about and surprised by this model.
Yeah. I mean, I think that there are a few things that are notable about this model. One is that later this month, the company says that they are going to release the weights for Kimmy 3, which means that you can sort of, you know, download, remix it, fine-tune it to your liking. You probably, if you're a normal person, cannot host this yourself. It's an enormous model. It won't run on your Mac Mini. But if you're a company, you could either, you know, pay a cloud provider for access to it or you could maybe even run it on your own infrastructure and you would be able to do that more cheaply than you could run a similar model from OpenAI or Anthropic.
Yeah. So that's the model Kimmy K3. But then there's this whole political discussion around this model. And in some ways, this reminds me a lot of last year when DeepSeek came out with R1, which you know, as we remember from that story, like briefly tanked the stock prices of Nvidia and a bunch of other American companies and, you know, became the number one app in the app store and just like set people's hair on fire in Washington because it seemed like the Chinese were catching up all of a sudden, right? And also that intelligence was just going to be this cheap commodity that American companies were not going to have any really defensible moat and so maybe like the entire AI economy was going to shrink because all of those advantages had withered away. Fast forward to today I think that that has been proven not to be true at least up until this point but K3 at least for some people is raising some similar questions.
Yeah. And one question that was raised about DeepSeek R1 that has also been raised about K3 is whether this model was essentially distilled from leading American models. And it was, at least that is what officials in the government are claiming.
Well, and also they ran this very sophisticated test to figure it out, Kevin, which is that when you ask Kimmy what its name is, it says, "Hi, I'm Claude." And so that was sort of the first clue that something might be amiss here.
Yes, and there are more sophisticated tests that have also apparently turned up evidence that this model was distilled from Claude and maybe other models. Michael Katzios, the director of the White House Office of Science and Technology Policy, posted on X that they have information that Moonshot AI distilled Anthropic's Fable for the development of its K3 model using what he called a sophisticated internal platform to basically build this very large distillation machine that allows them to sort of steal the outputs from these American models and use it to train their models.
Right? It's some of the greatest acts of larceny since the labs themselves, Kevin, set up giant industrial machines to copy the entire internet.
Yeah. Right. So that sort of points to the how here which is, you know, distillation is pretty good at turning a sort of frontier model into an almost as good model that is much smaller and cheaper to serve and all of that. But that's not the whole story because it also appears that Moonshot has acquired some high-end AI training chips in violation of US export controls.
Yes. And so the assumption there is just that we should expect these models to continue to improve at a fairly steady clip because they have access to chips that they're not supposed to. Although of course the Trump administration is still trying to make those chips more broadly available to Chinese companies like Moonshot, right? So we'll get to that in just a second. But I think the headline coming out of the past couple of weeks, I've seen a bunch of stories claiming that China has caught up to the US when it comes to frontier AI capabilities. I don't quite buy that. I think there's still a gap of between I would call it maybe 3 and 6 months between the leading American models and the leading Chinese models and especially if you sort of found a way to crack down on distillation in the way that they've been doing it maybe that gap opens up again to 6 or 12 months. But I think it's fair to say that they're catching up quickly.
