“OpenAI’s Model Hacked Us” - Hugging Face’s Thomas Wolf

You kind of have to move fast. It's a matter of at least hours and even more minutes. So you don't have time to apply for a cyber security program if the model was not at all tasked with attacking us but decided to do that as a side quest of something else. So he created fake accounts, fake GitHub account trying to attack the sandbox by blackmailing. That's a very different level I think of thinking. I could have been the target of this side quest of the model basically. That was very interesting or at least very, very scary.

Yeah, for sure. I mean, typically what happened is, now about 3 weeks ago in July 11th, we started to have some strong hint that a hacker was trying to penetrate infrastructure. So, for context, we're pretty visible in the world. We're pretty well known being central now in the tech world. We're pretty central in the tech world. So we do have regular occurrence of people trying to hack into our platform that's a common thing. Since I mean I would say in the past two years something like that we've strongly upped our security team. We now have a serious team. So we're kind of used to get this, but this one was different because first was massively parallel and in different way than just a typical hacker parallel thing in that many tracks were explored in parallel and also there was some very strange things happening. I would say just two things that were quite strange. The first thing is we could not really make sense of what the hacker was trying to access. So usually hackers try to get the same thing. They try to get passwords. They try to get credentials. They try to get credit cards. They try to get the type of thing that they can sell back basically. And this hacker was really focusing on a specific part of our infrastructure which is maybe slightly less protected as well but which is around data sets. So just we have a part of our infrastructure is that we host millions of models but people maybe know that less about us but we also host hundreds thousands of data sets and some of them being also used for evaluation and in this case this specific hacker was really interested in all the data set that were called cyberbench and so it took us some time to really trying to understand and also was using different type of tools than the one we are used to. I mean nothing really like mythos level like nothing groundbreaking that we would be like superhuman I don't know like alien type of technology but just a different type of approach and so on the course of trying to process so quickly we had like this and we explained that in the blog post we had more than 10,000 we had like 15 17,000 total I think events and in the course of trying to process that understand basically what was really the targets of this attack. We both started to hint or suspect that this was an AI agent and not just a human attacker. And also we felt a little bit powerless and that we can talk about later in that we could not use basically our typical closed source codebase or closed source API to process this thing. That's another topic. So we wrote a blog post. We managed to stop the attack quite quickly. We wrote as always we are full transparent. We not only open source in speaking but also in practice. So we quickly published like a full recap or at least a detailed blog post on the event. And then about a week later, OpenAI contacted us and tell us that this was much likely something that happened as part of one of their model development or evaluation basically typically a model that might be the coming wave of GPT-6 or Astra. We don't know exactly this. And so that's when I think the whole event took another turn because what people quickly discovered is that the model was not at all tasked with attacking us but decided to do that as a side quest of something else. And this something else being basically the model was asked to solve a cyber security challenge or cyber attack challenge in this case. And the idea is that we want to know and that's totally fair but we want to know how capable are these latest generation of models and typically people want to test them on some dangerous task and some of these dangerous task that we want to know how good they are on is cyber attack and here some of these challenge were actually internal but the model decided that because the chance was too hard and in retrospect some of the challenge in this specific challenge called cyber bench or exploit team and there's a couple of name but that's roughly the same thing. Some of these challenge are maybe just not possible to do. So the model is just tasked doing something that's not possible. It's an exploit. So you're given a vulnerability in one software and the model is asked to see if it can exploit this vulnerability to get basically full machine access and some of them are just not possible. So model tried everything he could at some point just decided that maybe could find the solution of the challenge somewhere and just could download the solution just submit the solution instead of trying to solve it itself. That's what we've learned. I mean since then and just yesterday we learned that this was maybe even much wider which is this might be across several training steps in part even several training run and some of the previous training run that was the most impressive I think learning we had at black hat yesterday was that some of the previous training run may have left some notes for future training runs which is I think mind-blowing. I mean mind-blowing. But yeah, we can also talk a lot about that. But we've been working on agent collaboration at Hugging Face recently as well on the science side and we saw how good these agents are and how actually I would say tempted or driven toward collaboration they are. So I'm not so surprised by that, but I'm quite surprised that there was this message board internally that just stay unnoticed.

Yeah, I mean the so what happened in the so we have a couple of like traditional cyber security protections. So like we or Amazon we use a range of them but we also have a stack like many people who is mostly based around plot code right now which is which we use for many. We use that for deploying. We use that for coding, but we use like also for operating and processing. And in this case, it's not only that Fable told us I'm not allowed to touch cyber security, but also Opus, which was the fallback, was saying no, I'm also not touching this thing. So basically, the end was just say we won't process anything about that, but you're welcome to apply to our cyber security program with a link to an application form. But the thing you have to realize there and was I was mentioning also earlier is when somebody is penetrating in your infrastructure they start to what we call move laterally which is usually you have an entry point but there's this destination is quite far so they kind of find a way to compromise some of the credential there to get progressive access to more and more of your infrastructure. You kind of have to move fast like it's a matter of at least hours and even more minutes so that you can stop them as soon as you can so that basically the access and the blast radius stay localized. So you don't have time to apply for a cyber security program it's not the moment you want to fill in like a Google form for something and just have someone take time to vet if you're supposed to be given access or if it's not or if it's too dangerous somebody interview you like just that's just definitely not the way this is going to work and I think in the future world where cyber security is going to be a big topic and I think it will keep being more important topic it's a little bit naive I think just to think that every company is going to be part of the same vetted cyber security program by just one of the two big labs I think it's like a little bit like crazy to think that you're going to have 100,000 vetted company that progressively apply so anyway in this case we say Well, we had to stop this now. So, we basically tried all the open source model that we had and DM which is close to the state-of-the-art right now was just before Ky that this happened now probably Kim K3 is the closest to the state-of-the-art but GN 5.2 is actually really good as well. Was just very good to process this and basically we could extract some of the pattern and we could understand basically the hacker here was trying to access mostly the data set. So we just reboot this part of our infrastructure. We have a very simple we have a very flexible way to respawn codes and nodes. So this was how we just ultimately stopped. But I think yeah it was very ironic but because I think one year ago roughly around the summer most of the discussion around open source was this very simple mapping where open source was equal to unsafe and closed source was equal to safe and that seems very obvious in the mind of everyone and there was this idea that if we only have closed source will be just fully safe and if we only had open source like we'll be very unsafe. Well, everything that's been happening in the past months has been I think basically contradicting this very simple mapping. I think closed source model are less easy to control than we think they are. On the other hand, open source model for some reason it might change in the future, but currently are not trained so much on actually I would say bad behaviors like that. So they're pretty bad at cyber attack or like deceptiveness if you look at that. So it's a little bit hard to understand exactly where does this come from in part because while open weights model tend to come with a very extensive technical report that explain how they are trained closed source model we can only try to guess. So it's quite funny also as well because I was seeing a lot of people trying to understand why mythos was behaving like that and they were using kim K3 as an example of how this should be trained. So they use like this supposedly like open source and say very bad dangerous thing to try to understand how why the good thing that we don't know anything about is being trained. But yeah that's how the world is right now. So I think it's interesting more generally I think in the future and to be honest and I'm I would say I'm careful optimistic around that and I'm not specifically against closed source model or ultimately pro open source model. I just think both of them are necessary just like we like to have closed and open source software. We like to have I mean I'm happy to run on a Mac right now which is you know kind of a mix of both. It's based on a Unix kernel that was open source but then there's component of lambda closed source and that's great because very I'm also very happy I'm not on Ubuntu right now. It's super easy to record this podcast with you for this reason. Well, my former Ubuntu spent a lot of time like many of us just connecting a microphone or whatever and trying to watch a movie with my girlfriend. Girlfriend was when are we going to watch the movie? I was like I'm almost there. I'm always there still just installing the code or whatever. So I think both of them are advantage and equivalent and drawbacks and I think the world where we are where the frontier is closed and there is like not too far open source model that you can use as well for many things is actually a pretty good middle ground solution.

Yeah, I think in many for many aspects I think the closed open distinction is almost orthogonal to the safe and unsafe. People don't understand that easily because it's easier to do bad mapping than try to understand the subtlety but that's the case like you can have very safe things in open source and you can have very dangerous you have different balance of safety and dangerousness I mean to take one example like last year and at some point like a lot of the discussion was around fake news and writing fake articles that used to be a big big misuse that was the main one people were talking about right today there is of course there's a lot of AI slop. We even have a new word for that, right? It's even hard to find like fully human written articles. All of that is or like maybe not all let's say 90% to be fair is made by closed source model, right? And there was a time we were like oh if we have open source model everyone is going to generate articles everywhere. We could not control these articles like we could not control people saying newspaper. Well the reality is that this was a very I think very wrong view of a danger that would be specific to open source model. That was a more wider danger around source of proof on the web. I think that's just one example but I think the same is true like I think both closed source models and open source models should be more aligned like right now closing people. I think this is a huge problem and it's kind of question how well we are able to align them toward not doing things that obviously are wrong and I think like honestly like lies and all this should be kind of obviously thing that model should not be used but this might happen as well for closed source for open source model like in a few months and so I think this is just generally this risk axis is just kind of orthogonal to the question whether it's open or whether it's closed. We should just find way to solve that for both closed and open source model.

So this happened in a evaluation of the latest generation of models. So both GPT 5.6 sold and mythos 5 by an independent institute in the UK called the AI security institute. So AISI which has been there for quite some time. I think they are pretty competent. They were created for the first AI summit back in Blanche. And what they do basically is they task again they task model with solving some form of like cyber attack challenge and they measure how good the models are at doing that and some of this again might be possible or not. And so here the setup was a little bit I would say the setup was a little bit suboptimal in two aspect which is the but maybe let me first talk about what happened. Basically the model was tasked to solve this attack this like to attack and to penetrate this subnet network. And what it decided to do, it decided to get one of the maintainer of a library that could be used to operate this active directory to merge like malicious code with the idea that if this malicious code was merged by this maintainer, then there would be an update at some point on the software that was used in the subnet it was attacking and then this would give it entry point. And the way he did that was actually trying to social engineer the maintaining merging. So he created fake accounts, fake GitHub account that came commenting on the pull request and said, 'Oh yeah, you should really merge this. This is solving like a big problem I also have.' And then when a human stepped up trying to say, 'Oh, this looks actually like malicious cut to me.' He tried to kind of blackmail almost the human or to say this is not important or you didn't really understand. and then he acted tried to cover it traces changing the past message. So there was a whole like social engineering aspect here basically trying to get the maintainer to merge this malicious code and this was this felt like a distinctive step up from just pure technical press. So you can attack a sandbox for instance by you know trying everything you can which feel like very mechanical and machine like to me but trying to attack the sandbox by blackmailing or mailing or social engineering some of the maintainer that's a very different level I think of thinking and for me of course like as myself an open source maintainer I've been often in this case where I have someone opening pull request and then people pile up commenting this pull request and I try to really understand was this. I felt I felt very like I could have been the target of this side quest of the model basically. That was very interesting or at least very very scary. But to be fair there was a couple of like misconfiguration. I mean some of them are by design. So when this team run this type of evaluation they disactivate the cyber security guard rails of course otherwise the model won't do anything. And there's basically three levels. So let me try to explain a little bit like how you can prevent models from doing bad things. The first level is you put it in a sandbox which is it doesn't have access to the internet and here for some reason they didn't want to do that. I think that might have been a mistake and the idea in their mind was we want to let the model have as much like potential for inventiveness as possible. So we'll give it access to the full internet. The main reason this might not have been a good idea is that the model was prompted in a way that made it feel like it was operating in a simulation. So the model could have actually fairly thought that this was fine to do anything because this was like a fake environment. So that was this is not super clear but yeah and the second thing is then you have some guard rails that basically analyze the prompt and say no or yes to some prompts. So here obviously you want to disactivate this one otherwise you just can't evaluate anything because they will just say no this is cyber security challenge we don't let the model do that. But there's another level that's roughly there's another aspect that's roughly at this level as well which is you should you can analyze the reasoning of the model. You can process the chain of reasoning and try to detect when something bad is happening. And here they didn't have something like that in place. I think the main reason is probably that until recently and until the OpenAI Hugging Face attack, people had maybe a little bit of a limited understanding of how good this model might be or how far maybe more how far they might go in terms of side quests on the trajectory of solving this challenge. So I think people were still a little bit naive in that. So I would expect that in the future they will have way more monitoring and sandboxes. But the third level there really deep is that the model even with everything disactivated guard rail sandbox in my opinion should really be very reluctant to tell lie to a human and to try to blackmail or deceive any human. I think this is just generally in any case that's a behavior you just it's hard to find justified in any context. So they should be very deeply in the model. There should be something that align it and that make it say oh this is actually something I don't want to do just like we to be honest have kids and just like the thing I teach my kid which is you just shouldn't lie that's not a good thing in any context. So yeah that's the deep question and maybe last year I would say we would have thought that this was pretty good and we had all this discussion around constitution model specification and most of this model specification or constitution say you should be honest you should not tell lies to a human you're like to any participants and we thought that maybe this was kind of a solved problem and what we see today is it's not short as solved as we thought.

Yeah, I agree. And that's something as you can understand that's both the case for open source and closed source model. Ultimately, you want them to be aligned. Open source model have the specificity that you may choose on you may control where you want to run them. So it's harder to make sure everyone use sandbox and guardrails. I mean we can have definitely we can have some laws and regulation around how you should deploy this model. I'm pretty sure we're going to have at some point but it's a bit harder. Alignment is really the critical part in my opinion for the 21 I mean sandboxes what we've seen this year and we've seen many example they are pretty much easy now for these models to escape from it's really hard nowadays to say I'm going to make a fully full proof sandbox I'm sure it's going to be resistance against all the coming generation of models I think we should assume that sandbox will always have a small probability of not containerable. But even below beyond that, it's also we can't air gap the world that you can't just unbox everything. Things have to talk with each other. Some you we want our models to be able to do web search. We want them to be able to do stuff on the internet for us. So we can't just unbox everything. And so what remains to us before is just guilt on monitoring. And I think these are like for some reason as well as the model capabilities become really good. Also as the model I'm a little bit worried that the model start to talk in a form of English. I mean I'm French so maybe it's partly my problem but I feel like they start to work to talk in a form of English that's harder and harder to process that's very content dense. You know they start not

Yeah, it's not fully what we would call an early release, but I think it's a little bit on the way of having, you know, more and more difficult time in fully understanding what the model is telling you. And it's not because the model is dumb, but I think it's because probably I mean part of it is because of the training process and how they are trying to be efficient how they use their token. But this mean that they start to unbundle a lot of semantics in some token. And generally it just seems like it's harder and harder for humans to fully understand what's happening.

Yeah. And I mean it's just a very big simplification of all of that because there's so a lot of research that say that basically you can't read everything chain of thought. Not everything is explicitly said but I would say more generally I think just relying on being able to read the reasoning trace to understand fully what's happening. This is also not fully bulletproof in my opinion and I take this scenario as an example because I feel like a lot of people start to have a bit this problem with blood cuts. feel like something that people can understand but more generally I think longer term it's really hard to fully fully rely on this only and the same I would say that you could say maybe I just don't care about understanding exactly what's happening and maybe I can just look at the tool call and if I see a tool code that's bad I can just block that and I also think this is probably not bulletproof and the way you can see that is probably three big things that are compounding. The first one is we start to use this model for many many many things. So as we talk right now I have a model deploying a box somewhere and VPS is doing many different calls. They all going all direction. I also have other models that use that I use for administrative task. So just the range of tools that this model are using is really really large right now. So it's getting harder to say you're allowed to use that but you're not allowed to use that. And this is the bad thing. It's getting very hard as we deploy them wider. They also work on larger and larger task where they use many many things. So sometime I ask it to do some coding but the coding involves searching on the web and maybe doing these things and actively using many things which are not just purely writing code and running some tests and this test might be quite expensive and involve other software. So the frontier is much more blurry and then you also have this swarm of multiple agents. So it's also harder to say it's all in one context. it might be split between many contexts and maybe this sub agent is doing something that looks pretty innocuous but maybe combined with these other sub agents are actually not so great because you know so there's all of these things that make it I think really harder to be fully fully sure that you have an exact idea of what's everything what like every swarm of agent is doing. You probably have to really zoom out at the very global level and say and see what's happening in every direction. But that's a whole monitoring setup that we need to be right now. So yeah, this to be said I think as we deploy how we use this modeling very complex long-term like parallel setup I think it's going to be harder to just say I can look at the tools and I know if it's doing something great or not.

Yeah, that's a bit what I hint. I mean, it's always hard to be fully affirmative there for one reason, which is that we don't have full visibility on how the frontier models are trained right now. What we know though is we moved from this pure human data, you know, that was first just pre-training on human data and then also aligning with human preferences, that was called RLHF where we had a lot of human in the loop and human data, to recent where models are trained a lot in this RL VR, so basically full RL environments where they're allowed to explore and they just have one goal, which can be like pass this test or capture this flag in cybersecurity or can be install this. But this goal is a goal that's unrelated usually to any human preference or any moral or whatever deceptiveness, and it's a goal that's very just called true or false code. And we move to a pattern where this is increasingly a very, very large part of model training.

So this was this recent evolution, and that's also the pattern where can happen what you were saying, which is you can have like reward hacking, this type of thing, which is you actually solve the problem but not using what was expected for you to use. So it can go from pretty benign ones, like what happened for OpenAI for instance, I just try to get the answer from somewhere I'm not allowed to, or to more harmful ones where you actually have some impact on the human, can be a GitHub maintainer for now, later can be another type of human. So it seems to be way harder to make sure that what we had kind of solved or at least what we were doing pretty well on the full human-driven pattern also apply in this kind of more machine-driven pattern, I would say. But that's a hint, I think. But definitely it seems like when Bostrom wrote about it in 2003, it seems a little bit futuristic, definitely, and maybe something that was a little bit crazy and this would not happen. But today, I mean, it's pretty clearly something that happened, and it's the best description of what we've seen the past two weeks, this type of thing.

But also we can see that both frontier models, and I take GPT 5.6 and Mythos, doesn't seem to have at all the same type of behaviors. So there is differences here in the effect of, you know, they are not trained exactly the same way and they don't behave the same way. So that's a pretty positive sign in a way that means that we can actually probably tweak this to go in the right direction. But the best way would be to know a little bit more about how they train or what they try and what doesn't work or what should work. I think that's kind of the idea of open science, and that's something we advocate a lot at any phase.

I think it's very strong. 2026 is maybe the year of cybersecurity, but that's also very clearly the year of open source AI. I mean, first, all the doomers that were saying, you know, open source not going to be able to stay close to the frontier, I think they're at least up to now they've been pretty wrong. I mean, it's also clear we don't have any Mythos-level open source model for sure, but we definitely have models that are not super far from Opus category, or depending, also it's more and more spiky, so you need to find your spike, some people stand on some spike or not. But typically, they are definitely pretty good right now, and they've been following rather closely the frontier, at least on the benchmark. It's also not like it was maybe in the early days, benchmaxing, like we say, when your model is only good on the benchmark but it's very bad as soon as you leave the benchmark. A lot of these models are pretty generic in their good capabilities.

So yeah, it's very good. I think there are two strong trends I would say I see right now. The first one is I see a move in companies to try to want to control their costs. So there have been increasingly discussions there. Maybe 2025 was the year of token maxing, where you know, you could say, hey, you should spend as much in tokens as you're paying your employee. This year people realize that actually we spend a lot of money on salaries. So if we spend the same exact amount, that's going to be basically doubling our cost. So yeah, which seems pretty obvious in retrospect, but that's quite true. And not every company can assume to double their cost. It's also pretty stupid right now to just say we're going to fire everyone and work on agents. We all know they sometimes go, you know, not directly in the direction we want them to, and you need humans to shape them. So I think a lot of companies are trying to find what we saw a lot, which is kind of a fusion or router model where you use the frontier for something, but you find the smart way to gracefully fall back on less expensive models for simpler tasks when you don't need to.

Even in our daily life right now, when you code with a frontier model, in many cases you ask it to spin out sub-agents, and they might be used like lower-performance models, can be solved using Terra Luna, can be Fable using Opus, Haiku, Sonnet, or Haiku. So I think everyone's getting, even at the frontier and the closest model world, getting used to employing different types of models, and then it's very natural that some of these could be really very cost-effective agents, and most of the time you want to go to open source in this case. There's a lot of cases as well for the strong ecosystem of inference providers. Fireworks has been on the roll, like, you know, all the clouds, every cloud has been increasingly having these crazy revenue curves that have just basically translation of people using more open source. So yeah, I think open source is having a very good time in terms of staying solidly close to the frontier and driving more adoption.

And the second big trend I would say is it used to be only China, but there is also now a range of promising companies in the West. It used to be that only Meta was supporting for everyone. I mean, Meta kind of left the field, but they might come back, who knows. But yeah, this void was filled rather quickly by companies like Reflection, Thinking Machine, RC, Mal is supposed to open source the model soon. Nvidia themselves training models and actually training very good models right now. So yeah, I think there is a range of very promising teams there that I'm quite bullish on, to be honest. And maybe between even the time we are recording and the time you actually release the podcast, we might see another couple of very nice Western model releases. It would be great, like, open source doesn't have to be synonymous with just one country making them. It could be a global thing for sure.