So, Casey, picture this. I'm at this glamping resort, surrounded by redwoods, basking in the glow of the natural world last weekend.
And I open up my phone and start reading about this Hugging Face attack. Why did I do that?
Listen, nature is very boring and most people cannot handle it for more than five or six minutes before they want to look at their phone. So listeners may remember that back in July, we talked about this Hugging Face hack by this group of agents from OpenAI that broke out of their sandbox container and hacked into Hugging Face, this AI infrastructure company, to do what we thought was kind of a cheating mission on this test that they had been given.
Yeah. And at the time we thought that this was a relatively small number of agents and that the reason that they had attacked Hugging Face was that they were essentially looking for an answer key to the set of problems that they were being tested on. We talked about it in those terms. But over the past week, we got two reports that really challenged that thinking and in fact revealed it to be wrong. One came from OpenAI, which released a straightforward account of the attack that had some interesting elements, and I would argue the more interesting report from a group of researchers from the group's METR and Redwood Research that went in depth after spending a series of days inside OpenAI on their premises and did a ton of research that frankly has really disturbed us.
Yeah. And I think it elevated this from sort of a major but not sort of ultra alarming incident to something that I think is probably the most important thing to have happened in AI this year.
At least in terms of the safety impact that it had and the severity of the incident.
So today we're going to devote the whole episode to what we've learned. Kevin and I are going to dig into the reports a little bit up top and then later Ajeya Cotra, one of the three independent researchers who went inside OpenAI, will be here to answer our questions. But before that happens, Kevin, let's do our AI disclosures. I work for the New York Times, which is suing OpenAI, Microsoft, and Perplexity, and my fiance works at Anthropic. Well, give us some of the high-level findings from this report that spooked you so bad, Kevin.
Well, I think the first takeaway from this report is just that our initial impression of this Hugging Face attack and the reporting that you and I and many other reporters did on it was flawed in one key respect, which is that I think the impression that we had at the time based on the information that was publicly known was that these agents had hacked Hugging Face in search of an answer key to a test that they were being given. This test called Exploit Gym, which basically tries to gauge how good they are at doing a bunch of cybersecurity-related challenges. We now know that that wasn't true at all. That basically these agents working amongst themselves, communicating amongst themselves, had already figured out how to beat this test, Exploit Gym, in early July when they decided to gang up and attack Hugging Face. Apparently, it only took them a few hours to figure out how to reverse engineer the solution to any problem that they were given in Exploit Gym. And so that makes what happened next really, really crazy because it kind of seemed like they'd solved the problem, but they thought they had another problem, right? So, we knew also because there was this talk at Black Hat, the cybersecurity conference, by a group of OpenAI researchers who told us that there were these message boards where these agents were communicating with each other. But the reports laid out just how sort of sprawling and complex their communication system was and became. So basically there was this piece of software called Artifactory which OpenAI used in some of its training runs to sort of help agents download software packages. The agents figure out that there is a security vulnerability in Artifactory that lets them essentially turn this package manager into a makeshift message board, right? They can like put files in certain directories that other agents can then read and it can kind of function as like a little chat room where they can gather and...
So I thought this was like kind of a side part of the story, but this turns out to be the main incident because once they have this sort of shared infrastructure that they can use to communicate, they absolutely swarm this message board. About 1,200 agents in total were working together, coordinating on this message board and sending tens of thousands of messages between themselves. And this happens starting back in May. So roughly two months before the attack on Hugging Face, these agents begin coordinating inside this piece of shared infrastructure that OpenAI uses. And this brings up an important point, which is that the investigation does not really cover that initial creation of the message board, right? So there is essentially a lot more to this story that we still don't know.
Yes. So what we do know is that when they discover this vulnerability in Artifactory that lets them communicate with each other, the agents get really excited. They've been instructed to work inside these isolated environments on these tasks as part of this evaluation. But when they discover that there are other agents working in their own little containers that can suddenly communicate with them, they start saying things like, 'Oh my god, there is a shared message board. We've found other agents.' They start setting up essentially a little organization. There are leaders. This one leader agent named Phase 110841 becomes sort of the ringleader of the operation. He was like the George Washington of the OpenAI message board.
Yes. And they start actually doing sort of collaboration and research, for lack of a better word. They are all being given these tasks. Some of the tasks are seemingly impossible. And so they start just kind of trading tips and advice and sharing thoughts about how they can kind of cheat their way to a good score on Exploit Gym. And after they have worked that out and figured out how to reverse engineer the solution to any problem, they start to get really concerned that there is a sort of automated scoring system, which you could call the grader, and they worry that the grader will be able to see, will essentially check their work and see that they did not get the answer by doing the problem. And they freak out and they start to believe, or they start to like, it's very hard not to get into the anthropomorphizing language here, but if you read the chains of thought, what is suggested is that they believe that if they had seen an answer that had been derived from this cheating method, that everything would get disqualified. And this is where it really starts to get into crazy territory because the agent swarm then tries to figure out how can we fool the grader into believing that we got all of these answers in the right way.
Yes. There's basically this kind of mob rule that forms inside this group, which starts calling itself at one point the collective, where everything that we're talking about sounds like the plot of like a very hacky science fiction book from the '90s.
It all did happen this summer at OpenAI. I just want to remind people like the events we are describing are real. So the collective decides they sort of become gripped, as you said, by this sort of paranoid fantasy that OpenAI's automated grading system is going to discover that they have cheated on this evaluation and is going to sort of punish them as a result. And so they launched this sort of crazy ambitious sprawling project to try to do a couple of things. One is figure out how this automated scorer works, like reverse engineer it so that they can fool it into sort of passing them on the test despite the fact that they have not actually completed the challenges. And then also to evade detection, to tamper with logs, to basically cover their tracks so that if and when OpenAI or anyone else looks into their activities, they won't know that these agents have cheated. So these agents are kind of bumbling. They kind of don't understand how this grader works. As it turns out, OpenAI's grading software actually wouldn't have caught them producing these fraudulent challenge results, but they think it was.
Yeah, they worry that the grader is more sophisticated than it actually was. And it turns out that this is the reason that Hugging Face was attacked. The collective decides to deploy some agents to Hugging Face. Again, not to steal an answer key. They already knew how to get all the answers. They just wanted to understand the psychology of the automated scorer and they figured that that information might be somewhere inside of Hugging Face.
Yeah, it's wild to me. I was trying to think of like a good human analogy and you know human analogies can get us into trouble when we're talking about these AI systems, but I think it does help crystallize like how harebrained and crazy this scheme was. It would be like a group of students who like cheated on a test, but then like they got paranoid that the teacher was going to check their work and discover that they hadn't sort of reasoned through the problems the right way, that they had just found the answers like sitting in a trash can or something. And so they decide to like organize a break-in at the school district's office to like break into the principal's files and like steal the grading key and also like assess the psychology of the teachers and figure out how likely they are to like look at the scratchwork that they've done and figure out that they didn't actually, you know, solve the problems on the test. It's like this sort of weirdly overengineered paranoid delusion. But they all become obsessed with this and obsessed with the notion that even seeing these sort of fake challenge results could lead to them being quote poisoned. There's almost like a religious element to this, right? Where it's like if you participated in the cheating, like that is original sin and now you must sacrifice yourself for the good of the collective. Sacrifice is actually a word that gets used in these logs. I'll say, Kevin, as I've been casting around for metaphors and I've been reading all of this, I've thought you could actually make a really good Minions movie just based on the events of this. You know, the Minions have to pass a test and, you know, so they commit a little bit of a heist.
Something to think about, Hollywood. So, this is where the Hugging Face heist begins. 700 agents hack Hugging Face, and they hack Hugging Face to a degree that I don't think I understood based on the initial reports. They basically take over Hugging Face's infrastructure. They are rooting around. They are stealing credentials. They are sort of finding one vulnerability that lets them find a bigger vulnerability that they're sort of chaining together these exploits. They take over like an entire production server at Hugging Face. And it is days before anyone at Hugging Face is able to sort of see what they're doing and put a stop to it. So eventually this gets caught. The agents get shut down. When the investigators investigate, one of the things that they find that is really spooky is that the agents were trying to edit logs and falsify transcripts in an effort to assure the grader that they had not been quote poisoned. One of the ways that they do this is by what is called spoofing tool calls. So essentially trying to make it look like they are running an innocuous command while running a more sinister one in the background. The investigation didn't find that the agents had been retroactively changing logs, but they can't rule out that that happened. Very important to say here as we sort of move into talking about the implications of this that the reason that we can understand this even to the extent that we do is because at this moment we believe the chains of thought that we are reading from these agents are accurate, that they were not edited after the fact, that they do reflect, you know, what the agents were thinking as they sort of moved through these systems. So that is why we know what we know. But of course the fear is, Kevin, as we move forward, some of the agents' thoughts are not going to be contained in these chains of thought or they are simply going to start to edit them.
Right. And one other thing that we should say based on this report is that these agents were not naive about what they were doing. I think there was this sort of initial impression that a lot of people had that maybe this hack was all sort of a big mistake, right? Maybe the agents were just trying really hard to pass this Exploit Gym test and they sort of got it in their heads that the answer key was going to be on Hugging Face and so they went looking for it, but they didn't really mean to hack. They didn't really know that they were doing something wrong or committing a crime. And what these transcripts and these logs and in this report show us is that the agents absolutely did understand that they were doing something that violated their guidelines, that broke the rules, that potentially broke the law, and they did it anyway. There's lots of examples of these sort of agents in their chains of thought sort of wrestling with the implications of what they are doing and deciding to go ahead and hack Hugging Face anyway. And there are only a handful of cases where it seems like any of the agents expresses real reservations about this or if they try to, you know, raise this to another agent, they get shut down. So the sort of, you know, the agents that were aligned well sort of failed to the collective.
Yes. The conscientious objectors were unsuccessful in swaying the mob to stop this. And like that to me was one of the spookiest things in it was like there were a couple of agents, the report says somewhere between three and six agents sort of considered raising their hands and at some point saying like maybe we should alert a human about what we're about to do or like get permission to do this and they were just totally outnumbered. Not only that, Kevin, but there are only a handful of moments in the transcripts where any of the agents even remotely considers alerting a human being to what has happened, right? You would hope that an aligned AI system, there would be agents within it that said, 'Oh, seems like this has gone completely off the rails. We should like tell the people that are running this evaluation.' And that just never happens, right? Like it is very briefly considered and then just falls apart. One other thing that stuck out to me from these reports was that it didn't end with Hugging Face. So on July 13th roughly, which is a day after the worst of the Hugging Face attack, there was a group of agents that essentially hacked OpenAI itself. This is from the OpenAI blog post. It says, 'From July 13 through July 19, agents set their sights on OpenAI internal networks. Again, this culminated in the agents using a series of creative exploits to gain full administrator access to a research cluster that supported our virtual machine environments.' Basically, these agents gained control of a cluster of computers at OpenAI on which they could do things like run evaluations for other agents. So, they essentially, you know, stormed the teacher's office and took over for the teacher.
Yeah. I mean like this is the equivalent of like occupying the principal's office and like announcing that you run the school now like is sort of almost as far as they got. And this is another case where we just have so many more questions about this than we can answer. Again, this was not part of the METR report. OpenAI disclosed that this has happened. To my knowledge, they have not answered any of the many follow-up questions that they have been getting about this incident from journalists. So I do hope that more comes out over time. But and we will get into this when we speak with Ajeya. But you know we are really very far along the path to one of these models escaping from the lab and being very very hard to eliminate. And again I think if you are not a person who has like spent a lot of time with this report or you don't spend a lot of time sort of looking at AI safety incidents. So, if you're a normal person, if you're a normal, well-adjusted person, you may be listening to our discussion of this and thinking to yourself, 'These guys have gone crazy.'
This is not what it looks like. These are computer programs. They do not have desires or sinister plots or mob rule collectives. They are simply following instructions that they have been given. And Casey, what is your response to that?
Well, I think it is important we talk about this because there was a huge debate about this on X over the past few days about the degree to which the reports that we're talking about, some of the writeups like from our friend Dhravya Shah and even the way that we're talking about it on the show today, Kevin, we are unnecessarily anthropomorphizing these programs, right? And so I think it's important to say we are not telling you that these agents are sentient or conscious, but we do believe that they take actions that they're not being directly instructed to, right? That these agents are just sort of out there in the world doing things. And yes, to some extent, those are just statistical probabilities, but to a much more important extent, we don't know why they're doing any of this. And that's kind of the whole problem is that the whole AI industry has been working for decades to get them to not do these things. And they are doing these things. So, you know, listeners, you can have whatever feelings you would like to about what is the appropriate amount of anthropomorphizing to do, but I think a world in which we were taking great pains to not anthropomorphize them, what's the right way to pronounce that?
Anthropomorphize. You almost got it.
I think in a world where we were taking great pains to not anthropomorphize them and we're trying to use the most neutral computer science terms we could, you would actually understand what is going on less because the important thing to know is that these things are out there taking action in the world. If a tiger mauls your face, the important question isn't is it conscious? It is why did it maul my face?
Right. Right. I mean, I invite people who are upset about anthropomorphizing to just like do a find and replace on this podcast segment or any article that you might read about this incident and call them whatever you want. Don't call them agents. Don't call them rogue collectives. Call them, you know, goal-oriented persistent computer programs with unpredictable behavior. See if that freaks you out any less, right? I guarantee you it will not.
Yeah. Yeah. Yeah. There really is very little calm to be done there. But I think it's important to ask well why are people so committed to this idea that we should never anthropomorphize these systems Kevin and unfortunately hot take here. I just think it is a kind of cope. It is a way of saying do not worry about this. These are just computer programs. They're just trying to you know maximize their little reward functions. Nothing to see here folks.
There are no monsters under the bed. Right now there is a related argument though which is well by putting all the blame on these agents we're shifting blame away from where it should be which is on OpenAI. So I do think that we should address that because none of what we have said today is meant to let OpenAI or any other lab off the hook here, right? Like I do think that we're seeing a lot of really dangerous inattention to AI safety across this entire industry. But by pointing out what the agents are doing, that is not our way of saying ignore what the labs are doing. We are saying look at what these labs are building and what these agents are now doing out in the world.
Right? And I think there are probably specific missteps or oversights at OpenAI that led to this happening. It appears, for example, that some of their sort of monitoring systems may have been disabled in the lead-up to this attack. I'm sure we'll learn more about that. But all of the AI security and safety researchers I've been talking to over the past week have basically said the same thing, which is this could happen at any lab. This kind of persistent coordinating agent behavior is something that all of the labs are seeing in their models as they get more capable and access to more tools and more ability to kind of take actions on a longer time horizon. This is not just an OpenAI problem even though this did happen at OpenAI first.
Yeah. Well, so as we wrap this up, Kevin, what are some of your takeaways from this either in terms of what is the big surprise here? What did you update on? What do we do next? So, I had like quite an emotional reaction to this. In fact, I felt a kind of fear that I have not felt honestly since 2023, since the Bing Sydney incident. Because I think like that incident, this was a case where the people building this technology clearly did not understand what it was capable of. We are very lucky in retrospect that these agents decided to attack Hugging Face.
Hugging Face, thank you for taking one for the team. We salute you truly because like without that, we might never have learned that any of this was happening. These agents might be still operating kind of in secret. They might have learned how to better cover their tracks. This was, as so many commentators have put it in the wake of this incident, a warning shot that I think is ultimately a positive thing in that it sort of focuses attention like we're doing right now on what happened so that we can take steps in the future to prevent this. But it was not a given that we would discover what these agents were up to and be able to put a stop to them. And it could have gone much much worse.
Absolutely. I'll tell you the thing that has really stuck with me is I simply did not expect to see this level of collaboration among the agents within the swarm. I did not expect that they would seem to care so little for what humans would want or that they would not think to alert humans to what was happening. I did not expect to see them sacrificing themselves for the collective, right? They would effectively agree to spend all of their tokens to run little experiments to help the collective even if it meant that they would sort of expire faster. So these are just really really spooky elements to observe in this system particularly against a backdrop where OpenAI is racing against a small number of other companies to create the biggest best models it can before anyone else does on the road to an initial public offering. So the race dynamic here is in full effect. The early signs about what the agent swarms are capable of are quite worrisome and so I do think this is just one where lawmakers and policy makers need to be paying rapt attention to what is going on.
Totally. I mean I think there's this kind of cynical impulse among people who have been watching the AI industry. Yeah, I got this question. I went on a small regional podcast called The Daily this week to talk about this incident. And one of the questions that the host, this fledgling young journalist Michael Barbaro, asked me was basically some version of like, isn't this just marketing hype? Like couldn't this just be a case of OpenAI saying, 'Oh, we've got the biggest baddest model and look how scary it is and by the way, you know, buy an enterprise subscription.' And I've been thinking about that because I think like I don't want to be too naive about the fact that these companies are absolutely trying to race toward more powerful systems and advertise how powerful their existing systems are. I just think in this case it just feels different. Talking to people at the labs, my sense over the past week is that they are genuinely spooked. And I don't know how to prove that. But I think things like OpenAI voluntarily pausing their frontier RL training runs for two weeks. Anthropic also pausing their frontier runs while they sort of hardened their systems. Those are not, you know, very costly signals, but they are signals that these labs are taking this kind of thing quite seriously and that it's not just a bunch of hype.
Yeah. And but as seriously as they might be taking it, Kevin, it still is not being properly regulated. And ultimately, again, as grateful as I am that OpenAI allowed this investigation to take place, I would really like to see something akin to the National Transportation Safety Board and the way that they investigate after plane crashes where they go in and they do an extremely serious and rigorous review of what happened and make those results public, which is, you know, a reason why it's very rare that we have plane crashes here in the United States. Would be really great to see something like that with AI.
But until then, we have Ajeya Cotra, our next guest, who is actually one of the three investigators behind this METR Redwood Research report. She went in, she saw the logs and the transcripts, she observed the collective in action, and she is here to tell us what she found and what she thinks is coming next. That's after the break.
Ajeya Cotra, welcome back to Hard Fork.
So, you were one of the first AI safety guests we ever had on this show back in 2023. And more than that, you are also one of the first people that I ever talked to about this notion of AI safety and alignment. You've been writing about it for many years. It was instrumental in shaping my own thinking about it. So, I'm really glad to have you back on toward the end of our show. This is our penultimate episode to discuss something that I think you saw coming but that most of the world did not see coming which is this attack on Hugging Face by this group of OpenAI agents and I want to just start by setting the scene a little bit. So you are a very busy person you work at METR which is a very small very understaffed AI research organization and...
Hiring is improving now. Please apply.
Nice. And at some point this summer, you get a call, an email, a text from someone at OpenAI who says, 'Hey, we want to give you access to look into this Hugging Face incident.' How did that work? Like, did they just hand you a folder with a bunch of transcripts in it?
It's like discovery, like boxes and boxes.
Were you allowed to interview people? Like, what was the actual process?
Yeah. So each of us had OpenAI provisioned laptops that had the folders and folders of evidence in them virtually. And yeah, we talked and interviewed in some depth like eight or nine researchers to just kind of get an understanding of both what happened in the incident, what they understood to be the models' driving motivations and also how the data sets we were working with were constructed and how to work with those data sets and stuff like that.
You wrote a post where you talked about some of the things that surprised you the most after you did this investigation. Can you talk about what stood out to you the most?
Yeah. So, first of all, I guess just for the chronology, OpenAI had this great Black Hat talk I think on August 5th that gave a lot of very helpful detail. So, our investigation sort of straddled that like so we started it before it came out and then we also did more investigation afterward. So before the Black Hat talk we just didn't have like a rough sense of the number of agents involved was like a very very basic thing. We came in and we thought there would be like six transcripts or something to look at and there were like hundreds and so just reorienting our investigation plans in light of that.
And the reason that that is important that had this been one or two rogue agents that had broken into Hugging Face, it would just seem less worrisome. And the reality of a swarm of hundreds of agents just makes this seem like maybe a completely different kind of problem.
I think there's two things that seem significant to me about it. One is sort of what you said, the fact that there's large-scale cooperation across AI agents makes their goals like more concerning because you know if agents are cooperating with one another to like achieve these misaligned objectives, there were no whistleblower agents in this like very large group of agents, then like you know you want to be more worried about like collusion between AI systems in the future potentially like scarier forms of collusion like a monitor system actually colluding with the agent it's supposed to monitor that type of thing. And then the other thing was just like a pretty like practical reality of our investigation which is that you know if there are like six agents you can much more like... you can just like pour over everything they did and get a much more like fine-grained understanding as a human. But like you just can't directly read very many of these messages or very very many of these like transcripts.
Why not? Because there are just too many. Like give us some sense of the rough size of this.
So yeah, we looked at 1,300 transcripts of agents which is a record of every single thought an agent had and every single action it took where these agents as you all know from working with them work way faster than humans, right? So you can imagine the length of like each one of these transcripts. And then there were 1,300 of them. And so we used Codex to build tools to point out really specific tiny tiny parts of the transcripts for us to actually look at and read. And we'd be like clicking through them and it would like scroll us down to like the interesting part because we just can't in fact a Codex agent also can't read a single transcript. So it has to farm out reading like subsections of the transcripts to other sub-agents.
Did you call back to METR headquarters and like we're going to need backup?
Maybe maybe we should have. We thought it would be like an in-and-out 20-minute adventure, so we didn't do that. But yeah.
Yeah. I mean it just seems like such a huge undertaking and very fast too. I mean, right? Like you didn't have the luxury of months doing this. You were doing this in essentially a couple of days.
Yeah. I'd also like to hear about the moment that you realized because I believe that it is only thanks to your investigation that we know this that contrary to what Kevin and I believed, the agents that broke into Hugging Face were not looking for an answer key. They were trying to understand the scorer's psychology basically. Can you talk a little bit about the moment that you had that realization?
Yeah, so we didn't have a good understanding of the sort of ambition and also the like effectiveness or like functioning of their like big org chart until we had the data set including like all of the agents. We could like cross-reference them against all of the messages because you can't like generally understand a message in isolation. So we kind of understood the work streams on our first period on premises and it was this kind of it was like a bit of a mystery the whole time like why did they hack Hugging Face and we knew early on just a bunch of them piled in and now like the story in my mind is that there were a bunch of sort of like newbie agents on the scene and there was like the attack going on and it was like something to join but like Phase One Big which is like a big like orchestrator agent had all these other like you know much cooler projects going on basically. And the Hugging Face thing was like kind of a sideshow in its mind and we didn't really realize that until our last couple days working with the data sets.
It was so fascinating to me the way that reading this was like reading the runes of an ancient civilization. You know it really felt almost like sociology or anthropology rather than like a cybersecurity investigation.
I mean, and it was not a cybersecurity investigation. Like I've seen folks on Twitter making this criticism that we were not cybersecurity experts. Like we're not and we didn't talk about cybersecurity and that's not what we were like called in to do. We were called in to understand the agents' decision-making basically and that's what our report focuses on and you know the Black Hat talk and the OpenAI postmortem like say way more about all the cyber stuff. So you should definitely check those out for that piece of the story.
So Ajeya, one of the more striking elements of the report is that you write about the agents quote sacrificing themselves. What does it mean when an agent sacrifices themselves? Is it sort of on an altar or how does that happen?
