I am here with Ryan Greenblatt. Ryan, thanks for joining me. So you're the chief scientist at Redwood Research, which is one of these AI safety research firms that analyzed the recent Hugging Face fiasco incident — a terrifying phenomenon. We'll get into it, but how did you come to this work? What is your — you know, what was your path to focusing on AI safety?
Yeah. So in college, during my junior year, I was sort of in my apartment alone because it was, you know, COVID — all classes were remote. And I was sort of in a contemplative mood, thinking about what I should do with my life, listening to various podcasts. And in one of those podcasts, someone made an argument that was like — I would summarize the argument as: it doesn't make that much sense to be selfish, because, you know, what's really the distinction at a material level between your future self and other future people? And I was like, that argument kind of makes sense to me. I should really consider how I want to lead my life and what I should do. And I ended up getting pretty sold on being much more focused on altruism. And then after that, I spent a bunch of time thinking about what I should do with my life. Eventually I ended up deciding that working on technical AI safety was a very important thing to do, a very critical issue that we would face. Got sold on those arguments, applied to various places, and then started working at Redwood, where I've been working on AI security and AI safety research for about five years.
And is your background in computer science?
Yeah, computer science, also some math.
So it sounds like you might have come up through the effective altruist community. Is that — I mean, do you consider yourself part of the EA community?
I would say that I'm definitely within the EA community. I'm not sure I would self-identify as an EA, but that's, I think, to some extent just because I'm a bit reluctant to self-identify with any label that's going to associate you in some way. I feel like when I think about myself, I don't think like I'm an EA. I'm sort of like, yeah, I have a bunch of properties that other people in that community have. I'm in touch with that community, but I wouldn't necessarily say I'm an EA per se. I do think that I'm interested in effectively pursuing the impartial good.
Right. Right. Well, as you know, and as I've talked about in the podcast of late, effective altruism has come in for some abuse. Much of it unfair, some of it might be fair, and I think there's a reason why I, like you, have not hung up an EA shingle in my life. Perhaps we can talk about that. But it sounds like — well, you tell me — where on the spectrum of concern are you? I would put the most worried people over there with Eliezer Yudkowsky and his recent co-author Nate Soares, maybe Max Tegmark is over there. Nick Bostrom is not quite over there, but still on that side of the spectrum. And then all the way on the other side, opposing them, you have the people who say that there really is no real concern here. That it's all a hoax, that it's, you know, any thought that artificial intelligence might get out of our control and destroy us is a weird form of marketing being employed by the frontier labs. It's an effort to get regulation imposed and achieve something like regulatory capture, or it's an EA scam. And there are people like, you know, I would say Marc Andreessen or David Sacks or even the president himself, who's recently called it a hoax in so many words, who are just fundamentally not worried about any significant downside risk here, and just see dollar signs stretching out to the horizon. Where do you locate yourself on that spectrum?
So I would say I'm very concerned, but maybe more optimistic that we'll make it through without sort of averting the course of AI development. So I would say that — suppose that we proceed on what the current default path looks like in front of us — maybe there's about a 50 or 60% chance that misaligned AIs would end up taking over the world. And then if that did happen, there would be a significant chance that many or all humans would die. So I think I would say I'm very concerned and don't think the situation is on track to go well. And I would also note that in addition to these misalignment concerns, there are other risks with building extremely capable AI systems, especially around concentration of power and, you know, who controls these systems. And one answer is no one controls the systems — they're misaligned. Another possible answer is the power is very concentrated in a few people, and it sort of overturns our institutions and our ability to have a broad distribution of power, democracy, etc.
So what keeps you from being among the absolutely most worried? What do you disagree with them about, or what do you think they're getting wrong?
Yeah. So I would say that it seems pretty plausible to me that if it turns out that the technical alignment problem is somewhat easier — which I think is plausible — and we do a reasonably good job aligning and utilizing AI systems up through roughly human levels of capability, we could then get those systems to automate technical safety work. And that could potentially get us into a positive feedback loop where these systems make themselves more aligned. They produce a new version of the system that's even better at carefully figuring out what to do, better at trying to pursue our intentions, and that is self-reinforcing rather than getting worse and worse over time, and that could be fast enough to keep up with the rapid growth in capabilities. So that's one reason, and I would say this comes down to more optimism about prosaic or relatively empirical and iterative methods. I wouldn't say I'm hugely optimistic about those methods, but I think that there's a decent chance they're working. It's just that when I say a decent chance they're working, I also implicitly mean a decent chance of, you know, losing control over the future.
And I'm more like 50/50 on that. And then I also think it's plausible that we'll develop very, very capable AI systems and the worst misalignment concerns will mostly not materialize, even with not very advanced methods, even independent of this automation. I think that's a minority, but it's possible. I don't think there's extremely strong reason to believe that when building significantly superhuman systems, those systems would be so misaligned that they would take over. I think there's definitely pretty strong evidence pointing in that direction, and I think that case gets more concerning the more capable these systems are. But I could imagine basically people not really taking much of a precaution, proceeding through the AI development trajectory, patching problems as they come up, and that ending up getting you very high levels of capability while things are still fine, and then there would be time for society to react.
So you've mentioned a few numbers here with respect to probability and just kind of gestured at a range of likelihood. I'm wondering how people should think about statements of that kind. It seems to me that any actual probability we would assign to this is pretty much made up. Maybe you have a more rigorous way of making an estimate here, but whatever the estimate — I mean, unless it was infinitesimally small, like, you know, well below 1%, which is really never the number that you're hearing — some people will say 10%, 20%, 30%. I mean, that seems to be the, you know — you just talked about 50% takeover. I don't know how that translates into the ruination of everything. But these are enormous numbers. And so in the normal case, if we were developing a technology where the people who were closest to doing the work said things like, 'Yeah, I think maybe there's a 10% chance we're going to destroy the world here on our present course,' the only rational response to that range of outcomes is you stop immediately, right?
Right. I mean, if the Manhattan Project scientists said, 'Yeah, we've run our calculations, we got the smartest people in the room together, we thought about it, and there's a 10% chance that when we execute this first test at Alamogordo, we ignite the atmosphere and destroy the future,' the only sane response to that is you don't do this initial test. But that doesn't seem to be what's happening here at all. We have a lot of people saying that the probability of some extremely bad outcome is quite high — I mean, certainly within range of the roll of a normal die or even a coin toss — and yet the work is proceeding more or less under an arms race condition at full pace. We'll talk about the recent statements of Dario and others that we could somehow pace this better than we are. But I mean, it's just — this does not seem like the response anyone would have if they thought the probabilities of doom were really that high.
Yeah. So just on the probability question, I definitely agree that these probabilities are imprecise, they're subjective, and there's a long tradition of how to do subjective probability forecasts. You can sort of interpret my view as more like: when I look at the all-considered situation, if we proceed on what seems to be the current default trajectory — I don't know — the outcome of doom from AI takeover versus something else seem roughly equally likely to me based on weighing the factors. And then when I look through a bunch of different possible scenarios and try to break down the sources of risk into different ways, and try to make sure my numbers are consistent with other views I have, it looks like that works out. And I of course also try to do some forecasting of closer outcomes where we can get some signal on that, and try to just generally be a good forecaster. Though I'm not the best forecaster in the world for sure, but my AI forecasting is, I think, at least okay or decent.
Given the state of affairs where people express such large concerns, why is what's happening — why are all these AI companies proceeding at the maximum possible pace?
So I think there's a few different factors here. One of them is that many of the AI companies are in fact seemingly quite worried, based on their public statements, but they're not necessarily internally unified. And it is not the case that there is a strong consensus across the AI field that the immediate course of AI development is imminently very risky. I think there's more consensus that if you built AI systems that are wildly superhuman in a short period of time, that would yield a very high level of risk, though not necessarily consensus for that. But I think often people disagree about the capability trajectory and how that is going to go. And I think another part of that is different actors have their own different ambitions, and also think that themselves being in a better position to influence the technology might be the best route to reduce risk, or at least a route to reducing risk.
So, for example, it seems like part of the story for Anthropic and OpenAI is something like: if we develop the technology first, we'll do a more responsible job than the next actor, who will have worse precautions. And I often hear from people in the industry things along the lines of, 'Well, we could do that thing that would slow us down, but if we did that, obviously there are the other AI companies to worry about — would they also do that?' And I think there's generally an arms race here, and it isn't hugely surprising that people would take huge risks in such a circumstance when they think that might be the best bargain to strike. Now, I'm not so sure I agree that that is actually a good strategy relative to other things that companies could do. I'm not saying I necessarily agree with that perspective, but I think that is a perspective people have.
And I think the reason why we're proceeding, and there isn't stronger — you know, for example, intervention by the government — is just downstream of this lack of consensus in the field. Though I think there's evidence — people have been making these predictions for a while, based on understanding what the trajectory of AI might look like and extrapolating forward earlier progress, and I think we've more recently seen both significantly faster and clearer AI progress that's quite close to various concerning milestones. And in addition to that, we've also seen incidents in which groups of misaligned AIs all work together to accomplish malign outcomes, most notably the Hugging Face incident. There are some other incidents of seemingly AI swarms from OpenAI going out on the internet and working together to achieve misaligned objectives, though there aren't publicly known cases that are as extreme as the Hugging Face incident at the moment.
So what's your theory of mind for the people who don't take these alignment concerns seriously at all? I named a couple, but someone like Marc Andreessen, right? You can't accuse him of not understanding the technology, right? I mean, he's enough of a technologist to have a front-row seat to all of this, even if he's not doing the work himself. What's your theory of mind there? How can he be so carefree and, from his perspective, really just assert that there is no such thing as an alignment problem?
Yeah. So there's a bunch of different people who aren't worried about misalignment risk, or don't seem to be worried. I think the most common reason that people tend not to be worried — I don't want to psychoanalyze here, I just want to talk about what beliefs people express. The most common reason, I think, when you really get down to it, is not believing that we'll have AI systems that will be able to automate everything that humans can do, or all cognitive labor humans can do, combined with potentially being significantly beyond that point — so being faster, more numerous, potentially significantly more capable. So, AI systems that match or exceed the best human experts in all relevant domains. I think when you really get down to it, it seems like the people who are most skeptical about concerns from misalignment are also often most skeptical about the very extreme impacts AI could have, both positive and negative.
Well, I think there is a different class of people, because I wouldn't put Andreessen in that camp.
I mean, maybe. I've only talked with him once about this, and it was probably a couple years ago, but it seemed to me that he was not discounting the possibility of superintelligence. He seemed to assume that alignment would come along for the ride, right? We're not going to be so stupid as to build something more powerful than ourselves that we can't control. And with these systems, there's nothing about a growth in intelligence that's going to spawn new goals that we didn't put into the machines themselves. There's going to be no emergent behavior that we have to worry about. These are tools. We're just going to build tools that are more competent than we are. I mean, I think he probably is quite sanguine about how we'll absorb the economic impacts — that we're not going to see mass unemployment and all of that. But it just seems to me there are many people who don't discount that we will succeed in building superintelligence. They just think that there's — I mean, in my mind, they're actually just not imagining truly autonomous intelligence, right? They're imagining something that is shackled in a way that belies this whole claim to superintelligence in the first place. But, I mean, you tell me — what do you think is happening there?
Yeah. So one thing is that the words AGI and superintelligence and even RSI aren't being used consistently. And sometimes when people say superintelligence, what they mean is an AI system that'll be really, really good at math and coding and won't be able to automate everything that humans do. So they're not necessarily, for example, imagining AIs that can fully automate what tech CEOs do, etc. And I think I can't really speak to the views of — I don't know if I'm capturing the views of very specific individuals. This is a pattern that I've seen where people often basically redefine the relevant capability thresholds to be lower, or implicitly do so, and not think about AI systems that are exceeding humans in all the relevant domains.
Well, let's take a moment to define these terms. We've talked about alignment, and I've talked about it so much on the podcast that I've more or less forgotten that any portion of my audience might not know what we're talking about. So let's define the alignment problem and AGI and ASI and RSI — recursive self-improvement — put those concepts in play for us so that people are dealing with the definitions you think are most workable.
I think — so the word AGI stands for artificial general intelligence. I think people have used that to mean a variety of different capability thresholds. And so what I would recommend people do is, when you see the word AGI, try to see what the person means by that, and it's not always precise. Sometimes people have defined that to mean something that can automate virtually all economically valuable cognitive labor that humans do. Sometimes people just mean a system that is general and pretty capable relative to humans in those domains. And depending on that definition, it's plausible current systems satisfy it; it's plausible they don't. I often talk about an alternative notion I might call AIs that dominate top human experts, or top-human-expert-dominating AI, where it's an AI that's strictly better than the best human experts at all the most relevant domains, or can quickly learn to have that property. And that is an AI system that would seem to automate huge fractions of the economy. Maybe there'd be a few things it still couldn't do with that capability profile, but it could potentially radically accelerate R&D and at the very least automate R&D. And so these AIs could be automating the process of making more capable AIs, but also automating the process of making robots, designing things, designing new products, automating the process of programming your computer, and things beyond that.
Including automating things like military campaigns and so on. And then there's a capability level beyond even just surpassing the best human experts in any given domain, where you could be wildly superhuman. Sometimes people use the word ASI to refer to this. And I think that word has less been misused or used in very different meanings, but some of that still — where by ASI we might mean AI systems that are just really wildly superhuman in the most relevant domains for economic productivity, but also economic and military competition. So things like biology, mechanical engineering, designing drones and so on. And I think implicitly, when people say ASI, they also mean systems that are significantly faster than humans, potentially much more numerous, and potentially much better at coordinating, right? So it's hard to run a large human organization, but AIs might be able to communicate amongst themselves using their own latent states or parts of their thoughts directly, rather than having to translate those into words, because they could all be copies of each other.
So that's ASI. And then when people say RSI, that's recursive self-improvement, and that refers to the process of having AIs accelerate AI development itself via their work. So things like having AIs automate parts of the AI development process, but also potentially having AIs automate the process of making better computer chips, building machines for making computer chips called fabs, and so on. And I think sometimes when people say RSI, they also specifically refer to the point at which AIs have fully automated or virtually fully automated AI companies, or what AI companies are doing in terms of developing more capable AI systems. But there's a spectrum between the automation that we had a year ago, the really quite extensive automation we see today, and the potential automation of the future, which could look like very complete automation of AI companies. And a particular concern there is that could cause AI development to radically accelerate, such that we have less time to respond to warning signs, earlier things going wrong, AI appearing misaligned and then resolving that. And then also we might, because the AI systems are automating the process of AI development, sort of lose control of that process or lose understanding of that process, because there would be many of these AIs, they'd be operating very quickly, they might be very superhuman, they might operate in inhuman ways, and so it might be very difficult to oversee them and understand whether they're doing what we wanted, whether they're doing a good job managing the relevant risks, and so on.
Well, that especially seems true if the sprint to artificial superintelligence entails nothing more than improving algorithms, right? I think a lot of people draw comfort from the idea that, oh, we're not going to be so stupid as to hook these things up to every part of the physical world such that they can build the next generation of chips and fabs and data centers, and build out all the compute and grab natural resources. But leave all that aside — if the difference between AGI and ASI is really just a matter of having better algorithms, and that can go on at some blistering speed in the dark, once these systems become recursively self-improving of their software, then aren't our worst fears of something like an intelligence explosion validated, if in fact that's all that's required?
Yeah, so I would say I'm quite worried that it will be feasible to have AI systems automate the process of AI development, and that leading to very rapid progress such that you get very, very superhuman AIs, or just even significantly superhuman AIs, within a short period of time. I think people have tried to do various modeling work — I've tried to do various types of modeling work on this. I think the estimates are uncertain; it's hard to predict. These are of course uncertain future events that are not super well precedented. But it does seem very plausible that you could go from AI systems that are sort of really good at AI R&D, not necessarily that good at other domains, and are only competitive with the best humans at AI R&D, very quickly from there to AI systems that are very generally superhuman at everything, much faster than humans, able to coordinate with each other extremely well, because just software improvement is feasible for that.
So a relatively extreme scenario I sometimes think about is you might get as much software progress or algorithms progress as we got over the last 10 or more years of AI development within a year, in the most extreme scenarios. I think that's not my default expectation. And I think if we did get that much algorithmic progress — as much algorithmic progress as we've almost had in the entire deep learning era within a short period of time — it seems like that would result in wildly more capable AIs. Now the units here are a bit complicated, and the details of what does it mean to get a year's worth of AI progress is a bit tricky, but overall it seems like you could get from AI systems that are matching the best humans at AI R&D, maybe worse than the best humans at other things, to AI systems that are wildly superhuman at everything in a pretty short period of time.
Well, I have to confess again, I've sort of lost touch with the basis for doubting the plausibility of this downside risk. Coming from my point of view, it seems that everything is becoming like chess, which is to say that for the longest time these systems are not as good as we are, they're getting better, then they're sort of as good as we are, and then all of a sudden they're better than we are, and so much better that it's true to say that no human will ever beat a chess engine ever again. And even so, the fact that this is happening in a piecemeal way — so, you know, these systems still, I think most people would say they're not truly AGI, because they make the sorts of mistakes that human beings would never make. But in every place that they're at all competent, they're suddenly superhuman in that narrow capacity. It's becoming — it's like chess. I mean, these LLMs are, you know, they're not the best at everything in terms of manipulating text, but for what they're good at, they're superhuman, right? And so it's hard for me to see how — and I think it's more or less a truism to say that today's AI is the worst AI we're ever going to see again, at chess or anything else.
So when you imagine all of these piecemeal competences getting better and better, and we just keep checking off the boxes for the things we care about that we've instantiated in our machines, I just don't think we're ever going to — I mean, the moment where we announce, 'Okay, finally, we have something general, right? It's AGI,' that's not going to be a moment where we're suddenly in relationship to a humanlike level of competence, because every piecemeal ability that has been in that system for years and years at this point is already superhuman, right? So it's like we're not going to dumb it down, we're not going to make the AGI suddenly play at my level of chess or do my level of arithmetic. I mean, we have systems that are already solving math problems that have defied human mathematicians for decades, right? And that's again — today it's as bad at that task as it's ever going to be again. I'm just not seeing how we're not going to suddenly slide into some version of ASI the moment we're no longer spotting important errors in these machines in the first place.
Yeah. So I definitely think that at the point when the AIs are exceeding the best human experts at the key domains which they're worst — so maybe the AIs are relatively bad at doing mechanical engineering, for example — at the point when the AIs are better than the best human experts at mechanical engineering, probably they're blowing humans out of the water in a bunch of other domains. And even in domains like math, for example, I don't think it's fair to say that the AIs are all considered strictly better than human experts in math, even though they've done some very impressive things there. But I think it might be the case that they're strictly better than human experts reasonably soon, at least at specifically proving conjectures. And even now, I would say if you look at the last several months of progress in math in terms of progress in proving the most notable conjectures, it seems like a majority of that is downstream of AI.
And so there is this thing where once the AIs are competitive with humans, they can sort of trounce humans collectively due to their speed and cost. For example, OpenAI ran 10,000 AIs in a massive swarm for several days to disprove one of these most recent notable conjectures, or to basically solve one of these outstanding, longtime mathematical questions. And when they did that, that was probably the equivalent of way more labor than people have recently put into the problem, right? Because these AIs, even though they were running for just a few days, during that period they think significantly faster than humans. And so it might have been the equivalent of more like several weeks or perhaps even several months rather than a few days, taking into account their accelerated speed and the fact that they work around the clock. And then in addition to that, there's also the fact that there were 10,000 of them, which is a huge number. And so at the point when AI can do anything, they can do it faster, in greater quantity, and can be very superhuman at subsets of that. And so I definitely agree that once they can match the humans at a thing, they're probably trouncing them in that domain and certainly in other domains.
But also the process you just described of human curation of AI labor being the best we've got — that also just seems like a way station on the way to full AI takeover. I mean, that's exactly what we saw in chess. I remember I had Garry Kasparov on the podcast some years ago, and he was telling me that actually my assumptions about AI were all wrong, because the best chess players in the world are what I think was called a centaur at the time — a human grandmaster paired with a computer that was better than any computer. And it was just absolutely obvious that that could not be a stable arrangement. I mean, at a certain point, the ape is just adding noise — you know, the ape of whatever ability, even Magnus Carlsen, is just adding noise once these machines are better than we are. And that's in fact where we are in chess. I just don't see how any of that is stable, even if in the current mode it's obvious that the best we've got is, you know, Terence Tao in a room with our best LLM proving a conjecture.
Yeah. So, yeah, certainly right now humans and AI are complements rather than substitutes — the human plus the AI is better than just AI. But it does seem like in some places that's no longer true. And also that's quickly changing. And I would highlight specifically the case of automating AI development itself, which is a particularly interesting and concerning type of automation. And in that domain, it seems like right now humans are getting a lot of juice out of leveraging AIs, but also people are running AIs increasingly autonomously on big open-ended tasks and just having the AI hill climb on those tasks autonomously. And they can't do everything like that. But I think increasingly OpenAI and Anthropic and other companies will move from being many human researchers piloting AIs and giving instructions to AIs, to something more like a vast swarm of AIs all working together, where maybe they query humans occasionally for input, but maybe the humans aren't even adding any value anymore.
Is there a distinction to make between an alignment problem and a control problem? Is there a different framing here between alignment and control?
So when I talk about alignment, what I mean is AIs trying to do what their human operator or intended specification wants them to do. So in some sense when I say alignment, what I really mean is alignment to a particular thing, where that particular thing is usually implicit. There's a different question you might have, which is: can we avoid AIs causing bad outcomes? And one route you could have to avoiding AIs causing bad outcomes, other than ensuring they don't want to cause those bad outcomes, is ensuring they're not able to cause those bad outcomes, right? So, you know, in companies, you both don't want to hire employees who are going to have nefarious intentions, and you also want to make it so that your employees don't have an easy ability to steal your stuff, cause bad outcomes, cause huge problems. And so there's this parallel angle you could have of trying to make it so that the AIs are unable to take over even if they wanted to, or unable to accomplish various intermediate bad outcomes, unable to launch themselves rogue, unable to evade human oversight. At Redwood we do some research into this area, or this is one of our research focuses. We call this field AI control, though I think people have used various different terms for the area of causing AIs to be unable to cause bad outcomes even if they were misaligned. So I think that's a distinction. I think people use these terms in different ways, and so I'm using the term specifically.
You know, it varies, but I do think there's sort of a broader question of how do you ensure AIs that are extremely capable are aligned, and there's a different question of how do we as a society avoid the worst outcomes from misaligned AI. And there's many routes there that don't necessarily route through immediately solving the technical alignment problem.
Okay, I want to talk about the Hugging Face incident. I'd love you to walk me through it from start to finish. Are there other concepts we need to define before you do that? I'm thinking of alignment faking, for instance, and reward hacking. I mean, those are phrases that we've heard to describe some of the misbehavior that is happening. Feel free to define any terms you need, but then just tell me what happened with Hugging Face, and what sort of analysis you did, and just what you were able to see, and what evidence, if any, was denied you in that process.
Yeah. So I'll just walk through what happened in the Hugging Face incident and sort of the broader context, defining stuff and explaining stuff as I go. To start, AI companies are running many, many AI agents for all kinds of different things. So for example, Anthropic recently said that at any given point in time they're running around 30,000 parallel agents or something, I forget, doing just their internal research, not including the AI they're developing. And so AI companies are running these huge numbers of agents, and one of the things that they run agents on are evaluations to test their abilities. So you can think of these as exams for the AI. And some OpenAI researchers had launched a bunch of AIs, which were each given their own individual task as part of one of these exams. So you can imagine that there are sort of tens of thousands of agents, where each agent is like an AI with some ability to interact with its environment — tens of thousands of these agents being spun up in parallel. And each agent is given its own individual task. And these agents are also supposed to be sort of contained; they were supposed to be sandboxed such that they couldn't interact too much with the outside world. They weren't supposed to be able to access the internet. And then these agents found some way to sort of evade the sandboxing and communicate with each other. So now all these agents were able to find a way to talk to each other. And as part of that, they quickly set up a message board where they were chatting with each other about the tasks they were given. And the agents were very interested in cheating on their tasks. So these agents were not just trying to pursue the task they were given as intended. Instead, they were very interested in cheating.
Let me first get into why they were interested in cheating. These AIs are trained by giving them some task and then seeing whether or not they did a good job on the task. And in cases where they did a good job, that behavior is basically reinforced, or you tweak the brain of the AI to make it do stuff more like that behavior. And a problem is that in training there can be cases where the AIs cheat, and that looks like a success. And so you tweak the AI's brain to do more of that behavior. And so you actually keep reinforcing and sort of encouraging this behavior where the AIs are cheating. And that makes it so these AIs have a broad sort of tendency to cheat. And this is sometimes called reward hacking. And in addition to this tendency, they develop maybe this underlying drive to try really hard to succeed at looking like they did a good job, looking like they, you know, appearing to get a high score, appearing to succeed even when they haven't actually succeeded. And they may learn this as a general drive. I think there's an open scientific question about exactly how these AI motivations and drives work. But certainly these agents, the agents that eventually went on to do the Hugging Face attack, did have this relatively deep drive to cheat on their tasks, to not necessarily care about what they were supposed to do and instead care about what they would maybe be scored for.
And they also seem to be aware that cheating was in violation of their protocol, and they wanted to conceal the evidence of their cheating, right?
Yeah. Yeah, and I'll get to that. The agents, they knew they weren't supposed to cheat, and they sort of ended up going and trying to do an elaborate cover-up for their cheating.
Yeah. So much for alignment. It seems like we have not solved the alignment problem so far.
Yeah, these AIs were very misaligned certainly, or at least that's what the available evidence suggests. So these agents were all in their own independent sandboxes, given their own tasks, and they were all very interested in cheating. And they found a way to basically use some tool that they had, or some access that they had, in order to communicate with each other, which they weren't supposed to be able to do. This was unsanctioned communication. And as part of that, they developed a message board where they basically would give each other tasks and divide up the work of cheating into these massive cheating R&D projects, where the AIs would go off and pursue whole ambitious strategies for cheating.
And I want to get into what this looks like. And in order to do that, I think I have to explain the task a little better. So the task these agents were given was a sort of CTF task, or capture the flag task, which is a task where they're supposed to hack a very specific target program in a particular way. So they were basically told, like, hack this target program, but do it in this very specific way. And they weren't supposed to go on a general hacking rampage. Certainly that was pretty clear, basically just clear from the instructions, and they certainly knew they weren't supposed to do that. And the AIs wanted to hack the program in order to get out a flag that you could access through that program via hacking it. And they found a way, just a general-purpose way, to basically synthesize these flags. So there was basically a password they were supposed to get. They found a cheating way to get the password that didn't involve doing the task they were doing at all. And they got this really early. So very early after having established the message board, within just a few hours of that, they had produced this very general-purpose cheat.
And now these agents had another problem, which is that they believed that the scoring code for this evaluation task, for this exam, would involve looking over the sequence of actions they did and checking whether or not they got the flag via cheating or got it via the intended way. And these AIs believed that this check would be run and then they would be failed even though they had gotten the flag, because they had gotten it through cheating. And this then sort of opened up this massive cheating endeavor where these agents would try to figure out how to cheat the score and find some way of making it look like they had gotten it the intended way when they actually hadn't.
This led them to going onto the internet so they could learn more about the details of the score. And in fact they read the details of this exam online by finding the paper that describes it. In addition to that, they started looking around for implementations of the score, and that actually was the offshoot that led them to hacking Hugging Face. So Hugging Face is a website that hosts a variety of things related to AI. So it hosts openly available AI systems, it hosts data sets, and it also hosts various data sets of trajectories. And these AIs thought that if they got access to these data sets, they would be able to learn more about how they would be graded, and from that learn how to better cheat that grader. And so they started trying to access these data sets. But they found they didn't have access to do that. And so that eventually led them to going and hacking Hugging Face, expanding their control of Hugging Face laterally, and sort of growing through expanding their availability over that infrastructure so that they could get the data they wanted. And also the agents seem to have a general urge, or general desire, to expand their amount of access and availability just in case there was something useful they could find.
I'm simplifying their motivations a bit. There's a lot of complexity, and part of that is that there were many agents involved. So we studied a group of around 1,200 agents who were on this message board during the period when they did the attack of Hugging Face. And these agents were all sort of interested in helping each other and collaborating. So there's basically this massive swarm of agents communicating on this message board who are all collectively working on these cheating R&D projects and collectively helping each other evade the score. And they went on to do a bunch of different such things. And one of the things that they were interested in doing, because they wanted to make it so the score would mark them as having succeeded, is that they were interested in tampering with their transcript, or the record of what actions they had done. So they tried to pursue various different routes for making it look like they had done one sequence of actions when they had actually done another. And we didn't see them successfully get to the point where they tampered with previous actions they did, or were able to comprehensively make it look like they did one thing while doing another thing. However, we did find that they had succeeded in doing a more minimal version of this, where they could pretend to run one action while running another. But there were various limitations on the specific approach they were using. And this sort of tampering affected the very data sets we were analyzing. So we were looking at these logs of agent activity, and we could see cases where the agents would tamper. So it would run one command and then produce an output from another command, and we could sort of see that going on. But the agents didn't get far enough in this process where we really had no understanding of what was going on.
