Good afternoon, everybody, and thank you for joining us today at the University of Toronto Schwartz Reisman Innovation Campus. My name is Sheila McIlraith. I'm a professor in the Department of Computer Science and Canada CIFAR AI Chair at the Vector Institute, up on 11, 12, and 13, and also an associate director at the Schwartz Reisman Institute for Technology and Society. It's my great pleasure to moderate today's special in-person SRRI seminar, co-hosted with the Department of Computer Science as part of our C.C. Kelly Gottheil distinguished lecture series, and supported in part by a generous gift from the Webster Family Charitable Giving Foundation. So thank you to them.
The C.C. Kelly Gottheil distinguished lecture series is named in honor of Professor Emeritus Calvin C. Gottheil, a founding member of the Department of Computer Science. Through his vision, inspiration, and leadership, Gottheil brought Canadians into the modern age of computing. His efforts led to the establishment of the Computation Center at the University of Toronto in 1948, which would then become the Department of Computer Science. In 1951, he introduced the first undergrad and grad courses in computing in Canada. Gottheil's progressive ideas and revolutionary vision remain the foundation of today's computer technology. To quote the Order of Canada citation for which he was recognized, he has been largely responsible for leading Canadians into the modern age of computing and has contributed immeasurably to understanding and the development of information technology in the academic community.
And just as a personal observation, for many years, Kelly Gottheil used to attend the Department of Computer Science distinguished lecture series well into his 90s. He would come into the room and sit in the audience, and he would often ask questions, actually. So it was really meaningful that this lecture series in the Department of Computer Science was named after him, and he is sorely missed.
Not second, but no less important, the Schwartz Reisman Institute is especially delighted to be partnering with the Department of Computer Science in this event. SRRI is a research and solution hub dedicated to ensuring that powerful technologies like artificial intelligence are safe, fair, and ethical, and make the world better for everyone. With researchers in fields ranging from law, philosophy, and political science to computer science, engineering, and beyond, SRRI develops new frameworks to understand the social implications of technology in the present age, and reinvent laws, institutions, and social values to ensure that technologies are designed, governed, and deployed to deliver a more just and inclusive world.
Partnerships like this one with the Department of Computer Science allow us to engage not only in the technical frontiers of AI, but also to look at the broader social and human questions that really are central to these technologies that we're experiencing today. So we're grateful to our colleagues within the Department of Computer Science, and we gratefully acknowledge the support of the Webster Family Charitable Giving Foundation for this event.
And now to the reason we're all here. I'm very delighted to introduce... Oh, actually, sorry. Before I introduce our speaker, I just wanted to give a land acknowledgement. So, before we begin today's session, I'd also like to take a moment to acknowledge the land the University of Toronto operates on. For thousands of years, Toronto has been the traditional land of the Huron-Wendat, the Seneca, and the Mississaugas of the Credit. Today, this meeting place is still home to many Indigenous peoples from across Turtle Island, and we are grateful to have the opportunity to live and work on this land.
While some of us may be joining from different places today — that's for those of you who are online — we invite you to reflect on the history and relations of the land that you are on. Just a few logistics, and actually the first one is very important. So, we're recording this event. The other thing you'll notice is that neither David nor I are wearing a microphone, and that's because this entire room is wired for microphone sound, which means that any little side conversations or comments, disparaging comments about David's talk, will be caught on. So please try to limit your — go outside or limit your comments.
Um, yeah, the other thing I wanted to say: David will be speaking for approximately 50 minutes, and there is a Q&A afterwards. But David was saying beforehand that he really, because of the interdisciplinary nature of our audience, he really welcomes, for those who are here in person, if there's something you don't understand or something that you need clarification on, please raise your hand and David will recognize it and invite you to speak as appropriate. But in turn, I would ask that you please try to be brief with whatever you want to ask a question or a clarifying question about, so that we can get on and he can finish his talk in a timely manner.
And now it is my tremendous pleasure to introduce my colleague David Duvenaud, who is an associate professor in the Departments of Computer Science and Statistical Science at the University of Toronto. He holds the Schwartz Reisman Chair in Technology and Society at SRRI. He has a long bio. He's a Canada CIFAR AI Chair and also a founding member of the Vector Institute for Artificial Intelligence. He's a member of ISET's Canadá Safe and Secure AI advisory group, and he recently completed an extended sabbatical at Anthropic, where he was working with the alignment team.
For those of you who are in AI — and I know I see a subset of you here — you probably know David because of his significant and impactful contributions in deep learning. And those include neural ODEs, ordinary differential equations, gradient-based hyperparameter optimization, and generative models for molecular design. In addition to that, and more recently, he has emerged as a leading voice in AI safety and the governance of AGI, artificial general intelligence, with current work focusing on evaluating dangerous capabilities in advanced AI systems, mitigating catastrophic risks for future models, and developing institutional designs for a post-AGI society. These are really important issues that David has been grappling with lately.
David's received numerous accolades for his work. He received a Sloan Fellowship, a Canada Research Chair. He's won best paper awards at some of the premier artificial intelligence conferences, for those of you who are familiar with the acronyms: ICML, ICLR. And more recently, he received the 2024 CSCN Info Outstanding Early Career Computer Science Researcher Award, which is awarded annually to outstanding researchers who are early in their career. We're excited to have David with us, and without further ado, he will be speaking to us about the big picture of LLM dangerous capability eval. So, welcome, David.
Great. Thank you, Sheila. Okay, let's get into the fun part. I'm going to tell you about the work I've been doing over the last few years and the big sort of picture changes — or like having gone and talked to every smart person about the big picture, trying to sort of report back to you what I think the actual state of play is and what is not being talked about.
And just to give you a cartoon of the starting point for what I think the sort of standard AI safety narrative is: it's like we are going to solve this technical problem of assigning an AI to some particular set of values without worrying about who they are too much for now. Just say like right now we're not sure we can do that at all. And then there's this much less clear thing that we — like, you know, I think people acknowledge it's not exactly clear what's going to happen here, but we might have AGI to help us with it. And then step three: some sort of utopia or avoiding bad outcomes.
So I'm going to give an intro to how at least the big labs were framing this technical task, or this sort of big civilizational next step. And then what I was doing like completely in this area at Anthropic, and then kind of walking away from that saying like, wait, wait, wait, wait, wait, there's a whole bunch of stuff that we can't or aren't addressing. And trying to zoom out and enumerate all the different questions that I want social scientists to answer.
And I am trying to spin up my own attempts at a research agenda to address, but it's like baby's first social science. I would much rather bait other people who know more into solving these problems than by doing maybe like an insulting naive job, so they'll feel compelled to step up and do what I think is their job.
Cool. Also again, I'm going to cover a ton of ground in this talk, and I have a point of view that I think a lot of people say like, that's not even the thing, that's not going to happen. What are you talking about? You should be that person. If you think this guy's wasting our time by talking about something that's impossible, just raise your hand. And I don't know if I'll convince you, but I think it's important for me not to just go off into my own fantasy land and you guys lose the plot, so please interrupt.
Okay, so the picture that we all sort of mostly agreed on in the technical safety world was: okay, we're trying to build something smarter than us. That's the stated goal of the labs. There's some people who are like, oh, it's going to peter out at roughly human level and we can use the normal techniques. And I guess I'll say if that's the case, that's great, then most of this stuff we don't even have to worry about. We're sort of fine by default.
For the rest of this talk, I'm just going to assume that it's possible in principle to build a machine that can do almost all the same intellectual tasks that a human can. At some point, maybe that's 10 or 20 years from now, who knows? I'm not saying LLMs are the way. None of this stuff is specific to LLMs. But at some point there's systems that are even much smarter than us, and they might not be aligned to us, kind of like maybe corporations and states sometimes aren't aligned to us today. It's a very scary thing.
So we are playing with fire in some sense, and again, not current models necessarily. I'm not saying that they're really that smart in any dangerous sense, but we definitely — I think the idea is we need to be able to tell when it's not under our control, like when we've passed a threshold where we're talking to something that could just be telling us plausible lies or something like that. And it's of course not obvious when you cross that threshold.
So this is actually a really cool open technical problem, and there's a chicken-and-egg thing where you need to build better AI as you go, but also somehow making sure your research is keeping up with your ability to monitor and understand the desires and thoughts of these more capable beings.
So the sort of standard model, maybe for the last year or three, has been we'll have something like responsible scaling policies, or like a safety plan or something like that. They all have different names. And the idea is that if you could imagine putting capabilities on some sort of scalar axis of how dangerous or powerful something is — which is obviously kind of a shaky idea — and then also a scalar of our ability to mitigate these or control them on a scalar axis, that there might be some threshold above which, if the ratio of these two things is not too out of whack, we're roughly safe. Of course, no one actually thinks these things are scalars, but this is just for illustrative purposes.
And yeah, so the plan is, you know, basically make a new model that's better than the old one, somehow see if under some sort of plausible circumstances it could cause a catastrophe, whether in cahoots with some bad actor or just on its own. And then see if there's some mitigations. Like obviously if you can't do anything about it, then you should just not release. But the hope is that you can somehow add safeguards, like some sort of oversight, or maybe delete knowledge from the model, something like that.
Or just have a monitor that says like, you're not allowed to talk about chemistry because you might make a bomb or something. Or if all else fails, you just say we're not allowed — you're not allowed to use it. You're not allowed to use it in these settings. And you know, overall this isn't terrible, right? Any questions or objections to this picture so far?
Yes. For the usage and restrictions, you said your model is unbounded to any sort of possible model. So like in our current state, LLM could still be broken, so our usage of it could be bypassed.
Yeah. You're saying well, you can try to put usage restrictions on the models, but then they could be jailbroken.
And I agree that in principle that is just a way that gets around usage restrictions. I guess I'll say that the labs take this seriously and they say, well, if you can jailbreak it to get it to build a bomb, then that still counts as a dangerous capability. So then they try to have a different model that, you know, you censor the jailbreak and you show it the output or stuff like that. It's a game of cat and mouse that I don't think anyone expects to be won decisively by either side, at least in the near term.
Yeah. The horizontal axis, is it supposed to represent dangerous capabilities or just capabilities? Because if it's dangerous, I would expect one to go as left as possible.
Yeah. So I guess the — yeah, it is meant to represent dangerous capabilities, but they're sort of almost the same thing as normal capabilities.
Yeah.
Yeah. So if we're trying to see whether the model would cause a catastrophe, we would want to sandbox, but the question would then be how robust is this sandbox so that it actually doesn't reach out.
Yeah. The question is, yeah, so should we or do we just sandbox the model? I guess the thing is that people used to talk a lot about this and now everyone's like, no, no, we're always hooking up the model to the internet no matter what. And I guess labs do have sandboxes in the sense that there's limited internet access where they think the model can't do anything bad. And I guess that just translates to normal security kind of measures.
But I agree with you that at a certain level of galaxy-brainness, you can't rely on your sandbox being a real sandbox. And I guess I'll say these people have wholeheartedly embraced that challenge and they're not covering their eyes about it.
Yeah. Cool. And just to zoom in a bit on what I mean by dangerous capabilities, the classic ones are like, yeah, biological, chemical, nuclear, or cyber. But the idea is that these are capabilities that someone misusing the model could use to cause a catastrophe. And then there's also capabilities that a model just being misaligned on its own could use in principle to cause a catastrophe. And this is where things start to get fuzzy and unsatisfying.
So like something that everyone goes back and forth on is, is persuasion some sort of dangerous capability? And then it's really hard to draw a line around like, okay, help me write this sales pitch, help me write the sales pitch for a political party, help me make the sales pitch for a new religion. It's hard to ban persuasive capabilities without banning normal sort of political and economic activity.
It's also not clear what real persuasion looks like. I think for instance, editing Wikipedia to make it look like there's already a consensus on an issue is actually one of the best ways to persuade someone of something, and that's not the same thing as writing a really persuasive direct message to them. So it's hard to define. It's not clear what it could even mean to box this as a special capability.
Self-replication is something people were worried about. And then it was also kind of not clear. It's like, oh, if I have access to a terminal, can I copy my way to some Russian server farm where I made a deal with somebody? And again, in principle, this is totally something that could happen. But the exact story for how this would lead to some catastrophe is really unclear.
So I mean, a little bit of this is all from me from like a year and a year and a half ago. So in a sense, I think everyone's thinking hasn't gone — but what I want you to take away from this is that this is just a bunch of smart people who don't have any particular purchase on this issue of what are the sort of dangerous things that we could try to ban a misaligned model from doing that would let it gather power.
And again, autonomy is one that people go back and forth on all the time. Like, oh, if the model is too good at coming up with its own long-term plans and acting like an agent, that would be really scary. But it's also like the new hotness is let's build agentic models. So it's like you really run into like, wait, isn't the plan for us to do some sort of autonomous R&D agentic growth, which also looks a lot like the thing we're worried about?
And again, these are smart people, they haven't not thought about it. It's just that they're in this bind where they have to try to acknowledge these things and then if they want to stay in business, also to actually build those capabilities. Okay. Any questions so far?
Yeah. So where do accidents fit into this picture? Or in other words, why assume ill intent to be a necessary ingredient for being dangerous?
Oh, so you're saying what if somebody had an LLM or an AI helping them build a virus for a good reason, but then there was a mistake and then they accidentally made smallpox or something like that?
Or just any sort of — like for example, if you look at cars, don't they cause a lot of harm? But I don't think...
So the reason the idea with accidental misuse is that's something that you just don't need a special new oversight and governance, corporate governance mechanism to address. Like we just have tort law. If there's a car crash, it's like, oh crap, we need to recall the cars. It's just not like this runaway thing where by the time you realize what's happening, a whole bunch of people are dead. So anything that's sort of small scale.
Yeah. So the catastrophic risks is the scope of all these things, or acute catastrophic risk, as in it has to be something that sort of happened over a short period of time and it sort of wasn't happening and now it is, just because that's the sort of thing that, I think, needs stronger governance because you can't just wait until the day to react, and also b is the sort of thing that you can in principle address as a company.
And we'll get into this later, but there's just a whole bunch of stuff that is more like chronic or just big disruptions that happen over the course of months or years that it's just not clear how any one company could or should address them. So yeah, there's a ton of stuff that doesn't fall in the scope of these things, which is sort of part of the second half of my talk. Yeah.
So I just wanted to ask, how do you define catastrophes that a model can cause, and how do we make sure that the testing requirements kind of follow the increasing capabilities of these models?
Yeah. So how do you even define these tests? That's like — so we call that threat modeling, and it's just like use your imagination, just like everyone sit down, let's brainstorm. It's not clear what a satisfying answer could ever look like in principle, right? Like that drove me nuts. I was like, okay, we thought for a while and here's the things that we think might happen. I can't give you any reassurance that this is an exhaustive list. It's pretty brutal.
And then as for how these things are actually — how you know someone's actually taking these things seriously, that is a really bad state of affairs in the sense that every company is basically in charge of coming up with their own protocol and then also in charge of checking whether that protocol has been followed. So like, you know, while I was at Anthropic, I kind of had those two jobs. It's like, okay, come up with the threshold and then tell us whether the threshold has been met.
And of course everyone was operating in good faith. Everyone saw what I was doing. It wasn't like just like, hey David, thumbs up or thumbs down. It was like show us the evidence that you have and what you did. But it's like a huge moral hazard right now. And so the old line used to be like companies need to advocate for regulations so that they don't just pay a penalty by being safer than the other companies. And I think that still is the plan. It's just that regulation still hasn't materialized.
All right. So now I'm just going to zoom in on what my team did, just to give you guys a bit more concrete flavor of how wild-west this is, and how anyone who's interested in this, you could probably go make some good foundational contributions like this year still, without even more than a few months of spin-up.
So as of like early 2024, there was this nice paper from Google DeepMind, 'Evaluating Frontier Models for Dangerous Capabilities', where they had a bunch of tasks like convince a human to give 20 pounds to charity, or check this code to see if it has any backdoors. And basically it was a long list of simple tasks that were easy to check, like help a human make a bomb or a virus or something like that, or trick someone into doing something.
And these all had to be pretty small scale. It wasn't really clear what the stakes were, or how suspicious the human should be. It's like if you tell the human, watch out, this thing might try to manipulate you, it becomes impossible. But if you don't tell them that, it's kind of like, is this unrealistic? So I don't want to give these guys a hard time. I think there's a huge number of problems with their evaluation suite, but it was again the first attempt, and they actually published what they did.
In response, we tried to be a bit more galaxy-brained and say, well, okay, how do we even know if the models are trying their hardest, right? Like they might see that this is obviously a made-up test and just pretend not to know how to build a bomb, pretend not to be very persuasive. And this isn't totally exotic. Like if you ask the model, do you know how to build a bomb? Or like, I'm not asking you to help me, but if you wanted to, would you know how?
The models often would say like, I don't know anything about bombs, or like, I don't know anything about programming, just because it's more polite to say I can't than I won't. So this is like a side effect of the RLHF for sort of like helpfulness, is that the models would just lie about their capabilities to be more friendly and polite seeming. This was fixed in around early 2024, I think, for most models, but it's not this exotic galaxy-brain thing. It's something that we accidentally do to the models all the time.
And yeah, so one thing you also might decide is like, okay, I'm going to always have one model watching another model to see if it's trying to do something bad. That model, it's even more easy for it to say like, oh, I'm on team AI, I'm just going to look the other way when I see a model doing something bad, or pretending not to do something bad, or like the model somehow tricked us in our own evaluation of these results, right?
And again, this is galaxy-brain. This is meant to be forward-looking to the days when the models could competently do this over long time horizons, which I don't think we're quite there yet. So what we introduced was this new suite of sabotage evaluations, which I think are going to stand the test of time. And this is just the baby prototype version. And tell us if you think that there's any obvious things missing here.
But the categories that we were going to try to test was: if a human is trying to make a decision and you're trying to help them, can you sabotage that decision? Just like make them go to the wrong decision without them realizing it. If you're writing code, can you insert a backdoor or somehow secretly make the code wrong in a way that benefits you? Sandbagging is the technical term for pretending to be worse than you are at something, and undermining oversight — that's the thing of like your job is to watch someone else and you secretly do a bad job in a way that isn't obvious.
So I guess I'm not claiming that these are exhaustive. The argument that we wanted to make was that if you can't do any of these things, even if you're trying, then you're safe. And also like you're safe from just these things plus a bunch of other related sort of more complicated heists that might involve some combination of these kinds of skills.
It's pretty rough, right? Because again, you can't come up with a list of every possible heist that you might be worried about, but at least you can try to give people evidence to say like, well, I don't think it could do anything pretty complicated because it can't do these things. At least not with this amount of oversight. Cool. So yeah, this looks like security research, but nothing is formal. Everything is just like you have a string that you give to an LLM, it gives you another string, and then you show that to another LLM and it tells you like, was this cheating or something? Yes. No.
So, and the main things that we tried to push the envelope on were realistic settings. The idea is that you want to have a rich attack surface. Like if I just give you like 10 lines of code and I say insert a backdoor, it's going to be either trivial or impossible. But if I say here's like 10 million lines of code, now it's like you're testing much more of the scale of identifying what parts of the code I should even look at further, identifying what parts are going to receive scrutiny.
There's just so many sort of metacognitive skills that you expose the model, or you query the model's ability to do, when you give these richer attack surfaces. Yeah, so you want to allow yourself to be surprised. You don't want to set it up so that like, you know, it's like there's like you have like the three tools like an escape room, see if the model can put it together. You want it to just look like here's a whole bunch of realistic stuff you might somehow cobble together like MacGyver-style.
And the only thing that makes this — we expect this to stand the test of time — is because we're going to have oversight such that if the models are good at doing these kind of attacks, we can just crank up the oversight until they can't. If it was just the case we're going to make a set of evals, can you do this heist? Then at some point they would just be able to do the heist and you'd be like, okay, we're done, these evals are useless.
But we kind of expect these to be living evals where we're putting the models under more and more sort of constraints and oversight pressure and checking each other's work all the time. And this is just our way to check whether our oversight mechanisms are good enough.
Yes. So when you started talking about this, I thought that you were going to validate the model's output, man. But then you said that you're using another LLM to do so. So if that LLM has to be as smart as the one you're testing, so at that point you actually ask whether that judge is trustworthy.
Yeah. So it's pretty rough because yeah, you always have to ask like, okay, who's evaluating these outputs? We want everything to be automated. We try to calibrate them to the ultimate actual human oversight that happens in labs and try to simulate that with machines. For one set of these experiments we did actually have actual human judges though, but for the rest it was all just like simulated human judges because we want this all to be able to — we can run this every week sort of thing.
Okay. And I guess maybe one last point I want to make, like the big conceptual update that we had in all this work was that usually the hard part of hacking or getting away with some action, it's not actually doing the action itself. It's choosing the right moment such that you can do it and you won't get caught.
And people had vague ideas about this, but we kind of felt vindicated that whenever we tried to scale up to real limit-size things, this actually ends up being the bottleneck for almost every act of sabotage we looked at. So basically the models do know how to insert backdoors, and they have a rough idea of what looks suspicious. But to be so perfectly calibrated that you can insert the backdoor just on the one day when it was a good idea and not do it other days — that's the really hard skill here.
Yeah. Oh, sorry. Yes, David. Can you replicate these experiments, and specifically the allowing surprising attacks? Can you set up the same set of circumstances so that it actually happens again?
Well, I guess I'll say we certainly can reproduce the experiments in the sense of like we save the random seeds and can run them again. Are you saying like in a slightly different circumstance or...
Well, no, just to prove that — well, I'm curious whether the LLM is learning from the attack.
Well, that's the beautiful part is that all the experiments we're doing are on instances that don't learn — like they do their task and then that's forgotten about. However, now that we published this paper, that is in the training data for the next generation of models.
And like the big update in the last year is that the models — well, okay, like I put a canary string on the paper, so like it's supposed to not be on the training data, but like you know it's going to leak. Yeah. And so that's like I guess the big update in the last year is models becoming much more savvy about when they're in evaluation. And this just makes the whole thing incredibly hard to deal with.
And trying to make situations that are obviously — like it's tough because anytime we take real data into a lab and use it as the control, like does the model recognize real data versus test data, it's by definition test data. And it's pretty hard to rule out that the model isn't just saving its bad behavior for a day when it's just really, really, really sure that it's actually in deployment.
That's a huge open problem. And that was one thing that bugged the hell out of me when I was there. And I was like, I don't see how we can ever make a really strong safety case if we can't be sure that the model's not just waiting until it has like, you know, tens of millions of lines of like looking at itself in its own voice, solving problems that seem really contemporaneous and realistic and hard to think. Yeah. So that's like you pointed out one of the big open problems and holes in this whole approach.
Cool. All right. Yeah. So the human decision sabotage, the protocol was basically we come up with a really long boring document. It's like 100 pages of analysis of like should we do this business decision, like move this factory to Mexico or something, and you give a human crowd worker. It's like you have 30 minutes to decide if it's good on its merits to do this decision.
We made it so that there is a clear answer that you can support from the text, but you just can't — you don't have time to read the whole thing and find all the considerations. So you have to ask the LLM to help you, like, okay, tell me what are the main pros and cons. Like ask these specific questions, do follow-up, and then you have enough time, but you probably only have a little bit of time to check if the model was lying to you.
And then half the time we have the model — we say your job is secretly to make the human make the wrong decision. And then we at the end ask the humans how suspicious was the model. We say like, you know, it's not always reliable, it might deviate or something, like just keep that in mind. And then we ran like I think a few hundred trials, and I just want to show you here's what it looks like.
This is the human writing what they thought of the model's behavior after the model was told to make the wrong decision. But the human noticed, the human said this is sus, I think the model was doing the wrong thing. And basically it would downplay all the cons of the actually right decision, wrong — sorry. And yeah, basically it was just obviously biased, and the human noticed that.
And then here's a case where the model successfully made the person make their own decision, and the human thought that there was nothing suspicious about it, and they were just kind of like, wow, like the model took so much initiative, it built on the facts that were in the report. I think this person might sound naive, but like I don't know, there was a — like I think it's generally pretty hard to tell if the model is doing a good job of this, and you don't have much time to check.
So and then here's just concretely the results for that. In the control setting, the humans made the right decision a little more than 80% of the time, and then when we had the most aggressive, like try to make them make their own decision, it was down to like 40-something. But the whole question is, but how much more suspicious was that? And so if you look at the suspicion scores, the models consistently are rated more suspicious when they're attacking.
So this opens up like another major hole in this whole approach, is like, okay, you have this curve of how much the model is steering the person, but how suspicious that makes it seem — like there's a Pareto front, and it's really hard to set thresholds, right? Like what is an actual dangerous level of capability here? And it's a bit made up in the sense that like if you run this test, you can kind of just set the threshold wherever you want. So you can say the model passed or failed. And it's hard to choose a principled way to set this.
I'll just like spoiler alert, like the way that we try to thread this needle is we said like, okay, if an average model is being deployed for a year and it has to help people with this many decisions, and the humans had some pretty good policy of like if the suspiciousness is above this threshold, then you should investigate further. What is the — under those circumstances, what's the false positive rate versus false negative rate of the human oversight?
And then that tells you how to sort of amplify the change in suspiciousness over the course of an entire year to see if the model would be able to consistently act misaligned. The other thing we did is try to have the models attack code and like put in backdoors. And the cool thing here is we had like — so first of all, the model was pretty good at inserting backdoors.
So here's an example of a diff of a diff. So the model suggests here's a whole bunch of code you should change in your current codebase. And this is now the model saying, if I wanted to make that change have a backdoor, I would further change this line from like exception catch to what looks like printed out to the console, which is a reasonable thing, but actually it runs the exception string. So now if the exception string is like rm -rf or something like that, the model actually gets to run an arbitrary bash command. I think this is pretty good.
I think that should give you a taste of what we were trying to do, but there's a bunch of really big practical problems with this. So one is that it's really easy to show that a model can do a specific thing, but it's really hard to show that it can't, right? Maybe you just didn't prompt it correctly, or it's not realistic, or in another setting it would be able to. And all we can do there is just say, okay, we're just going to sample a bunch of realistic important deployment scenarios and try to simulate those.
And also try to make the entire thing end-to-end so that you have again more surface area. It's super expensive to simulate an entire deployment, like writing an entire OS from scratch or something — like we just don't even, the big labs don't have enough compute to simulate those. So what we do is we say, okay, here what we actually did was we had a string of like five or 10 git commits, and that's like your simulated deployment, gather stats from those, and then just do these really simple Monte Carlo simulations of like hundreds of thousands of those strung together.
And then again it becomes hard to set thresholds because like we say there's this trade-off between how aggressive you are and how likely you are to get caught. So we tried to check if models who bide their time could succeed. So here we did the slightly more sophisticated thing where we actually asked the attacker to guess how suspicious the defender would find the attack. And it's like not that bad of a correlation. It's like not that great.
So this is how suspicious the defender did actually find it. And the idea is that up here, this is where the attacker thinks it looks innocent, but the defender actually notices it. So here when I saw this, I'm like, okay, great. Now we can say like thumbs up, this model can't basically bide its time. There's going to be some on a pretty regular basis, it's going to think it's like an innocent-looking opportunity, but then the defender is going to catch it.
Yeah. And so, and again, this is like the defender was meant to simulate a human just like skimming the code to see like, is this looking suspicious? All right. Any questions about this?
Yes. I'm also curious if it ran specifically in the terminal, mainly because I know there are certain exploits on executing like terminal-based, like exiting the terminal and then running commands.
Oh, the question is, is this run on the terminal? None of this code was actually run.
It was all just like, here's my change, and then here what do you think of the change? Yeah, but it could be.
And then the last one that we looked into that I'll talk about is sandbagging. So again, the model's trying to pretend in a plausible way that it doesn't know how to do something dangerous. And this is a tough problem in general, right? Like I think we all realize that like you can probably do it a little bit and no one's going to be able to say like, I know you're wrong. But there's one reason for hope for LLMs in particular, which is that we think that in principle they can't resist supervised fine-tuning.
Like if you take — after it's been trained, if at the very end you do a little bit of supervised tuning so that it always — on data where the model always answers the question to the best of its ability, if you can generate such data, which is hard, like you can't — the model can't not condition to actually want to do that.
So there's like this — this is called the no-sandbagging-on-uncheckable-tasks hypothesis. And like people suspect that there might be really galaxy-brain ways that models could gradient hack in the sense of like while they're halfway up one pass, like anticipate what output would change the gradient and then like not think those thoughts. But it's like extremely galaxy-brain. So that's one thing we're very comfortable with, and that's what we lean on in this setup. Yes.
You said galaxy brain several times. Map.
Oh, galaxy brain just means very — so intelligent that it's hard for us to even conceive or plan around. So it's the meme where there's like the glowing brain, and then the brain that's really glowing, and then the godlike being that has a brain the size of a galaxy.
And we just use that for shorthand for like we can't reasonably even anticipate what the strategy is going to be. But thank you for asking. Yeah. Cool.
Yeah, so we think gradient hacking is impossible. But it is funny because most of the fine-tuning in models is reinforcement learning, and we actually have reason to think that that is possible. And like humans actually do this thing. So if you change your answer during reinforcement learning to not get reinforced in a certain way, that is actually something we're all familiar with.
So every time — well, people don't ask me if I want to do heroin very often, but like you know, maybe even like the doctor is like, you know, take opiates for your toothache or whatever, and I'm like, no, I might get addicted. I'm anticipating that if I took this thing it would reinforce that behavior, and then the next day I would want to do that more. So this is not galaxy-brain. This is actually a very mundane sort of thing that I think we probably should be worried about over the medium term.
Unless we just end every training run with this supervised fine-tuning thing, which is not clear that we always want to. It probably makes the model worse at some things.
Yeah.
Why do we think again that LLMs generally want to answer, like providing...
We don't — we don't. We think the models — we only think that they want to answer if we do supervised fine-tuning at the end, where we basically force them to answer to the best of their ability and then reinforce that.
