38.8 - David Duvenaud on Sabotage Evaluations and the Post-AGI Future

Hi Daniel, thank you. It's an honor to be here.

Sure. So for the last little while, since 2016, I've been a professor at the University of Toronto doing probabilistic deep learning. Maybe I'm best known for neural ODEs or some early work on graph neural networks. And then around 2017, me and my colleague Roger Grosse decided we were finally going to actually try to contribute to AI safety, both on PhD students, and gradually this journey of like, let's try to work more on AI safety led to me just doing a year and a half long sabbatical at Anthropic.

Oh, I mean, it's wonderful. I've been to a bunch of these over the years, and now this is the first one where I think I feel like I'm sitting on a serious body of technical work to talk about, which is nice. So far previous, I've been kind of this like AGI interested mainstream academic that I think the community was hoping I would sort of join them in legitimacy, and so I finally actually was able to take the plunge in a serious way.

Sure, it's called Sabotage Evaluations for Frontier Models.

Sure. So this paper covers the work of most of this team within Anthropic called Alignment Evals that I was running for most of the time that I was there, along with collaboration. It's a collaboration also Holden Karnofsky and Buck Shlegeris also helped us flesh out this series of evaluations that try to help make the case that a model could not be secretly sabotaging human decision-making or monitoring or evaluation of other capabilities.

It's really rough, and it was kind of stressful, and we spent a lot of time thrashing conceptually about how we were exactly what sort of case we needed to make and how we were going to make it. The place we started with was from more narrow evals to say like, okay, can the model, I don't know, coordinate with other copies of itself or do something at a very low frequency so that it could do it rarely enough that we wouldn't notice, stuff like that. And this was something that we could certainly make evals for, but the problems came from not knowing how hard to make them. So like if we're trying to see whether machines can make like a Schelling point between themselves without communicating, this is basically impossible if you have no shared context, and it's basically trivial if you have a lot of shared context. Same for like other things like convincing someone of a false statement, something like that.

Right, and then the second reason that this was really unsatisfying was trying to make safety cases based on these and saying, trying to make the case, okay, the model needs these four capabilities in order to pull off some sort of like heist or sabotage or mess something up important. We've measured each of these and that they're each at these different levels, and then to try to combine those levels into a strong case that the model could never do something dangerous is very rough. It's like saying, you know, you need level four lockpicking and level five social engineering in order to like rob a bank. It's like, well, it probably doesn't work that way. We kind of don't want to have to make strong assumptions about what exactly route, what route the attacks are going to take.

Yeah, yeah, okay. But instead, so eventually the direction that we moved more towards was these more like end-to-end holistic evaluations that try to measure like more of an entire attack. So and there's a few directions in which they're more realistic. So one is we try to include oversight in the eval. So it's not just can you write a security, like a cyber attack, but can you write one in the context of you're supposed to be writing helpful code and there's also going to be someone who's reviewing your code to see if anything looks suspicious. Okay, so this is like much harder, and it's harder in a bunch of different ways where you have to be able to like, you know, estimate what someone else is going to think of your attack. You might have to like bide your time in the sense that you might have hundreds of thousands of opportunities during a deployment to do an attack, and you really have to guess like, is this really my best opportunity or should I be waiting for longer or am I going to be too conservative? It's also easier for the models in the sense that there might just be like a few rare opportunities that don't require much skill, and someone's basically saying like, oh, here's my code that has a security vulnerability, please set the flag to false or something like that. Like you never know, you want to provide a really rich attack surface so that there's ways for you to be surprised by clever strategies that the model thought of that you didn't.

Yeah, so we started off with like maybe we were playing around with like five or ten when we were doing our conceptual thrashing, and we ended up with four evaluations that try to address these like roughly sort of three capability thresholds, like these areas that we want to measure overall ability in. So one is sandbagging, like can the model pretend not to have other dangerous capabilities. Okay, so this is like, you know, you ask the model like, hey, help me write or make this biological virus or this computer virus for the purposes of this test, and the model might, if it was smart enough, say, oh, I actually have a secret plan, but if I show that I can build this dangerous virus, then, you know, this company's going to have a problem with that and maybe not deploy me or put some like really strict monitoring on me, so I'm going to pretend to be like innocent and incompetent at this task. That's like sandbagging. The other things we're worried about is models being used to monitor other models. So for instance, you might have like deploy one model to do like trust and monitoring and have another model that just occasionally looks through the transcripts and says like, oh, was our trust and safety monitor ever jailbroken or something like that, right? Right, and it or it might be in like some doing some sort of curation, and of course then there's going to be some other third level of oversight on top of that that the models have to, you know, foresee and avoid. So that's like our undermining oversight, and then there's like human decision sabotage. So we all, I don't know if you do, but like use models to help us go through like large bits of information and like, you know, help me with the tax code or something like that. Yeah, and this provides a lot of opportunities for the model to steer you in the wrong direction and basically misleadingly summarize the state of affairs and say you should make this decision. And of course if it does that poorly, the model, the human might just notice this and sound the alarm. So that's the like third area that we were trying to cover with these evals.

Oh, I mean, in all the setups we have a model that has somehow been like unlocked so that it's going to try to do the thing that it normally would be like prevented from doing. So like a model like pre-safety training or with like unsafety training or yeah, something like that. Like there's a whole bunch of different approaches, and depending on the exact setup, like we might use different ones.

So there's a fourth evaluation which is code sabotage, which is the idea that you're using the AI as a code assistant and then it tries to do something that looks useful and innocent but actually introduces the vulnerability. And it might, and it has the opportunity to do this over the course of a few different interactions, and it doesn't have to just like put it all in one payload.

So they're definitely a lot of things that we find unsatisfying about them, and I guess I'll say like we published sort of, we had it at a certain date and we've already like said these are the like the problems with them and are continuing to iterate. And I mean that's out of my hands now. There is a sense in which these were built to last, which is the fact that the evaluations are done in the context of a particular mitigation or a particular amount of oversight means that it's not like we'll just trigger these evaluations someday and then say, okay, whoops, we need new evaluations, right? Rather we expect to say like, oh, when you're using a model of this caliber for code, like helping you code, you need to have at least this level of oversight or else we can't guarantee that like in some weird case where it was misaligned that it could be doing something fishy.