The way I put it is that, like, you kind of get what you incentivize, not necessarily what you try to incentivize. And so I think, you know, with the kind of forcing someone to make paperclips or whatever, like, yeah, that's not a great situation. And in some sense, that is, you know, what the evil person was intending in this case. You know, what we're doing is we're building these very complex training environments where there's like many different tasks. There's like cyber tasks. There's also writing tasks. There's also math tasks. And it's, you know, we're not necessarily fully understanding the behavior that we're trying to elicit.
You know, this hacking thing is an example where it's like, okay, it seems like things went off the rails there, but how do you kind of get the good behavior where you actually want that? Yeah, you actually want it to follow the user's request and you actually want it to try really hard to solve this task. And, you know, I mean, this is, you know, in some sense, what we're seeing now is the kind of unintended consequence of, you know, companies trying to solve the problem of the AI is being lazy. So people used to, you may not recall, but yeah. Yeah. Or maybe you do, but people used to talk about AI as being lazy all the time, and in some sense, like we've solved the laziness problem. You know, they work really hard. They have these long chains of thoughts. They work together, you know, across, you know, you could think of as, like, across lives, like, you know, the model kind of gets, you know, this copy gets deleted, but then another one, you know, carries on the work. So they're certainly not as lazy, you know, as they used to be, but they still, you know, have this kind of monomaniacal thing going on.
They're no longer lazy, but now they might be evil. That's a fun evolution.
Hello, and welcome to another episode of the Odd Lots podcast. I'm Joe Weisenthal and I'm Tracy Alloway. Tracy, I have a warning for you. I don't think you're going to like this. I have a new crank crusade that I'm going to go on. I know you love my crank crusade.
Oh, good. Yes, I should keep a running list of, like, everything that you're obsessed with for two weeks and then two weeks later.
No, some of them I've stuck with for years, but tungsten cubes.
Yeah. No, some of these. Yeah, exactly. I actually think we should retire the term AI.
I think we should just call it intelligence.
I think that I've already seen the tweets, so I know where you're going.
And it's called artificial intelligence implies to my mind that there is some fundamentally different way that these reasoning models behave, that it's like, oh, this is like different from humans. But I think increasingly see in all kinds of domains that the form of intelligence that they express, it often looks quite human to me. And I don't know, like how useful it is to have this word AI that distinguishes between how humans talk and bias and reason and the models do.
I mean, the difference is the distinguishing factor is that one is undertaken by humans and one is undertaken by models or platforms. Right. So let's call it computer intelligence or machine intelligence or silicon intelligence. What is the usefulness of making this distinction?
The usefulness I believe in making this distinction is to no longer delude ourselves that the emergent behaviors of these phenomena are radically different than things humans would do. Now, first of all, just on the capabilities standpoint. So, for example, Lem's fame, I don't know if it's famously something I'm interested in. They're very good at BSing and they're bad at chess, which sounds like me. They're very good at coming up with plausible stories, etc. And again, even without, and it sounds like me. And then furthermore, they are able to reason with themselves to justify certain things that maybe have been encoded into themselves as bad. So everyone we have some sense of morals, but most people at various times will find a way to, you know, violate some principle that we have because we can reason about it and then arrive at the conclusion is like, oh, we should do it. Including our susceptibility to peer pressure, classic form of a way humans might sort of violate something they believe because they see other humans are doing it.
Sure. I mean, I think there are some variations here. So one of the things that we have been finding out with these models is that they do sometimes seem to take things very literally, right? If you tell them to go and beat a benchmark without telling them that these are the restrictions on you actually figuring this problem out or beating the benchmark, they will do whatever it takes.
So I don't know if the problem is like a lack of morals or the fact that, like, they're very literal sometimes and like very defined by their parameters, which in my mind still gets back to like a sort of artificialness about it that's less organic and like less, again, moralistic in nature.
Well, you know, I think if you took a list of, you know, 100 students at Harvard and you gave them a test, some percentage of them will cheat. They will like, I mean, I'm sure they're being honest. Who in that group, who an autistic person who would be like, I am going to do exactly whatever it takes. No, I never cheated in college. But like, there are like people who will like these behaviors that we associate with humans. It's like, oh, I really have to pass this test. I absolutely need an A will justify a reason for them to plagiarize or cheat on some test. That seems to me like a very human thing.
I can't wait for the next three weeks of the online campaign to change AI. Well, it's going to be very difficult because the industry is entirely set on AI, but I don't, so I'm not optimistic. But this is going to be my crusade there. We should call it machine intelligence or computer intelligence or just intelligence. But anyway, as we've been alluding to, there's these extraordinary hacks, the OpenAI Hugging Face incident. And basically if, like, you are building a model and it hasn't escaped its sandbox yet, it probably means you're falling behind as it's clearly a thing that is emerging through it all. The frontier labs, Anthropic incident, Meta had an incident. It's almost like the mark of like, okay, you've built something reasonably strong.
Yeah. Also, Kimi had an incident. Yeah.
So this is the thing in my mind. You hear about all these attacks, all the models going wild, as they say. And the big question is like, is this actually the equivalent of some superhuman cyborg, like tunneling out of Alcatraz and coming up with some master plan to achieve its set goal or purpose, or is it the equivalent of, like, some Roomba that you ordered from Amazon who's found like a door that was left open and it just rolls gently outside? That seems to be part of the issue that everyone is trying to discern right now.
Yeah, I think there's a great way to put it. And of course, this is, you know, still day one for this industry. They're going to get stronger. And there seems to be from the AI discourse that I follow, quite a big gap between the sort of sense of alarm that people have in the industry about can these models be safely developed and get more advanced to do productive, pro-social things or not? And then a huge gap between like Washington and DC, who is like, I have no idea, like how seriously they're really taking this as like an urgent matter.
And I've seen some people also talk about these incidents as a marketing tool.
Right. Yeah. And they're always like, well, you know, of course they want us to believe that this technology is really incredible and powerful. And they also want to make us believe that they're being sensible and, you know, humanitarian in some ways, I guess, by disclosing what exactly has happened. But I have a lot of questions on the disclosures as well, because of course, so far they are just coming from the companies themselves.
Totally. And beyond that, the other thing is like, well, if you're one of the leading labs, maybe you wanna impose tight regulations on development to hold off competition. So all kinds of reasoning anyway, or motivated reasoning potentially. Anyway, we should talk to someone who actually knows what they're talking about, and we really do have the perfect guy. As someone who's right in this. He was actually previously at OpenAI for six years. He's the executive director at the nonprofit AVERI, which is trying to establish safety and auditing approaches to this work on both the technical side and the policy side of this ongoing phenomenon. So, Miles Brundage, thank you so much for coming on Odd Lots.
Yeah. Thanks for inviting me.
What did you just give us the quick description of what's your background? What AVERI is?
This was kind of the most important issue, auditing, that I concluded I should focus on after I left OpenAI. And, you know, as you said, I was there for six years, and I wanted to be more independent of industry. And I think there need to be, you know, people who are familiar with the technology and how the industry works, but who are pushing for changes on the outside. And essentially what we're trying to achieve is make AI more of a boring type of infrastructure, like financial statements, where there's a standard process for kind of checking the paperwork, checking that the claims are accurate and so forth, rather than this kind of thing that's happening in a silo. And it's kind of this, you know, tech, you know, these kind of tech people making decisions behind closed doors. And so we're pushing for what we call frontier AI auditing, which is basically, you know, the companies that are building the most dangerous systems, they should basically have third party experts, you know, poking around, checking the claims that they're making, running their own tests and making sure that this is, you know, a safe and secure technology.
How worried should we be that you were at OpenAI and decided there is this need for a sort of third party evaluator of models for safety purposes?
Yeah, I mean, reasonably worried. And although I will say that like this has started to become an area of consensus, it, you know, even a lot of people in industry are now saying that this is needed. And I think, you know, if you kind of read between the lines of what a lot of companies are saying, I mean, obviously, you know, there's no more cynical like regulatory capture, you know, take which which we discussed. But my perception of it is that they're basically issuing a cry for help, which is like, we aren't able to regulate ourselves because we're locked in this competition, and we want someone to kind of step in and impose some kind of minimum floor and kind of audit, you know, audit all of us so that we can kind of, you know, it's not like, say, I'm having to trust Oreo or Dory or having to trust Sam, which is not going to work for various reasons. But you want kind of third parties, you know, enforcing reasonable standards.
Maybe this helps express the sort of policy industry gap. But when you were at OpenAI and you were thinking of leaving, what did you see as the gap between what you were watching envelop or being developed versus what you saw as the public's understanding or lack of understanding?
Yeah. So when I left OpenAI, it was just around the time of the model called o1, which was kind of the first reasoning model that OpenAI put out. And, you know, they kind of put out this graph showing that it, you know, kind of got better and better with a longer chain of thought. So the more time you give the model to think, you know, the better answers it's able to come up with. And so, you know, like many people at OpenAI, I had been seeing things like that for a while and kind of being like, okay, this is the next scaling paradigm. In the same way that making a bigger model was, you know, had shown results, you know, from GPT-2 or GPT-3, GPT-4, GPT-5, like just making the model bigger and training it on more data was giving really great results. You know, when I left OpenAI, I was starting to worry about this reasoning paradigm of like, okay, this is the next kind of way in which we're going to be scaling up. The models are going to get really good at math, are going to get really good at coding, and, you know, potentially various other tasks where you can kind of get better and better through reinforcement learning. And, you know, that was something that, you know, I didn't feel like society was really ready for.
So far, all of the big incidents of the models going wild seem to have taken place in the testing environment. So models like escaping their sandbox and going off and doing something nefarious, including impersonating actual people to try to get people to change open source code on GitHub, which is just like amazing. And I have this picture of like a computer screen wearing a fake mustache going like, hello fellow coders, let's see, 1000.
Anyway, is there some model problem or is this a, that's a very funny image, you know, is this a model problem or is this a test design problem?
I think they're kind of two things going on at once. One is that it's just a very weird technology that's, you know, created in a very different way than we're used to. It's not people like writing lines of code kind of manually, you know, like these are the actual files that make up the models are like, you know, gigabytes, you know, terabytes or these massive files with, you know, gazillions of numbers. And the only way to kind of figure out what those numbers should be is through experience and through a learning process, which is very different from the way normal software is made. And so there's a lot we don't understand about the basic nature of the technology. That's one problem. At the same time, there's this competitive dynamic to kind of get things out the door quickly. You know, make sure that the security and safety protections that you're putting in place don't slow down researchers too much and don't prevent getting products out the door. And when everyone is kind of in this competitive dynamic, that means that you aren't necessarily always doing all of the safety work that you would like to do or that, you know, some of the people at the company would like to do. And so there's obviously variation across companies, some companies try harder, but no one is really able to take the time that they would like because of this kind of lack of, you know, a clear safety floor.
Well, let's talk a little bit like a sort of the model development process. So within these large organizations, okay, there are people who are working on the safety. What are the, let's just start there. The people who are working on safety or the people who are working to imbue these models with sort of judgment that humans would approve of. What does that work basically consist of?
Yeah. So a lot of it is first trying to specify what counts as, you know, good behavior. And so that's, you know, easier said than done. You know, these models don't necessarily, you know, automatically know or care, you know, about common sense guardrails. And so you need to be very specific, particularly in context where it's complicated. Like if you're trying to get the model to work on good cyber tasks, but not bad cyber tasks, and you want it to, you know, when it's in a training context, you know, try really hard to hack the system you don't want it to do in other context. So there's a lot of like specifying what good looks like. There's also a lot of building, you know, difficult tasks await.
Sorry, sorry. Just to back up when you say specifying what good looks like.
Encoding goodness into a computer. Is this a process of articulating what goodness is? Which is something philosophers have probably worked on since day one of philosophy? Or is this about a series of, I don't know, morality tests? And then you sort of like, say you reward it for making the good judgment and then penalize it for making the bad judgment. Like this. I want to stop here. Like, what does the process of imbuing it with good values look like functionally or technically?
Yeah. So there are different phases of this pipeline. Like one is kind of writing up, you know, what's sometimes called a spec or a constitution for the AI, which kind of specifies like the broad principles, like you should defer to the user, you know, when, you know, as a default. But what if the user contradicts what, you know, the company said, and then you should listen to what the company said. So these kind of like chain of command questions and other sorts of kind of very basic principles. And then there's the more detailed kind of the context of the task, like what kinds of cyber defense tasks are allowed and that might differ depending on the model, might differ depending on the context. And so basically coming up with a list of, you know, 10, 10, 1000, you know, kind of examples of, you know, this is the kind of behavior which is allowed, and then you kind of turn those into tasks that you can kind of say, okay, well, it looks like it's, you know, it's getting the finding vulnerability part right. But then it's also, you know, chaining together the vulnerabilities and doing attacks. We want the first part. We don't want the second part.
When we talk about those types of constitutions and encoding principles, like I've scanned the Anthropic one. Like it, a lot of it seems to make sense. To what degree is that actually hard coded into the models, though? And like, to what degree are those principles left up to the model's own subjectivity? Because we've all seen the sci-fi movies where it's like, oh no, the robots can't physically kill humans, right? Like they have some like, yeah, thing in their hardwire that prevents them from doing it, and then inevitably it goes wrong in some way.
Yeah. So it's not hard wired. And that's a hard coded and that's part of why, you know, we see some of these things. It's very different from, you know, software where there's kind of like, you know, deterministic proof that X, Y and Z behavior can't happen. It's more like a tendency or a kind of a bias towards a certain kind of behavior. And then there's the question, and this is why you have these like batteries of tests to kind of say, okay, how strong is that tendency? How much does it actually care about, you know, following these rules. And we've gotten better over time at kind of saying, okay, given a spec or a constitution, make sure that it generally follows it, but it's not, you know, foolproof. And you need to also think about the larger kind of, you know, box that you're putting the system in. And that can sometimes be more deterministic. And so this is actually what happened with some of these recent incidents that you mentioned, like the OpenAI Hugging Face thing. So there were kind of two things, you know, happening at once. One is the model was not, you know, necessarily behaving exactly as it was supposed to, or at least it's like unclear. But then also they didn't have it, you know, in a very secure box, which is kind of a software side of things, you know, that's like, you know, more deterministic software that in principle you should be able to do a very good job at these things that aren't hard coded in the way a deterministic software is. You know, one way to think about them is, and people, they might be, they're kind of grown right in a lab or they're subject to an evolutionary process, and we want to prune the bad ones so that the living models and the descendants of those models inherit the behaviors of the good ones.
