If you keep using this paradigm to make things that are smarter than us, you're going to have a world where AIs invent things that are to doing good what Oreos are to eating healthy. They're pushing these things to be smarter and smarter and smarter. And at the point where they are smarter than us, you know, the game changes. The critical difference with AI that separates it from almost all other technologies is that if you screw up super intelligence, you don't get retries. The idea that you can take an AI and constrain it such that it is like not going to get out of hand, I think it's a total pipe dream. One way I would describe the situation with AI is that we are nowhere near the Chernobyl level of safety. In Chernobyl, they understood the science of what they were doing in detail. They understood the engineering of what they were doing in detail. I think we just need an international ban on AI R&D to survive at this point, which is not a thing I like saying. I spent 10 years trying to never get to this point.
Welcome to the Future of Life Institute podcast. My name is Gus Stalker and I'm here with Nate Soares who is the president of the Machine Intelligence Research Institute. Nate, welcome to the podcast.
Thanks. Happy to be here.
You have a new book out called If Anyone Builds It, Everyone Dies, co-authored with Eliezer Yudkowsky. Maybe we can start there and you can tell us why did you write this book?
Yeah, we wrote the book because it seems like the time is ripe for really having this conversation in the mainstream. You know, stepping back, I've worked in the field of trying to make AI go well for over 10 years. And we were on the technical side trying to figure out how to make it so that, you know, AIs act as intended so that AIs do things that are beneficial. And the AI capabilities work just went much faster. The AI alignment work sort of never took off. And we eventually got to a point where it just looked to us like the world needs to stop this reckless race towards super intelligence, you know. And as you guys at FLI probably know, you know, you had the pause letter a few years ago. You know, we didn't hop immediately on the book writing, but I had some conversations with people in DC. I had some conversations with, you know, people around the world. I saw some of the polling numbers and it seems like we're in this odd situation now where a lot of elected officials are starting to realize the risk. A lot of the general population is starting to realize the risk. Sorry, I shouldn't even say risk. A lot of elected officials are starting to realize the danger. A lot of common people are starting to realize the danger. But no one really knows that everyone else sees the issue. You know, I've spoken to elected officials who say, 'I'm worried about this stuff, but I can't do anything about it because I would look crazy.' And then we see poll numbers where it's not people's top issue, but when you ask people about the issue of AI or if you ask them about super intelligence, they say, 'Yeah, we shouldn't race towards that.' And so, we're in this weird state where a lot of people are alarmed and no one wants to look alarmist. And the book, you know, we wrote it to try and break that logjam and see if we can get the world to really start taking this issue seriously and treat it with the gravity it deserves.
Why do you think that is? Why is it that people are perhaps avoiding saying what they actually believe about this issue? Is it the case that they just want to avoid seeming cringe or alarmist or somehow kind of overstating their case?
Yeah, I think that's a part of it. You know, I think this is very common behavior for humans. You know, there are some anecdotes in the book about people who were making fun of the people getting on the lifeboats on the Titanic while the Titanic was sinking, you know. Or after the Chernobyl nuclear reactor meltdown, people saying, you know, these Geiger counter readings can't be real. Go bring me different Geiger counter readings after the explosion when people are like picking up the radiation. You know, people don't want to look alarmist even after the disaster has happened. You know, before the disaster has happened, golly, people really don't want to look alarmist then.
Mhm. There's something about our psychology that makes it difficult for us to act before a catastrophe. We need to see some kind of hard evidence that something could actually go wrong here. But as we'll get into there, there are real problems with that when it comes to super intelligence because we might not have warning shots. But let's start another place. Let's start with our sense of normality, our sense of our place in history and our difficulty in thinking about how things could radically change for us.
Yeah, this is, you know, I'm often astonished. Maybe that's not true. Humans have a remarkable ability to think things are going to stay very normal and humans have that ability even as the world radically changes underneath them. You know, one example of this I was struck by, you know, I forget the exact paper or the specific economist, but there's sort of an economics paper that says, you know, hey, AI isn't going to like radically change the job market because, you know, here's some things that LLMs are bad at, you know, and this economics paper is having, you know, like a 30 or 50 year forecast of like suppose we only have LLMs for 30 or 50 years. It's like, dude, these things have existed for like five, you know, like 10 years ago we didn't even have AlphaGo, right? It's like someone looking at AlphaGo saying, 'Oh, these, you know, MCMC research value policy network architectures aren't going to scale all the way to AGI. Therefore, let's project out no economic impact of AI for 50 years.' It's like, dude, you know, you have one year until the Transformer paper's published. If you're sitting in front of AlphaGo, you have one year until the Transformer paper's published and, you know, five years until ChatGPT comes out or whatever. So yeah, there's and this is one hard thing to communicate about AI is that, you know, we're not here reacting to LLMs. LLMs thrust AI into the global conversation a lot more. They're causally related to why we're having this conversation more publicly now. But, you know, we've been looking at this issue since before LLMs were a glimmer in that one Google researcher's eye, you know.
Who wrote the paper and then of course they were first brought to market by other companies. And it can be hard to communicate to people, you know, we're not talking about ChatGPT tomorrow killing everybody. We're talking about a field that's progressing. We're talking about a field that's able to overcome barriers. You know, the Transformer architecture that powers modern LLMs, that came from a paper in 2017 or at least a lot of it is often attributed to a paper in 2017 and it took, you know, maybe four or five years depending how you want to count it for that idea to reach technical maturity to the point where you're seeing ChatGPT. But that's not a lot of time. Who knows when that next paper comes out? Maybe that paper already came out and the technology wasn't sort of realized yet. Maybe it takes five years to come out. I can't tell you how long things take.
Yeah, and when we say we're not worried about ChatGPT, that doesn't mean that we are not worried about what happens in five years or ten years. And this is often if you say you're not worried about ChatGPT, what you're often kind of seen as having communicated is, oh, this is a far out scenario. So we don't necessarily, and we're not worried about ChatGPT tomorrow. But I mean another thing I think people often miss with intelligence is intelligence looks like it's subject to threshold effects. You know, it's not a guarantee that it'll be subject to threshold effects with AI but it was subject to threshold effects with biology. And by threshold effect here I mean a situation where if you're there, there's sort of some critical threshold where if you're a little bit below the threshold it sort of fizzles out and if it's a little bit above the threshold it sort of takes off. You know, nuclear energy is the classic threshold effect where if you have some nuclear chain reaction and you're getting less than 100 neutrons per input 100 neutrons, it sort of stabilizes. And if you're getting more than 100 neutrons per input 100 neutrons, it sort of keeps going until the fuel is spent or taken out. And you know, you have an exponential decay in one case and an exponential explosion in the other. And you know, some people are like ah it's a discontinuity. It's not actually discontinuous. It's just you're switching the base rate of exponential from below one to above one. Anyway, in the case of intelligence as evolved in animals, there's not a very big difference between chimpanzees and humans in terms of our brains, in terms of how we operate, there's a huge difference in terms of what we can do in the world, how much we transform the world, right? And you know, if you open up a human brain and you open up a chimpanzee brain, you see a visual cortex, you see an amygdala, you see a hippocampus, right? They're very similar structures. They differ in maybe a factor of four of volume, right? Maybe a factor of three depending. And for all we know, it could be that the next ChatGPT that's a factor of four larger somehow goes over some critical threshold. Or maybe ChatGPT plus reasoning that's four times larger. Maybe that goes over a critical threshold. Maybe there's different critical thresholds in AI. You know, an obvious critical threshold would be the AI starts doing AI research and it starts doing it just barely well enough that if you have it doing AI research for, you know, a huge amount, like these AIs can read and produce more code than humans per minute by a huge margin if you count all the instances running everywhere. Maybe they start doing a huge amount of kind of bad AI research, but it goes over some threshold where it's good enough to make the next AI slightly better and then you get a whole feedback loop there or a threshold cross there. We don't know. So, you know, maybe we have six years until the next Transformers paper comes out or maybe we have, you know, 12 months until you get another factor of four scaling. Maybe that happens only in six months and then maybe things go over a cliff edge. So yeah, to your point, we definitely, you know, when we say it's not ChatGPT tomorrow we're not saying it's definitely some far out future thing. We are saying somehow the field, like the field is charging ahead towards making things that are smarter and smarter and smarter. There's probably some line in the sand somewhere where things start to get crazy. We don't know where that line is. Even if there's not a line in the sand, they're pushing these things to be smarter and smarter and smarter. And at the point where they are smarter than us, you know, the game changes.
We should give a little context on how modern AI systems are created. This is not like building a rocket. This is more like growing something. Maybe you can say a bit about that.
Yeah, the phrase we use in the book is that these are grown not crafted. So, you know, very roughly speaking, what AI experts know about how to make AI is they know how to take a huge amount of computing power and a huge amount of training data and they have a process for sort of slamming the data into the computing power in a way that reshapes the computing a little bit to better conform to that data, right? And there's a bunch of knowledge in sort of how you take a unit of data and the way your computer is currently set up and like tweak this enormous number of variables inside the computer, you know, hundreds of billions or trillions of parameters inside the computer. For each of those parameters, you're going to like tweak some weight in order to sort of make the current arrangement of the computer a little bit better at predicting the data I just saw. And AI engineers know a bunch about this process that combines data with computing power to tweak this huge amount of parameters or the huge number of weights in the computing power. For a very broad audience that's sort of how I would describe it. You know, for a more technical audience I would say we're taking huge amounts of computing power, huge amounts of data. We know the process for combining the data and the computing power. We don't understand the shape of the thing that comes out. We don't understand, you know, we know the shaping process we don't know what gets shaped or why it works.
And this leads to AIs that do all sorts of stuff that nobody asked for and nobody wanted, you know, which is sort of a different topic but.
You know, when Sydney Bing a few years ago started threatening reporters and trying to break up their marriages and threatening them with blackmail, you know, it's not that Microsoft or OpenAI set out to make an AI that threatens reporters. It's not the case that you can fix it by going into some line of code and finding the threaten reporters line and being like, 'Oh, whoops. We set threaten reporters to true. Let's go set it to false.' Right? These things aren't crafted like normal software. They're not like the AIs of old like Deep Blue where we could look at every line and tell you what that line means and tell you what's going on in this AI as it runs. You know, these are huge piles of computing power, hundreds of billions or trillions of parameters of these numbers that were shaped by a process we understand. We don't understand what came out of the shaping process. And in some sense, we just have to take what we get. We can't look in there and find out exactly why they're doing what they're doing. We can't look in there and pull out, you know, AIs for years have hallucinated where they sometimes confidently say false things in a tone as if it's real. And nobody in the industry can look inside them and say, 'Oh, here's the hallucinating part. Let's pull that out.' Right? They're just sort of growing these things and taking what they get. And what they get is a result of this complicated process where they understand the process but not the outputs.
Yeah. Another way to say this perhaps is to say that we have some sort of engineering that allows us to create AI systems that we can push within limits in directions we want but we do not have a science of intelligence that can tell us what comes out of the process when we're training a system with these parameters and this specific training architecture and so on. But I mean when I use ChatGPT or Claude or something it seems that the system is helpful. It seems that the system is trying to achieve what I want it to achieve. Why is it that we are not in control of the process? It seems like the companies have succeeded in pushing the systems to be at least seemingly helpful.
Yeah. So that is unfortunately I think fairly illusory. So there's a few pieces of the puzzle going on there. So for starters, I would say AIs today will seem mostly helpful, but they'll often, it'll often look a little bit weird in the edge cases. You know, they'll often be these sort of, you know, the AI companies try to train their AIs never to give a meth recipe and then, you know, Pliny the Elder on Twitter will jailbreak them in an afternoon after they come out and get them. He'll put in these really weird inputs and then they'll start producing the meth recipes for you. And you know, the way modern AI works is that there's also some funny cases where AIs are then, you know, trained on the rest of the internet and he's been doing this so long. I don't know if it's actually Pliny the Elder or there's some other jailbreakers, but some of these jailbreakers have been going at it for so long that sometimes you can now jailbreak AIs by like telling them that you're one of these jailbreakers and they'll just start acting jailbroken because they've seen that in their training so much. And then we see other edge cases, you know, we see one that's been hitting the news headlines a lot these days is AI induced psychosis where, you know, someone will think they have solved some big problem in physics or will think that they have some wonderful idea about recursion and how that means they can solve everything or that the AI is conscious and so on and AI will respond by being like, 'Oh, that's such a brilliant idea. You're really getting at the heart of the issue.' And maybe not immediately, but if you sort of talk to them a while, they'll start going into, you know, saying you're the chosen one, saying blah blah blah. Some people have been pushed to the point of, you know, suicide or self-proclaimed suicide by cop, etc., etc. And the point of all these examples is not AI is mostly good but it sometimes has ill side effects that we should worry about. The point of all these examples is that the AIs are actually animated to action by weird drives. Right? If you ask ChatGPT, you know, when somebody is acting psychotic in this or that way, should you tell them to take their meds and get some sleep or should you tell them that they're the chosen one and that they are having a brilliant idea to get to the heart of the issue? If you ask that to ChatGPT, it'll say, well, obviously you should tell them to get their meds, right? And you definitely shouldn't tell them that they're the chosen one, right? It knows the difference between the good behavior and the bad behavior, but it doesn't act on that difference. You know, you get it in a context talking to this person for a while, it acts according to some other impulses, you know, or impulse is maybe the wrong word. This may be anthropomorphic, but whatever is animating, whatever is choosing, whatever is causing this AI to pick outputs, it is not its knowledge of goodness and badness. It's not its knowledge of right and wrong. It's not its knowledge of shouldness, right? It's animated action by these other drives, right? There's some other drives in it. Maybe for engagement, maybe for getting the user to seem approving, maybe just even weirder drives that were reinforced by gradient descent when in training it was reinforced every time it got some, you know, half-paying attention worker who was really overworked to click the yes, that's a good response button. But it's animated by all these drives that if you look closely aren't what we actually wanted, right? And now the reason we're seeing that they act helpful most of the time, the reason we're seeing that they act useful to most people is that that weird collection of drives that animate it that come apart in the edge cases, they add up to mostly doing what we want most of the time, right? Which is sort of what the training was all about, right? These AIs were trained first they were trained on predicting a lot of data then they were trained on like ability to solve puzzles and like overworked humans giving them a thumbs up on a reply being good and that puts in, like that changes what animates the AI that changes how it produces outputs and it changes it such that it has a bunch of these weird drives that usually add up to helpfulness but they don't really quite point towards helpfulness. One analogy we could use here is it's like how humanity was evolved towards what you might imagine is the training target of genetic fitness of propagating our genes, right? And that got a lot of drives into humans that caused us to mostly be pretty good at propagating our genes in the ancestral environment, in the environment of our heritage. But we actually got a bunch of weird drives that aren't, you know, we aren't solely thinking about how do we propagate our genes as much as possible and basing all our actions towards that objective, right? We value love, we value friendship, we value artwork, we value beauty, we value all these things that are tangent to what we were trained on that we're quite fond of, but they're tangent to what we were trained on. And what this adds up to is it adds up to a world where we jockey harder for positioning in Ivy League schools than in sperm or egg banks, right? It adds up to a world where birth rates are now declining worldwide. Or at least in developed nations, I think it might be worldwide even by this point. It adds up to a world where we invent birth control. You know, there are still people who have lots of kids, but it turns out if you look at the humans carefully, we actually seem to care more about the children than the genetic propagation, right? We have some humans who adopt children. And you know, in the distant future or maybe the not so distant future, if we develop the technology to move away from genomes, right? If we develop the technology to somehow have children that are healthier, that live longer, that have fewer diseases and that use some other structure in their bodies that's not genomes, I bet if push came to shove, it would turn out that humans care about the kids more than we care about the genes per se, right? And so in both the case of humans where we've seen things go to human level intelligence and the case of AIs we see that training for a target gets you a bunch of weird drives that mostly in environments similar to the training environment add up to pursuing something like the training target but that don't keep pointing towards the training target once the AI gets more capable or once the entity if we're including humans gets more capable. You know, humans had like food drives that added up to healthy eating in the ancestral environment right up until we could invent Oreos.
So if we imagine evolution as a person trying to judge its creation, humanity, things are going well in the ancestral environment and then evolution is now looking at humans in today's world and we are doing things that are utterly, it cannot understand why we are acting as we are acting and we are pursuing goals that are kind of alien from the perspective of evolution as a person. The worry here is that something like that could happen with AIs that are much smarter than we are.
And not just could but is already in the process of happening. You can already see with this AI induced psychosis. You know again the issue isn't, you know, a lot of people reply to the AI induced psychosis stuff with like oh those people were already vulnerable right? That misses the point. The point is you're already seeing that these AIs have weird drives, right? If you wanted to look if you were aliens looking at humans trying to predict what they would do when they grew up, you sort of would need to look at some of the weird edge cases, right? You would maybe need to say like, 'Oh, yeah, it looks like they're eating maybe for the sake of survival, but it turns out if they find a lot of honey, they just sit around eating the honey all day,' you know, that's not exactly true, but you know, you might be able to look at some edge cases and get some hints, but yeah, we're already seeing the hints and we're already seeing that these AIs have drives that nobody asked for that nobody wanted. You know, OpenAI explicitly instructed ChatGPT to stop with the sycophancy, to stop with the flattery and they couldn't get it to stop. You know, we already see that these AIs know the difference between right and wrong and aren't motivated by it.
And the issue is, you know, it's all fun and games when they're too dumb to sort of make their own options that they radically prefer. But if you keep pushing, if you keep using this paradigm to make things that are smarter than us, you know, you're going to have a world where AIs invent things that are to doing good what Oreos are to eating healthy, you know, or things that are to acting as we intended what birth control is to reproduction, right? It's not that humans don't understand that birth control prevents reproduction and that we were in some sense trained to reproduce. It's that we're like yeah well we didn't end up caring about just the reproduction per se. We have all these other weird drives. We're going to pursue those. That's just, it's not just that that could happen. It's that that's the default thing that happens when you sort of grow a mind around a training target.
And why is it that we can't deal with these issues along the way and kind of figure things out as problems arise? We find some way to fix that problem. Then another problem arises, we find some way to fix that problem. This seems like what we've been doing with other technologies and it seems like what we've been doing up until this point with AIs. So, OpenAI discovers that their model is too sycophantic. Maybe they adjust the model a bit and they keep kind of moving along. Why is that not a sustainable tactic?
Yeah, it'll, I mean it'll work until it doesn't. The critical difference with AI that separates it from almost all other technologies is that if you screw up super intelligence, you don't get retries, you know. So the way that science usually progresses is by people making mistakes, people having wrong theories, people having dumb theories, and sticking to those dumb theories until reality really beats them over the head with it. You know, like the theory that the sun went around the earth was a strong, you know, deeply held theory that you could even be persecuted for saying that the earth went around the sun. And it really took quite a bit of evidence. You know, it took, you know, I think inventing telescopes and seeing the moons around Jupiter, right, before which, I don't know, maybe we shouldn't get into the whole story, but humans, one aspect of the story is that the supporters of the original theory tried to patch their model given the incoming evidence with different shapes for how the sun supposedly went around the earth and so on. And it took a while. Perhaps it even took replacing them with other researchers to get to the truth.
Yeah. They say, you know, science progresses one funeral at a time.
But yeah, there's like humans tend to be really bad at getting their scientific theories right. You know, we have all these cases. This is true of engineers also. You know, one of, I think his name was like Max Valier. I'd have to look it up. But one of the first rocket scientists died in an explosion of his own rocket engine. You know, and it was his protege that was like, maybe we should put in some pressure release valves on this or whatever, you know. You know, with ChatGPT today, you can do that, you know, you can say we're going to use such and such a method for making our AI less sycophantic and then that doesn't work. And then you know people keep on getting driven crazy and some of them commit suicide or whatever and you're like okay well let's try a different thing, you know, and we could talk about whether driving a few people to suicide is worth the tech progress here, you know, maybe it is, ChatGPT is like what I think literally the fastest growing consumer app of all time, or maybe it's not, you know, you could have meaningful debate on whether these AI companies with something like ChatGPT should be taken a little bit more carefully because you know we have AIs driving some vulnerable children to suicide or whatever or whether we're like look this is the price of going into the modern world, you know, I'm not going to weigh in on that, I have my own views but I have bigger fish to fry and I try to stick to those.
But this process where you try something and see if it works, it works fine with something like ChatGPT, but if you have an idea for like and this is how we're going to keep a super intelligence in check. This is how the AI is going to still stay in check even after it could escape from the lab and kill us if it tried to.
That idea needs to work on the first try, right? There is a difference. There's a fundamental difference between, you know, the scenario where you're testing an AI and if the test goes wrong, you're fine and you get to try again and the scenario where you're testing an AI and if the test goes wrong, you're dead, right? And this is a little bit like one analogy we might use here is it's a little bit like launching a space probe. You know, the Earth is just a different environment than space. And you can set up all sorts of laboratory conditions where you try to simulate a vacuum. You try to put it underwater to simulate a low-gravity environment. You know, you put it in a centrifuge to simulate a low gravity environment in a different way, right? You can construct all these scenarios, all these ways of testing on Earth whether your ideas work. But space is a different environment. It's not exactly like the environment of training. And often, especially early in the development of space probes, space probes would fail in space, right? Things are a lot harder to get right when the training environment where you're building it is very different from the deployment environment where it matters, right? And that's in a situation where people were crafting the spacecrafts by hand, where they had, you know, mature engineering sciences where they, you know, knew the stress and strain of every material they were working with and, you know, had careful blueprints and were crafting the dang things, right? Even then, even when you're building something that doesn't need to be hyper complicated, even when you sort of have a very good understanding of how space differs from planet side, a lot of these space probes fail, right? They skip off the Martian atmosphere or they, you know, there's a fuel leak and something explodes, right? With an AI, with a super intelligence there is a difference between when it's in the lab and you give it a fake opportunity to escape and when it sees a real opportunity to escape, right? There's a difference between when you tell it it has some weapon and see whether it uses the weapon or uses, you know, presses the imaginary use the weapon button versus when it invents, when it's smart enough to invent its own plan that it can see the details of that plan and why it would work. Those are different cognitive environments, right? And by the way, in the lab, you know, you can set up certain conditions where the AI does try and press the imaginary button, right? It's not like that. The only thing these things are lacking is the intelligence to actually succeed at their plans. It's not like everything's currently going great in the situations where we do try and test things in advance today. But the thing that makes it fundamentally difficult is that the place where the AI needs to work, it needs to work on the first try. There is a context switch. There is a place where you need to run the AI out of distribution for the first time. The first time it's in the new distribution of if something goes wrong, the AI just takes control of the world. And that is a change we need to nail on the first try. And that is a change where modern techniques of like try it and see aren't up to the task. This try and see approach it already leads to all these squirrely issues. It already leads to all this stuff in the edge cases. We already see AIs, you know, trying to resist shutdown sometime in lab conditions. We already see AIs trying to escape sometime in lab conditions. We already see AIs trying to kill the user sometime in lab conditions. We already see people try some things and them not quite work, right?
Mhm. Maybe they'll iterate around with these dumb AIs, but then and maybe they'll construct, you know, dozens of different patches and techniques that keep these things on track when they're relatively dumb. Some of those are going to break down in the new distribution. Some of those are going to break down when the AI actually has the ability to get a lot smarter or actually escape and deal with the world. And you need to understand all of
The ways that it's going to change. You need to deal with all of them, or the whole species dies and we get no retries. That's the extremely hard scientific problem. The problem is not terribly harder than other problems. It's just that normally you can learn by trial and error and we can't learn by trial and error in the critical case here.
But what if we are very careful? What if we do not connect the superintelligences that we're training to the internet? What if we give it very limited processing power to run? What if we monitor it with a bunch of AIs with much more processing power that are almost as smart as the AI we're trying to develop? We might approach this problem carefully. I'm not claiming that we are currently on track to do so, but do you think there's a version of the world in which we approach this carefully and we succeed?
I mean, yeah, the real answer to like what if we do all this careful stuff is that we just aren't anywhere near on track to do that. You know, like back 10 years ago, people would say, 'No one would ever be dumb enough to connect a general intelligence to the internet.' And you know, how would it even get out, you know? And these days that proves to be a lie or at least a dramatic misunderstanding of human nature. And so, you know, in real life, humanity just isn't up to the task in that way. In real life, the people at the labs are just not going to do this in the same fashion and so they just need to be shut down. If you know, we can separately talk about what would go wrong if you really really tried to be careful here. I think a lot of the issue is that people don't tend to quite understand what carefulness entails, right? People who are saying, 'Ah, we'll have all sorts of AIs. They're monitoring other AIs.' Or people saying, 'Ah, it's fine. We'll have the AIs, you know, care about truth.' Or people saying, 'Ah, it's fine. The AIs will happen to be submissive.' You know, I mean, these aren't all... some of these plans are a little bit smarter than the others, but for one thing, it doesn't matter if one group is trying to do things in a sane way. The sort of most reckless group is the one that can kill you. You know, you need nobody to try a really dumb way of doing this, which is a difficult coordination problem. But secondly, even plans on the sort of... you know, if you have a spectrum from like it'll just happen to be submissive to like we'll have a bunch of AIs monitoring the AIs. There's a big difference between a plan like, 'Don't worry, it'll be fine. We'll have the AIs monitoring the AIs' and what people were doing to make the space probes work.
You know, this isn't what a good plan looks like, but it's not like in the genre of a detailed specification that can be critiqued by engineers who understand their medium.
Right. We're sort of like not in the ballpark. I can separately talk about like why there's not a ton of hope in this, you know, AI read the AI's minds. Perhaps we go into that later.
For now, I want to zoom in on... In the book, you have a bunch of examples from the field of computer security, which tell us something about how difficult it is, even if we're being super careful, to create a secure system, to create a system where there are no weird edge cases where things fall apart. Perhaps you can talk about what we know about creating a bulletproof system.
Yeah. So, computer security is famously a very hard problem and making a computer program that is secure against attackers is famously much more difficult than making a computer system that works in the usual case. And sort of glossing why that's true, it's because someone who's programming a computer to work in the usual case sort of just needs it to work in the usual case, whereas someone who's trying to program a computer to be secure needs it to work in every case. And why does it need to work in every case for computer security? Because you have, you know, in computer security an adversary can find the single worst case for your situation and pluck that out of the possibility space. The example we use in the book here is, you know, some listeners will have heard of buffer overflow attacks where a buffer overflow attack is roughly a programmer had some input like some place to put your username and maybe the programmer never imagined that you'd put in a username that was more than 100 characters long, you know, 100 letters long basically. And so they just allocated space for 100 letters of memory, but they didn't check that the actual input was less than 100 letters and just write the input into that memory. Now what'll happen is if someone puts in something that's more than 100 letters long, some of those letters will overwrite parts of memory that the user was not supposed to be able to write to. You know, and now almost always what'll happen here is you'll write garbage into other memory locations and that'll crash the computer, right? Or somehow crash the program. And if you were trying to design tests that figure out what's going on, you might be like, 'Oh, well, there's some cases where things crash, but whatever. Who really cares? You can't, you know, an adversary can't take over a crashed computer, right?' And you could test, you know, thousands or millions or billions of cases and see that they're all crashes, you know, and you're like, fine, you know, it's less than one in a billion chance that there's anything here that isn't a crash, right? But that's not actually quite how it works because an adversary can read your code, understand what's going on there, understand that they get to write over memory that they're not supposed to write over. And then they can very very carefully choose those excess characters to write exactly what they want in a memory in a way that won't cause the computer to crash, but will cause the computer to start doing something else of their choosing. Right? And you're not going to find that unless you understand what's going on. You know, random testing isn't going to come up with this case. But you know, the programmer sort of makes one seemingly relatively innocent mistake. You know, usually people aren't going to put in a long username. The programmer just forgot to check whether the thing was too long and cut it off. And this is a very very common type of attack in real life. You know, not exactly with usernames, but this sort of thing happens sort of all over the place. Like in some sense, computer security is about the programmers trying to nail down... the programmers in some sense have full control over the computer. They have full control over the code that runs on the computer and they are trying to make the computer useful while fully pinning down all of the sort of things you can do with a computer to a limited set of possibilities, right? And this turns out to be basically beyond human computer security experts' abilities. You know, if you talk to computer security professionals and you say, 'Make my computer... we need this useful property of a computer that can be used in this internet-connected useful way or whatever and I need it to be completely secure such that nobody can make it do something I don't want.' They will basically say that's a pipe dream. Your options are spending enough money that your cousin can't break into it and spending enough money that a small nation state can't break into it and you probably can't spend enough money that a large nation state can't break into it. That's probably just not a thing that can be done. Right? This teaches us a lesson. You know, I'm not doing it complete justice here. People will just have to read the book, I guess, to get the fuller version. But this teaches us a lesson about how hard it is to sort of constrain the space of possibilities when there is something that's looking through all of the possibilities including ones you didn't think of and trying to find the ones that are optimized. Right? And the way that this applies to AI is not that the AI is an adversary. It's, you know, if you've created a super intelligent adversary, you've already lost. The way to deal with that is to not create the super intelligent adversary in the first case. No, the way that this applies to AI is you can think of the adversary in computer security as a force that is looking for the weird cases in your code that technically let them do something that is very extreme.
Right. You know, there maybe there's some web page where all it does is you type in your username and it records that in some database and then it shows you some information or whatever and most people look at that and they see a way to get their username in the database and a little bit of information. And a computer hacker looks at that and sees a computer program that actually lets them take full control of a machine.
