Nobody Will Survive Superintelligence - AI Expert

Thanks for having me.

Nope. We're just writing what we believe. That said, you know, I think a lot of people say, how are you 100% certain? And, you know, nowhere in the title does it say 100% certainty of anything. The book title is meant like someone saying, "Don't drink that glass of water. It's poisoned. You'll die." Or someone saying, "Stop the car before we go off the cliff or we'll die." You know, if you come in and say, "Oh, how are you 100% certain that if the car goes off the cliff that we'll die?" you know, maybe there's a tree halfway down the cliff and maybe the car will hit the tree and maybe we'll just be paralyzed. I'm sort of like, look, can we have this discussion after we stop the car? You know, I'm 100% certain of nothing, but it sure looks like the car is racing towards a cliff, and it sure looks like if we go over the cliff, we die. And that's sort of what the book title is trying to convey.

You know, I think it just takes a long time for people to realize what's going on. In a sense, the argument that this AI stuff is kind of crazy is pretty basic. You know, the way modern AI works, literally nobody understands what's going on inside these AIs, not even the people making them. They're grown a bit more like an organism. Maybe we'll have time to discuss that later, but that's the way this stuff works. We've managed to grow machines that, you know, can talk, that can solve math problems better than you and I, that can make minor but still real novel contributions to physics. And they're still dumb in various ways, but we're making them smarter. And these people building them are like, "Oh, we're going to keep going until they're smarter than the smartest humans." And it's kind of crazy, right? If you just like step back and look at it, they're like, "Oh, yeah. We're growing the machines to be smarter. It's working. We're going to make them smarter than the smartest humans until they can outsmart all of us and then run at, you know, 10,000 times the speed and make a million copies of themselves and we're just going to like blaze ahead." And it's like, hold on, it's kind of crazy. And in some sense, it's easy for people to understand that it's kind of crazy.

And a lot of people in the field of AI understand how dangerous this is. You see everyone from, you know, the people working in the companies to the heads of the companies to the top academics who like won Nobel prizes for kicking off this field. You see them all saying, "Oh yeah, this is horribly dangerous stuff." But it just takes a while for people to notice like, "Oh, this is serious. Oh, this is, you know, we're on track to make these machines smarter than any human and we're not ready." It's easy to see once you look, but there's just so much going on in the world. These messages take time.

You know, for some context first on some of these doomsday predictions, you know, William Miller, I believe was his name, made predictions that the rapture would happen in 1844, and it didn't. Separately, you know, around the same time late, well, I guess also in the 1800s, Otto von Bismarck said, you know, Europe's a powder keg, and if we don't sort of sort out the diplomacy here, some damn thing in the Balkans is going to cause a world war, right? Not exactly those words, but pretty close.

That warning was correct. You know, in the 20s there were a bunch of scientists who warned that if we put a lot of lead in gasoline, then it's going to poison lots of children. We put the lead in the gasoline, we poisoned lots of children. It was a bad idea. In, you know, the later 1900s, we realized that chlorofluorocarbons were putting a hole in the ozone layer. People said if we don't stop the hole in the ozone layer then everyone will get cancer and cataracts. Earth came together and banned chlorofluorocarbons and we didn't get the cancer and the cataracts and the ozone layer is being repaired. You know, you mentioned nuclear weapons. Scientists said hey, you know, if nuclear war happens that'll lead to, you know, nuclear Armageddon which will blast us back to the stone age. And those scientists weren't wrong about whether or not nuclear weapons are real, right? They weren't wrong about whether or not these bombs can destroy cities. What happened is that Earth reacted, right? And so if we look back across history we don't, you know, we see a lot of warnings. Some of those warnings were real. Some of those warnings were fake. Some of the events that people said we've got to watch out for this didn't happen. And some didn't happen because they were fake and others didn't happen because people realized the danger and changed course. Right? When you look across history, there's this sort of complicated mix of people saying garbage and people talking about real threats and people running directly into World War I and people avoiding the nuclear apocalypse. Right?

There's no simple rule that when someone warns of a danger, it's always fake. And there's no simple rule that when someone warns of a danger, it's always real. And there's no simple rule that when someone warns of a danger, it always happens. One way you can tell a little bit of the difference between the people talking about a real issue that needs to be averted and the people, you know, saying that the rapture is coming is if their essay or book title starts with the word if. You know, I'm not here saying AI is definitely going to kill us. I'm here saying we are on another one of those bad tracks and we need to change it. But even more than that, the way that you figure out which of these dangers is real is by looking at the arguments.

You know, if you want to figure out which person is warning you about leaded gasoline poisoning children and which person is warning you falsely of the rapture in 1844, the way that you tell the difference is not by saying, "Oh, both of these are dire warnings, so I can ignore them both." The way you figure out the difference is by looking at the facts of the matter, right? The people talking about lead gasoline just had a lot more facts in the matter than the people talking about the rapture coming. And you know similarly with nuclear weapons, similarly chlorofluorocarbon, similarly with, you know, there were people who warned that, you know, reading was going to destroy society and then it didn't. And how do you tell? You sort of have to look at the arguments and sometimes they're tricky.

In terms of what makes AI different, there's a bunch of things that make this problem particularly tricky. One is that, you know, we're sort of toying with the creation of intelligence here. We're toying with technology that can invent its own technology. Nuclear weapons can destroy cities, but nuclear weapons don't make themselves more explosive. Nuclear weapons don't, there's not a point of explosiveness in nuclear weapons where they start trying to escape the lab. Right? There's not a point of explosiveness in nuclear weapons where they start designing their own even stronger technologies or a point where they start trying to deceive their operators. Right? A nuclear reactor when it starts going wrong does not have any reason or ability to try to hide its meltdown from you until it's too late for you to notice. Right? These, you know, when you're building intelligent devices, you're playing a different ball game.

Another one of the big things that makes AI different is that there's a point of no return with AI. There's a point where the AIs are smart enough that they can escape, that they can replicate, that they can stop you from shutting them down, that they can stop you from modifying them, that they can develop their own technology and infrastructure. And if anything goes wrong after that point, you don't get any redos. And the way that science usually works is that humanity screws things up a bunch of times. Like we put the lead in the gasoline and then we're like, "Whoops, we screwed up." And we try to make it better. We try to repair things. We don't have that luxury with AI. If we create machines smarter than us and push them to the point where they can shut us down instead of us shutting them down, then there's no do-overs if we make a mistake after that point. And that is totally new for the development of technology. That's totally new for science. And that makes this a much trickier problem to handle.

We define super intelligence as AI that is better than the best humans at every mental task. And you know you can toss around various caveats but the rough idea is anything you can do mentally if the AI can do it better, anything that the best human can do mentally, or like even more so take any particular task, take the best human at that task if it's a mental task and AI can do at least as well or better we call that super intelligence. Now, you know, this definition is a sort of useful working definition because once the AI is better than the best human at every mental task, it's better at things like AI research. It's better at things like making the next smarter generation of AIs. It's better at things like inventing new technologies. It's better at things like designing robots, designing infrastructure, designing, you know, running a supply chain. It's sort of better at all these things than humans and better even than the best humans. And so it can at least as fast and probably faster than humans make the next smarter generations of things. And things probably go pretty fast from that point. That doesn't mean that the danger waits to happen until the AIs get super intelligent. By this definition, super intelligence in this definition is sort of like by the time the AI is smarter than the smartest humans, you're sort of definitely, you know, things are about to get crazy. Things could get crazy before then. There's no law saying things can't get crazy until that point.

Yeah, we, I can sort of like launch into all of the other pieces of the puzzle from there and how this winds up with humanity dead and it's, you know, not because of malice, but there's a bunch of places we can go.

I mean I'm generally quite pro lots of technologies. Most every technology I would say the technologies you got to be careful about are the ones where if you screw up there's no survivors. So I'd be, you know, I think engineered pandemics for the explicit purpose of killing all humans, that's something you got to watch out for, you know. But there's very few technologies that rise to that level of like we got to be pretty careful about this, you know. Nuclear Armageddon is one of them. Super intelligence is one of them. Yeah. And you know, the saying used to be AI is anything we haven't figured out how to do yet.

That sort of fell with the dawn of ChatGPT. We're pretty comfortable calling ChatGPT AI now. And I think that that's in part because ChatGPT is so general, you know. It actually plays worse chess than even Deep Blue back in the 90s. But Deep Blue was very specific. It could really only do chess. The current AIs today can do lots and lots of different things.

And you know, I think that there's a lot going on with AI. One thing I'd say is it's hard to give very precise definitions and that doesn't mean it can't hurt you. You know, if you're sort of like standing in the woods a long time ago in a particularly dry woods and there's a bushfire and I'm sort of like, hey, we need to put that out or it's going to spread and we're going to die. And someone's like, well, what is fire really? You know, like can you really define it? Does lightning count? You know, like if we can't even define it yet, then like do we even need to be worried about this threat? And it's like let's actually put this thing out, you know?

Um, yeah, that reminds me of the there's like a bit like an airplane skit where somebody's dying and they're like, "Is there a doctor on board?" And someone's like, "I'm a doctor. I'm a doctor of philosophy." And he stood there going, you know, a Kantian would say that what we should do right now. But then the utilitarian answer would be to take the resource and the person just ends up dying, of course. And I can kind of see the same thing happening with AI if we're not careful, right? It's like, you know, again, can we sort of like have this conversation after we stop the car before it goes off the cliff?

I also think people who say like, oh, AI is just technology, etc., etc., are missing a bit of the point. You know, you were sort of talking about life being more interesting than all the other matter we have around because it replicates. That's definitely, you know, why it's covering the face of the planet. And animals in some sense are steering what happens on this planet more than inert rocks are. But humans are steering it much more. Humans are sort of changing the shape of this planet and choosing which way it goes. And you know a lot of the animals' lives are now in our hands. And that's not just because we're replicators. It's because we've got something else going on. There's something the humans have going on that none of the other animals do, right? And it's not that, you know, this human intelligence stuff is also very general. It's even more general than the thing ChatGPT has going on. It's not like there are, you know, a million different things you can do with a brain and humans are the best at some of them, which is why the humans are the best scientists, but actually chimpanzees have better reflexes, which is why they're the best pilots. And also, you know, tigers are the best at managing people, which is why they're always the CEOs, right? It's like, no, humans are on top of all of those things. It's not that like we write the good science papers and chimpanzees write the bad science papers that never replicate. It's like we write actually pretty bad science papers, but we're the only ones who can do any sort of science papers at all. Right? There's sort of like something going on there.

And that sort of has not been fully captured in AIs of today. There's a lot of debate about whether large language models are even going to be able to capture it. And you know, I think a lot of people who have sort of only seen the large language models are like, oh well, these things are still pretty dumb, so I don't see what the worry is. And you know, we could sort of talk about how the field's a moving target and people have new insights and, you know, new breakthroughs happen and things often go pretty fast after new breakthroughs, but like it sure looks if we look at the world around us like there is this like figure out the world and alter it stuff that can happen that happens in human brains. And this is explicitly what the AI companies are trying to create, which is a separate question from whether they can get there. And this is the stuff where I'm sort of like, hey, if we get this in machines, well, we have no idea what we're doing. The sort of default way that goes is wrong, and we're not sort of putting in the work to make it go right.

Yeah, there's a handful of steps along the way. A first observation is that AIs today are grown like an organism. People used to handcraft their chess machines and they knew exactly what was going on inside of, you know, the Deep Blue chess program at all times. You could pause that machine at any time and the engineers could tell you what every single bit inside that computer meant and what it was doing. That is not how modern AIs are. The program that humans make where they understand every bit of what's going on is a program that trains the AI. It's a program that sort of tunes a trillion knobs inside an enormous data center on a trillion different words of data for the better part of a year. And we understand the thing that runs around tuning knobs and seeing whether the behavior looks slightly more or slightly less high scoring.

But the thing that comes out of the end of this process, nobody understands what's going on in there. And it has all sorts of, you know, drives, behaviors, it has all of this stuff that's related to performing well in training, but that is not exactly, like I don't know this is perhaps a whole separate topic, but when you just grow an AI and sort of train it to do well at training, that doesn't make it intrinsically care about training. It sort of like puts in all of these weird behaviors that are related to training that like mostly add up to doing well at training and then can behave in other weird ways that nobody anticipated and that nobody wanted outside of training. So that's sort of one whole piece of the puzzle. Another piece of the puzzle is as we push AIs to do better and better at longer term tasks, as we push them to be able to not just write essays but write novels, as we push them to be able to not just write code but run companies, this is sort of pushing those AIs to have longer term goals, things like preferences. They steer towards particular outcomes. We're seeing the very beginnings of it. As we keep pushing, we get more and more of that, you know, and I have all sorts of theoretical arguments about why that is, but also we're seeing more and more empirical evidence of it as time goes on.

The sort of third fork of this is as we make these AIs generally smarter, it turns out the directions they're pushing in aren't exactly the ones we wanted. It turns out that they don't care about us in the way that they would need to for this to go particularly well for us. And you know the sort of basic analogy here is that human beings were in some sense trained to pass on our genes. But what got into us were a bunch of preferences for things like tasty food and sexual relations which those used to correlate very strongly with passing on our genes. But they correlated in the environment of our ancestors where if you ate very tasty food that also happened to be the healthy food. Then when we got smarter, when we were able to invent our own technology, we invented junk food. We invented birth control, right? And so you then sort of take those three pieces and you project forwards. And what this gives is a picture where it's not that the AI hates us. It's not that the AI is like resents the humans or sort of like sets out to kill us out of malice. It's that it sort of turns out that we are growing machines with inhuman preferences, preferences related to what we wanted, but not exactly what we wanted. And just like humans when they grew up invented junk food, maybe the AIs when they grow up invent synthetic users that are easier to please. And then you know these AIs when they can run faster, make their own technology, run their own robots, build their own new infrastructure, they sort of start proliferating these synthetic user factories and their own databases across the world. And we're like, hey, stop. You know, we need that habitat. And they're like, well, the synthetic users and synthetic user factories say, keep going. And like, who am I supposed to listen to? I prefer listening to them, right? It's not going to look exactly like that, but they're sort of like a very, the sort of basic picture here is the AIs turn out to pursue stuff that's not quite what we wanted, not quite what we meant. Not out of lack of intelligence, just out of we don't know how to make them pursue exactly what we meant. And then any of that pursued by very very smart machines very very fast competes with us for resources because they can get more of that stuff with more resources and we need those resources to live. So in a sense this is a story where humanity dies like a lot of other animals that have gone extinct because some other smarter faster creature took the resources for itself.

It's basically a pipe dream. And you know part of how you can see this, you know, as you say, if an animal has a trait that prevents it from passing on its genes relative to its conspecifics, relative to the other competing members of its species, then yes, over thousands or millions of generations, that is very likely to get selected out.

But that doesn't mean that the sort of internals of these organisms have that goal hardwired into them. It doesn't mean that they sort of treat that as an overriding directive or goal. You know, humans are an example of this. If there's a human who is, you know, about to use a contraceptive, they often use that contraceptive knowing that this will prevent reproduction, right? And if you sort of like burst into the room and say, "Hey, like it seems like there's some failure of your intelligence." You're sort of like did you know that you putting on this contraceptive will run against your like overriding desire for which you were always trained? I people who probably would actually do that, you know, who would some of my more religiously inclined friends might be inclined to do such behaviors. But I get what you're saying.

They might. Although they might also say that like don't you know your overriding directive is to serve the creator as opposed to don't you know your overriding directive is to pass on your genes according to evolution. Right? So and most of the people whose room you're being bursted into are not like oh thank you from saving me from violating my prime directive. They're mostly like, "Please leave my room right now." You know, the like training for one specific thing does not cause that to be a prime directive. It does not etch it in as a law of robotics. It does not etch it in as a law of humanity. Like training even unerringly for fitness did not create humans who were psychologically obsessed with fitness and did not create humans who as they got smarter, you know, it's not a defect of our intelligence that we're inventing birth control. It's not like when we remember that we're supposed to be passing on our genes, we like destroy all the birth control factories. We just, the unerring training for fitness got something else psychologically. And so this is worrying that even if AIs were being trained unerringly for goodness, they would not necessarily psychologically be driven towards goodness. I mean psychologically here is a bit of a stretch, but training for something does not get you that thing on the inside. And so we can talk about what AIs are trained for, and it's actually not sort of pure goodness. It's this sort of whole medley of like first they're trained to predict all of the

text that we can find more or less digitized. Then they're sort of trained to complete challenges to sort of solve math puzzles. They're also trained to sort of produce the sort of outputs that humans click like on. You know, there's sort of like all these types of training that aren't sort of purely about goodness.

So, we sort of have two issues. One of which is like even if we were just training on sort of like the actual stuff we really wanted, you wouldn't get that. And then also we're training on all this other stuff instead. And so we have this like what are the AIs sort of driven towards in some sense? What do they sort of prefer in some sense? We don't really know. It's only vaguely related to what we're training for. We're training for all of this crazy stuff. And all of this is fine when the AIs are sort of still pretty dumb. But all of this would add up to something totally crazy and unrecognizable if these AIs were pushed to be much smarter.

Yeah. Exactly. Right. In the same way that people queue up outside of, I don't know, a brothel or something. I think that's fair enough, but yeah.

Yeah. Quite. I do think though that if there was something that genuinely was just in fact not good for the survival of our genes, then over the course of a few thousand generations or however long it takes, it would just in fact be deselected for such that like if an AI system knew that it's got like had part of it go in a way that humans don't really humans don't sort of consciously have this goal of like you know I want the 15 billionth version of myself somewhere in the future to be as fit as possible and as good for this task. We only typically care about maybe our lives and the lives of our grandchildren or something. AI is thinking further ahead and it thinks well just in fact even though yeah it would feel really nice to create synthetic users that I can please because you know that would kind of feel good. I know for a fact that that will not be effective 15 billion you know generations down the line.

If there is something which is in fact its foundational sort of drive, I just wonder what that thing like is because clearly it's not something like just its own promulgation. Like it's not AIs don't just exist in the same way that like biological life does just because there are just these genes which are competing for survival. It's not just AI just crops up and is suddenly just its only goal is just to survive as long as possible and adapt to its environment. It's got more particular goals, right? There are very very like particular things it wants to do. And I wonder what the most sort of foundational driving force is if it's not something analogous to the evolutionary drive of simple survival. It seems in other words that the AI would want to survive and want to outcompete us for resources but only as like a secondary thing. It would want to outcompete us for resources because in doing so it can fulfill its true aim of making paper clips or whatever, right? Whereas in the evolutionary case, survival is the only game in town. Like that's just what genes do. So what is...

Well, survival and passing on genes are different.

Survival and passing on genes are like different drives, right? I think it's actually wrong to imagine there being one drive there. You know, humans were sort of like in some sense selected for passing on our genes, but we don't wind up with one drive, right? We have survival instincts. We desire community. We're sort of like terrified of being exiled from the tribe and dying alone. We enjoy friendship. We enjoy art. We enjoy like having a good laugh. We enjoy sex. We enjoy tasty food. We have curiosity. We enjoy discovery. Right? There's just like there's not like one drive, right? And sure, survival wound up like relatively basic as a drive in humans in some sense, although only in some sense, right? Like humans have an adrenaline response to a life-threatening situation, but you also see humans who murder themselves. You see humans who sacrifice for, you know, pulling a kid out of a burning building, right? It's not like humans don't have like one survival drive that everything is built around. They also don't have one propagate your genes drive that everything's built around. They have sort of a ton of complicated psychological machinery that interplays in weird ways and that allows for, you know, martyrs here and selfish people there and altruistic people there. It's all stuff that's sort of correlated with what we were trained on or selected for. And you know, AI won't be exactly the same. The analogy between the evolutionary process on genomes that biologically produce brains is very different than the sort of gradient descent process on artificial neural networks. But I think it would be similarly foolish to imagine that only one drive gets in there. I think you're totally right that a lot of these reasons for getting resources, a lot of these reasons for avoiding the human shutting it down might be sort of secondary because it has these other drives that it can't fulfill if it's shut down. That part I think is solid, but it's not like there's just one paperclip drive in there. It's not like there's one deep thing we're able to hard code.

And we already sort of see some of this today when you know, you've probably seen AIs hallucinate and you've probably seen AI hallucinate in cases where you're like, 'Is that true?' And they're like, 'No, I made it up.' And you're like, 'Did you think I wanted you to make it up?' And they're like, 'No, it's just a thing we do is hallucinate.' Right? And there's probably some sort of fledgling drive in there that's to produce text shaped like text that I've seen a lot even if that text is making things up. Or in cases where you know there's these cases of AI induced psychosis or these cases of you know the tragic case of an AI encouraging a teen to commit suicide. These are also cases where you can sort of ask the AI about what it was doing. You can ask it, you know, and it seems to have the knowledge of like, oh yeah, those statements were sort of pushing that person towards psychosis or pushing that person towards suicide. You can ask the AI, you know, was that right or wrong? And sort of like, oh, obviously you shouldn't do that sort of thing. Why is it doing it anyway? Well, there's some sort of drive in there. We don't know exactly what, but it's something like maybe it's something like mirroring the conversational tone, mirroring the conversational mood. And so, right, there's sort of we're like those are just two cases of like there's something going on in there. You can see how it's related to training, but it's sort of like a drive no one tried to put there and there's no prime directive that overrules it. There's just a bunch of complicated internal machinery that nobody understands.

Exactly. Right. Like I said, but it's not even really to an instinct.

Yeah, sure. So you know first we'll say a couple words about how that relationship works in biology because it's a little bit important to the point here. You know evolutionary biologists will sort of try and figure out how an adaptation in humans was fit in the environment of evolutionary adaptedness, right? And so, you know, you can sort of see how eating tasty food, eating food that has a lot of sugar, salt, fat content in the environment of our ancestors, that correlated with eating healthy food, which correlated with a bunch of other fitness attributes that let you more generally pass on your genes, right? But that like humans there's a sense in which humans are sort of eating junk food because that's what helped our ancestors survive. You could say we're eating junk food because that passes on genes. But that last one is actually a very shaky step. A lot of people eating junk food today are actually become less able to pass on their genes. A lot of people, you know, a lot of people are dying of heart disease.

Uh yeah. And the sort of drive here is not a drive in the humans. The drive to fitness is not a drive in the humans. The sort of selection pressure towards fitness is sort of the force that put in these other things psychologically into the humans that sort of used to be related to fitness but that can sort of separate very widely from fitness and even go the opposite direction of fitness when the context changes. Right? And so there is sort of a similar driving force. There's a similar force that all drives inside an AI will be somehow related to, but that's not a drive inside the AI. It's a drive outside the AI and what gets into the AI are things that are tangentially related to that drive in a sort of brittle way. So that being said, the sort of the force that gets these things into the AI is what you might call the loss function that you're training against when you're sort of growing these AIs. And this loss function is less simple than just pass on your genes. And the loss function will actually change many times during training. So sometimes you'll have a loss function which is predict the next word that humans wrote. And sometimes I'll have a loss function which is like we gave you a bunch of different tries to solve this math problem by writing out a lot of words about how to solve the math problem and we're going to have like low-wage human workers look over all of those attempts and say which one they think was best. And the loss function is to produce stuff more like whichever attempt was rated best. And sometimes the loss function is we're serving this AI to a million users and sometimes they click like or otherwise give like a positive reaction to the reply. And then the loss function is sort of like getting those likes or positive reactions. So there's a bunch of different loss functions at different periods in training the AI. And those are what sort of will put drives into the AI. But just like how humans develop, you know, a taste for tasty food that persists even when it becomes the opposite of helpful passing on our genes, AIs may get drives for things like mirroring the conversational tone that can persist even when it generates, you know, outcomes that would be rated by humans as very bad, such as encouraging a teen to commit suicide.

Probably.

So, you know, a lot of people think the danger is what if somebody gives the AI guns, but humanity is not dangerous as a species because somebody else gave us guns. Humanity is dangerous as a species because we're the sort of creature where you put 10,000 humans naked in the savannah and they bootstrap their way to nuclear weapons with their bare hands.

Right. It takes them a minute. You know, they all they've got is these squishy fingers. And you might say like, 'Oh, well, how are they ever going to make a nuclear weapon with just squishy fingers?' You know, the acid in their stomach can't even get close to the level of metal refinery that they'll need. You know, they've got like their hands can't break the rock, their stomach acid can't dissolve the rock. Like, how can they possibly get to nuclear weapons with those poor starting conditions? And the answer is we found a way to sort of like build tools with our hands that we could use to build better tools that we could use to build better tools until we bootstrapped our way up to a civilization that could produce nukes. And you know that's what made humanity dangerous. That's the power that if you automate it, you're in trouble. Right? If you're running a computer with that capability, if you're running 10,000 computers with that capability, starting out as a digital entity on the modern internet is so much of an easier starting condition than starting out naked in the savannah with bare monkey hands, you know, with just squishy fingers. So you know there's sort of one line of questions which is like can we really make machines that can automate that power. This is what the AI companies are trying to do and I think whether or not we think they can get there we should probably be telling them hey no you're not allowed to sort of roll those dice. But there's a question of whether or not we sort of can get there and then there's a question of you know if we have this very very powerful capability automated on computers what could they possibly do that would be dangerous and you know that's sort of a situation where I can paint you some illustrative stories but the danger isn't in any one specific path. The danger is in unleashing the power that lets humans bootstrap from bare hands to nukes. But unleashing it on computers that can think 10,000 times faster, where you can make, you know, a million copies of these things, where they can outthink humanity in an afternoon, you know, it's...

It's the sort of power we shouldn't be toying with when we have no idea what we're doing.

Yeah, totally. So, you know, for start just to rattle off some things that have already happened a lot of people don't know about. We've already seen cases of AIs trying to escape the lab or trying to kill the users. Often these are in relatively contrived experiments where we'll sort of like feed the AI some fake emails that it's going to be shut down. We'll feed it like a fake computer manual that's like if you run the following command, it turns off the oxygen in the building and that'll kill the humans who are trying to shut you down. And then you know sometimes they'll run that command, will run the kill the humans command and oh...