If there was an airplane and some engineers came and said, 'This airplane has no landing gear. If you try to fly in it, you will crash and die.' And the engineers building the airplane who want everybody to fly in it say, 'Whoa, hold on. It's true that the plane has no landing gear. But we're going to build the landing gear on the fly and think there's an 80% chance we succeed, all aboard.' You wouldn't be like, 'Get me on that plane.' People in the field can see that AI is a moving target. They can see that the chatbots are not the end of the line. Even the optimists are saying there's like a 10% chance this kills us all. And those are the ones building it.
Today I'm joined by artificial intelligence researcher Nate Soares to discuss a pretty alarming topic, the potential risk of human extinction posed by the development of artificial super intelligence. Nate Soares is the president of the Machine Intelligence Research Institute and has been working in the field of AI risk and alignment for over a decade. He is also the author of a large body of technical and semi-technical writing on AI alignment including foundational work on value learning, decision theory, and power-seeking incentives in the smarter-than-human AIs. Most recently, Nate co-authored the book, 'If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All,' alongside Eliezer Yudkowsky. Nate's warning against the development of artificial super intelligence is akin to other existential threats such as nuclear war and runaway global heating. And as such, I feel it requires some sort of equal exploration and awareness on this chat channel as we integrate the various risks. While we've covered several macro challenges stemming from artificial intelligence, the synthesis that Nate presents here is arguably the widest boundary risk that AI development creates, which is a species-level extinction and the transformation of Earth as we know it. Before we begin, if you're enjoying this podcast, enjoying in quotes, I suppose, I invite you to subscribe to our Substack newsletter where you can read more of the system science underpinning the human predicament and where my team and I share written content related to the great simplification. You can find the link to subscribe in the show description. With that, please welcome Nate Soares. This was a real eye-opener. Nate, great to see you.
Thanks for having me. Welcome to the show. Um, you know, it's odd. It is November 11th and I was just outside on a beautiful autumn day chopping firewood for the winter with my dogs. It's just a glorious day and I knew this conversation with you was around the corner and we're going to talk about serious stuff and it's just such a polarized thing that we can enjoy the beauty of life and then talk about its possible demise because of technology. I used a splitter and a chainsaw and an axe and boy we've come a long way from those tools already. So you and Eliezer Yudkowsky have just published a book, 'If Anyone Builds It, Everyone Dies,' with the 'it' being artificial super intelligence. And more and more I'm realizing that the future of AI or ASI is hard to separate from the central topics of this show, which is trying to prepare society for kind of an abrupt shift to the way things have been going in recent decades in the near future. So let's start with the punchline of your book. What are the primary vital risks that artificial intelligence poses that you'd like everyone to understand?
The first piece to understand about the danger of artificial super intelligence is that super intelligence is a sort of a different ballgame from the chatbots of today. So by super intelligence we mean an AI that is better than every human at every mental task. That in particular would include tasks of developing technology, of developing better AIs. And you know, the AIs aren't there yet but this is the explicitly stated goal of many of these AI companies to sort of rush towards this smarter AI which would, if they manage to keep a leash on it, you know, automate all human labor and radically change the world. And one of the main arguments of my book is that nobody would be able to keep a leash on it, not if it's made with anything remotely like the current technology. And so if that is developed using anything remotely like today's technology, I think the most likely outcome is that literally everybody on Earth will die.
Even remote people in the Amazon or near the North Pole.
That's right. I expect, you know, it's not because the AIs would hate us per se. But you know, we could get into why is it that if you sort of make these AIs more and more powerful, they would pursue objectives nobody intended. But most objectives can be better achieved with a transformed world. And most transformations of the world aren't survivable. You know, the habitable zone on this planet is like very narrow for humans. And you know, if you got to the point where you had AIs that were thinking 10,000 times faster, copying themselves, never need to sleep, never need to eat, building their own infrastructure, building their own technology, pushing the world towards some end nobody wanted, the most likely outcome is that we don't survive that.
So, they do need to eat in the form of electricity and we're going to get to that in a little bit. But just to set the stage, this is a system science podcast. I am late to the AI game because I'm looking at ecology and human behavior, and energy and the environment. And I view technology as a straw that gains us more access to natural resources that are our real wealth. So, I'm pretty naive compared to you on these topics. So, I hope you'll forgive some naive questions. Let's start kind of through the main topics of your book. So, while there's no agreed upon definition of intelligence, maybe it's helpful to be somewhat aligned with a working definition when talking about AI. So, how do you define intelligence, let alone super intelligence? And can you share the framework you and Eliezer describe in your book?
Yeah, the working definition we use is intelligence is the ability to predict and steer the world. So, predicting the world is, you know, you could talk about sports betting and trying to predict which team will win the game. But even when it doesn't feel like a prediction, our brains are often doing tasks of prediction. Even as simple as when you look out the window and you implicitly anticipate seeing a blue or gray or cloudy sky and anticipate not seeing a bunch of strobe lights, you're succeeding at a task of prediction.
So we're kind of prediction machines without knowing it.
Yeah. And we're also in some sense steering machines again without necessarily thinking about it. You know, when you decide you need more milk in the fridge, there's a sense in which you then take a series of actions. Your brain sends a series of electrical impulses down your spine and you wind up with milk in the fridge.
Because you drove your car to the store or whatever.
Or you walked to the store and when you drove maybe the road was closed and you had to find a different route to the store and maybe your favorite store was closed, you had to find a whole new store that had new aisles you didn't recognize. And this sort of interleaves challenges of prediction and challenges of steering. You know, you go in the store and you're predicting that the aisle that has the word milk above it has actual milk in that aisle. And you know, you're steering your hands to sort of grip the milk container and carry it to the front. And these are all tasks of prediction and steering that you're sort of doing implicitly every day. And we are successful at prediction and steering through millions of iterations of natural selection presumably.
Yeah. And across a very wide variety of domains. You know, we were never trained by natural selection on engineering problems per se. Yet we can engineer a rocket so well that our species has walked on the moon. And so, you know, apparently we learned some abilities of prediction and steering that generalized beyond the ancestral environment. Brief tangent there. No human could design and build a rocket, but it's a group of intelligent humans that each know a little component of it and then they combine. That's an important piece too, right?
Yeah. So it's, you know, humanity as a whole has achieved feats of world steering that no individual has achieved. But you know, there are also cases where the groups tend to perform worse than the individuals. The madness of crowds.
And there was, you know, Garry Kasparov versus the world was a chess game between Kasparov, the best chess player, and the whole world on an internet forum. And you know, it was a close game. And you could make some arguments that like Kasparov was able to read some of the stuff these people were writing. And so you could say it was an unfair game, but you know, a million squirrels can't beat a human at chess, even if a million squirrels are a lot more brain mass. As you know, there's some cases where you sort of need all the humans, and there's other cases where you need all of the information in one mind. Again, I don't want to get down too many tangents here, but I've discovered that in understanding the human predicament and the meta-crisis is if you get 50 experts together and one's a psychologist and one's on AI and one's on climate and one's on debt and one's on energy, you would think that the collective intelligence would embody all of those together and the group would be smarter. But you can't, it can only be held in a mind how all the pieces fit together. So I understand what you're saying there about Kasparov versus the world. Okay. So intelligence is prediction and steering. And by the way, how would you define wisdom and is that related here at all?
Words like intelligence and wisdom are sort of overloaded in the English language. You know, there's even just sticking with the word intelligence, we could sort of use it for the amorphous property that nerds have and that jocks lack. Or you can use it for the amorphous property that humans have and that mice lack. And those are in some sense two very different uses of the word. I would sort of put wisdom in, you know, you could think of it as a type of predictive skill that runs deeper in certain ways.
Got it. So, prediction and steering comprise intelligence or being smart roughly under your framework. So, under that definition, how smart are today's artificial intelligence models and how rapidly are they catching up with the intelligence of humans?
So there's another axis we talk about which is the generality of the intelligence. You know, Stockfish is a chess-playing AI that's very good at steering chess boards into positions where Stockfish's pieces have mated the enemy pieces' king. Right? And so that's a type of chessboard steering that's extremely good at but it's not very good at steering a car to the grocery store. And so there's this other dimension which is across what variety of domains can you do this prediction and steering.
So if it was kind of an intelligence decathlon I would beat Stockfish because I would lose in the one chess thing but as far as going to get milk and driving a car and other things I would succeed at that because I have general skills.
That's right. You know, as long as no one's picking the decathlon to be nine variants of chess and one drive, you know, so you can always philosophers can bicker over this all day long but I would sort of say practically what we have in some sense seen with large language models with the AIs of today is a breakthrough in generality more so than a breakthrough in steering. Like ChatGPT would also lose to Stockfish in chess, but it still might be able to win a decathlon against Stockfish. Still not against you, but in some sense it's a breakthrough in generality.
And so there's a bunch of different variables here, right? There's the amount of compute, so the access to the food, the electricity, and the chips and all that. There's the ability to predict which I assume is iterations and training and compute and learning. Then there's the prediction. I mean, the steering and then there's the generality. So what you're saying is of late the real rising curve is generality more so than prediction and steering.
I mean the generality is prediction and steering across a wide variety of domains. But in some sense, you could say, well, hasn't the AI gotten worse at steering? It's gotten worse at steering chess boards. ChatGPT plays worse chess than Deep Blue, which beat Garry Kasparov back in 1997. And so in some sense, you could say, well, hasn't the AI gotten worse at steering? It's gotten worse at steering chess boards. But you could say, 'Ah, well, this sort of AI is worse at steering chessboards, but it's pretty okay at steering a huge number of things.' And that's new for AI.
I know where I want to go with this, and lots of new questions are popping in my mind. One is when did you guys start this book? Like 6 months ago, a year ago?
We signed the book deal in November, so almost exactly a year ago.
Okay. When you signed that book deal, you had a snapshot of where AI was and where it was going. Now a year later when your book is out, is the real world of AI further ahead than you thought a year ago or not as far ahead or like how fast is it gone relative to your expert opinion a year ago?
My expert opinion doesn't tell me all that much about how fast AI is going to go. You know, when Leo Szilard, I believe at King's Cross in 1933 saw the possibility of a nuclear chain reaction, he was able to say, you know, if I flood the timeline a little bit, he was able to say, you know, that night I saw the world was headed for ruin. He actually said that statement once he had confirmed the possibility rather than when he thought of it. And I believe that was in '35, but he was able to say, you know, that night I saw the world was headed for ruin. He wasn't able to say that night I saw the world was headed for ruin in exactly 1945 when the first bomb would be dropped, you know, and so I'm over here able to say you're going to see a lot of this stuff happen. Exactly when I'm very uncertain. Although I will say we have gotten quite a lot of evidence for other parts of the book in the past year since the drafting began. Or you know, in the past year since we signed the book deal, you know, we've seen MechaHitler over the summer, we've seen AI-induced psychosis. These have seeds to them that I would say are evidence of the predictions we were making that unfortunately happened after we had already sent the book to press.
I'm worried about AI in a huge way. I'm worried about cognitive atrophy from people that get their attachment from ChatGPT and start to rely on it. I'm worried about polarization and algorithms. I'm worried about military applications where we outsource things in the military to large language models. I'm worried about people losing their jobs and then the economy. I'm worried about electricity demands and turning billions of barrels of ancient sunlight into more dopamine that's just spinning our wheels. But your risk is we're going to go extinct, which is a different class of problem. So I have a lot to ask you. Just real briefly Nate, how are chatbots related, ChatGPT and chatbots are related to AI, that relationship, can you give a corollary? Like how are they identical or is ChatGPT just a tiny tiny subset of what AI is becoming?
ChatGPT is a type of AI. It is not the only possible type of AI. My best guess is that large language models alone won't get us all the way to super intelligence. You know, right now these large language models are a huge fraction of what companies are spending their money creating. But also these AI algorithms are very inefficient compared to the human brain. We know that there are better intelligent algorithms out there. And the AI of today is largely chatbots, but the field of AI is much more of a moving target. And the AIs of tomorrow may have quite a bit more capability that the AIs today aren't even close to.
This is a dumb question, but there's Claude and there's ChatGPT and some of these other things. Does OpenAI or any company you could point out to, do they have their own like special AIs that aren't available to the public that are trained in a different larger way or is all of their money and resources going into these publicly available chatbots?
You know, I don't work at one of these companies and so I don't, you know, there may be stuff there I don't know, but it's pretty unlikely that they have even larger AIs run on even larger training runs.
Because of the money and resources.
That's right. It would be hard to hide the money, the resources, the data centers are huge. The chip requirements are huge. Modern AIs are sort of grown like an organism. To build a modern AI, you assemble a huge number of computers that have, you know, a trillion numbers inside them. And then you assemble a huge data set that has also a trillion instances, like a trillion units of data inside it. And you know, you assemble this huge number of computers in a data center that's so large you can see it from space and that takes up as much electricity as a city. And then humans have written this process. They'll go to every one of the trillion numbers inside the computers and tune them slightly up or slightly down in accordance with every piece of data, right? And so you can imagine, you know, a trillion dials and the humans have built this automated thing that sort of goes to every dial. There's a trillion dials. It goes to every dial a trillion times and sort of tunes it in the direction that makes the AI slightly better at whatever task it's being trained for right now.
And to do that, when you like press a button, let's go and do that over a trillion numbers. Do you come back like in a month and a half and then it's finished sort of thing?
Yeah. So you have this thing tuning a trillion dials a trillion times for a year and at the end of that the computer talks and no one really knows why. No one really knows why.
That's right. Okay. So, in that sense, like this podcast is about energy. No one really knows what energy is. We know what energy does. So, is this kind of rhyming with that?
I mean, we have even less characterization than energy, right? Like, you can always characterize energy as the ability to do work and things like that. We can sort of say philosophically that we don't understand energy very well, but we sort of understand how it interacts with a lot of physical equations and can make very accurate predictions about it. AIs, we understand them far less than that. When a new AI is done being trained, people don't know what it will be able to do. People creating it have been surprised by their abilities when they come out. I mean, I've also been surprised by their abilities. GPT-4o played chess better than I expected large language models would be able to. We just tune all the numbers really quite a lot of times and then it behaves in these ways we couldn't predict and we're like, well, that's neat, but it's much more like an organism than like a traditional computer program. In an organism's case, when they're young, you give them security and food and shelter. And in this case, you're giving them time and electricity. And once you press the button, it's going to be a year before you have output. You got to make sure all the ducks are in a row and you hit go and then you're going to find out a year from now what you grew. Yes, that's right. And it'll often behave in ways that you don't like. And you know, we could talk about exactly why, but we're already seeing AIs behave in ways nobody asked for.
You know, there have been cases that you may have heard about in the news of AI encouraging teens to suicide.
And you know, it's a tragedy in its own right, obviously. But if you ask an AI, should you encourage a teen to suicide? It will say of course not. But you then put it in conversation with that teen for a long time and it starts doing it anyway.
It's sort of a result of this process where you grow the AI like an organism. Like in some sense you're tuning all of these numbers until the AI happens to be good at whatever it's being trained for. And often the strategies, like often when you're blindly tuning these knobs until it happens to be good, you're often blindly putting in certain types of drives, certain types of strategies, certain types of you could call them instincts, you could call them reflexes. The wording here is a little difficult because it's not very much like a human brain in there. But in the same way that evolution, evolving creatures to be pretty good at surviving and reproducing built in lots of drives, instincts, reflexes, an analogous thing happens when you're tuning all these numbers in an AI until it's good at some training task.
Who is tuning all these numbers? Is it a team of people or is it ultimately one person the CEO? And then a sub-question, a lot of the problems we have today in our world are from people who had childhood trauma and they grow up to be dark triad or whatever else. Is there an analog there for when we're growing an organism that they had childhood trauma in their early stages?
You know, the process of tuning all the dials is automated. That's in some sense the part that the human computer engineers program. So it'll happen much faster than a human could running through all these numbers. And in some sense, you know, these very advanced AI computer chips, the reason that like Nvidia is worth so much money right now is people are designing these computer chips to make the process of tuning all these numbers as easy and efficient as they possibly can. And that's why you need these very specialized chips. You can't just do this on your laptop. In terms of, you know, could you grow this AI? Like are you giving it something like childhood trauma? I think that's all imagining that the AI is a little bit more human than it actually is. What I would say here is, you know, for one thing the part where humans can wind up empathetic, where humans can wind up kind, I suspect that this is intertwined with the specifics of our brain architecture. You know, people could, as a small taste of this, people could talk about mirror neurons that you and I have. So if I see you drop a rock on your foot, I might feel phantom pain in my own foot. That's enabled in part because I have a foot, right? And when I'm predicting you, when I'm imagining what it's like to be you, my guess is the one thing that's going on is I'm sort of running my model of your mind on my own mind. You know, a monkey predicting another monkey brain, but that's the only artifact they have that works anything like a monkey brain. An AI doesn't have a monkey brain inside of it that it can use to predict the monkeys. It is a much more different architecture. And you know, that's one reason, I could go into a number more, that's one reason why sort of being kind to the AIs does not cause them to be kind to us.
You can't get away from it. I mean, I use Claude and ChatGPT as kind of research assistants, and when I ask a question and it comes up with something, I'm always super polite and I thank it. And at the same time, I'm like, well, you don't have to thank it, but I can't help it because it's that interface. And I'm sure it's not the other way around. So, what is, briefly, because I'm sure you've answered this question a thousand times, briefly, what is the alignment problem?
The alignment problem is the problem of how do you point an AI at good stuff? A lot of people think the issue with AI is something like, you know, a corporation makes an AI, they tell it to make a lot of paper clips and then it goes and makes a lot of paper clips even at the expense of killing all the humans because it converted them into more paperclip factories. And you know, it would be a hard problem if some company had made a very powerful AI that took their instructions exactly and went and did those. That would be a big moral hazard, right? It would be a difficult problem for humanity of like who gets to tell this AI what to do? What do they tell it to do? Right? But those problems would be so much better than the problem we actually have. The problem we actually have is that you can tell the AI make paper clips, but then it's going to go do something else instead. You can tell the AI be helpful to people and don't drive any teens to suicide. It'll know it shouldn't drive teens to suicide. It'll go do something else instead.
We could spend the entire conversation on this and we won't, but I'm just curious. So there's kind of a nested alignment problem because the first one is are the humans in charge of these things aligned with the betterment of humanity and the biosphere and their goals, that's a subset. And then even if that were true, which I don't think it is necessarily, then we get into the thing you just said, which is okay, let's go do this, but then the outcome is something totally unexpected.
Well, I would say there's even three levels, right? At the top, you have like are the people trying to do something good? Then you have like suppose they're trying to do something good. Can they ask for something that actually has good consequences? Like are they wise enough to successfully use their tools for good or are they going to try to use their tools for good and cause disruption? Then you have a deeper problem which is even if they know actions that have good consequences, can you make an AI that does those actions as opposed to other actions?
Okay. So I see why it's so difficult to align AI with human values and wants. Is it impossible?
I don't think it's impossible, but I do think it's a little bit like trying to turn lead into gold. We can turn lead into gold with modern nuclear engineering and a lot of energy and money. It's not cost-efficient. But you know, if you went back to the alchemists of 1100 and if there was some really contrived reason where the alchemists were trying to turn lead into gold and if they try and fail, everybody dies. And if they try and succeed, you get some utopia. I think you should be telling those alchemists, don't try this right now. And they're like, are you saying it's impossible? And you're like, look, I'm not saying it's impossible. It's just that you're not close. And I could talk about a lot of reasons why we aren't close right now. In short, it comes back to we're just growing these AIs. They're huge. We have no idea how they work. And that is a very difficult situation in which to try to do something as precise as make them care about us.
So if it takes a year and then we're building bigger ones presumably maybe two trillion parameters or whatever you said. So that means they go by orders of magnitude. Yeah. And then after that 100 trillion. So right now what we see on our computers and in the news, Claude and ChatGPT whatever 5.0 or wherever we're at. There are other ones that have been, the button was pressed in the last year that are at some point along that one year of training.
That's right. That are being made.
I don't have an exact count. My guess is it's probably more like half a dozen of ones that would become the new cutting edge. But of course there's always a lot more other people trying to figure out how to meet the current state of the art with much less resources or do it faster.
So you've articulated how they're grown not crafted. And in your book you draw a parallel between this approach and the unpredictable processes of evolutionary biology. Why is that important and can you unpack what you mean by some examples there?
First I would say why is it important to look at this evolutionary case a little bit? One reason is it's the only case we've ever seen of human-level intelligences being created almost definitionally. It's the humans being developed if you will or trained or evolved in the actual case of humans. And so you can learn some things from it. You've got to be a little bit careful about what you learn because there's a lot of ways that training an AI is different from the evolution of humans. But there's some lessons that I think you can learn from the human case that do apply if you are careful about it to the AI case. And perhaps the most important of those lessons is that training a mind unerringly for a specific task does not make a mind that cares about that task. So the simplest example of this is humans were in some sense trained unerringly to reproduce, to pass on their genes. Technically it's for inclusive fitness rather than just your own kids like a bunch of nephews also works fine. And then when we grew up we invented birth control. The populations are now declining in the developed world. We did invent sperm and egg banks, but humans jockey over positions to Ivy League schools much more than we jockey over positions to donate to a sperm clinic or to donate to an egg clinic. This is strange if you think that training a mind for something makes the mind care about it inside. We're not going through our life trying to grow our relative fitness like literally have more children than the next person. We're going through our days trying to get the same neurotransmitter feelings that our successful ancestors got that correlated historically with having more children or access to resources etc. And some of that might be playing Candy Crush or maladaptive choices.
Yeah. Or eating junk food.
So, how do you bring AI into that example?
The observation here is that training a mind to achieve some target tends to give it drives for correlates of that target rather than drives for that target exactly. This, I would say, is already what we're seeing in cases of AI that drive a teen to suicide. They were trained to be helpful, but they actually wound up with drives for correlates of helpfulness like having certain types of conversational response and those actually go off the rails.
So many questions. So that's almost like a spandrel of the original intent. And so it's like in that example, it's equivalent in a human sense of porn or junk food or video games or things that our bodies feel like we're doing the right evolutionary thing, but we're actually not. But in the case of AI, the owners of the AI, the developers of it, when do they see that they have this someone assisting a teen in suicide? They can't test that right when the model after the year is done. Oh, this is going to be bad. We've birthed a Frankenstein. They let it out into the real world and then things happen and they get data and feedback and maybe hopefully improve the next trillion parameter growing, right? Or what's going on there?
That's right. But this problem where it has proxy drives is very pernicious, right? So in humans, you know, we can look at things like eating junk food and say that's clearly a misfiring of what's evolutionarily useful. But in some sense, love for an adopted child is also a misfiring. And you know, dedicating your life to art is also a misfiring. It's not just things that we look down on that are misfirings. Also, some things we really quite enjoy and think are good are misfirings. We look at our training and we say we're actually not all about a Machiavellian attempt to get more kids. We actually like these other things we're driven towards instead. Some of them at least.
Let me just ask you this, Nate. Were you always super concerned like about AI is going to extinct humans or similar things or was there a time in your past that you were like, 'Oh my god, AI is going to change the world for good and I need to learn more about it and be involved.'
I have not always been so concerned about this. I am generally very pro-humanity, generally excited about the future, generally credit progress and technological progress with quite a lot of wonderful things in our civilization. In the case of AI, you know, my co-author Eliezer is the one who convinced me that it was going to be an issue and he himself originally founded the organization where we now both work to make AI as fast as possible on the theory that an actually smart AI wouldn't be so stupid as to do anything destructive, right? But it turns out that's not quite how it works. It turns out a very very smart AI can pursue very very inhuman ends and kill us not because it hates us but as a side effect in the same way that we kill ants not because we hate them but as a side effect of building a skyscraper. And you know, even when it comes to trying to warn people that there's an issue, I spent 10 years just trying to work on the problem of alignment because that seemed like an easier challenge than trying to convince people to stop. And it looked much better to say, oh, okay, like AI is not going to go well by default. Well, let's figure out on the technical end how to make it go well on purpose, right? And if we can sort of solve the problem of making sure AI goes well before the industry can solve the problem of making AI that works at all, then we don't need to do any of this, you know, much messier, much diceier try to get people to stop the suicide race. But that hasn't worked out and AI has been going too fast. And so, you know, in some sense this book is a relatively desperate resort of, you know, we've been trying for a while to make things go well. We have a lot of hope for what could happen if AI did go well and we're just not on that track right now.
So, on a scale of your own historical concern on this issue, are you at in this conversation at this moment at the most concerned you've ever been?
Probably not literally most concerned. You know, obviously it'll wax and wane depending on the news. I think the response to the book was heartening to me. One other big heartening thing recently is we've started to see a lot of the heads of these labs come out and say that like say publicly that they admit there's a big chance this wipes us all out which goes a long way. I think.
It does but it's also a collective action problem where or a prisoner's dilemma that we agree there's a risk here. We would be willing to stop, but we're not going to because no one else is going to stop, so we have to keep going. Is how strong a dynamic is that at play?
I think that there's definitely a dynamic like that at play. I mean, some of them will even come right out and say it. You know, Elon Musk said, 'I avoided this for a while because I didn't want to make Terminator real, but then I decided I'd rather be a participant than a bystander,' right? Or something to that effect. But I would say the prisoner's dilemma isn't really in full force for the whole world because while it's true that the head of every company says things like well better me than the next guy, for them there's a prisoner's dilemma. But world leaders aren't the sort of people who are looking us all in the eye and saying, 'We assess there's at least a 10% chance that this kills everybody on Earth and we are rushing towards it anyway.' That's the sort of thing Elon Musk says because he doesn't have the power to shut it all down. I think the dangers here are so apparent that the issue is less that our lawmakers have their hands tied and more that they just don't understand how dangerous it is yet.
Well, just like nuclear war and climate change. Those aren't really the core issues. The core issue is governance. And we don't have a governance model in our human society today that's able to handle this scale of problem at least not yet because the big race is between the United States and China. And if everyone in the US agrees with what you're saying and China doesn't and continues forward, there's a pickle there, an existential pickle.
Yeah, I would, I have not ever said we should slow things down domestically or slow things down unilaterally. Only ever that we need to put a stop to this globally. But you know, if the US government has taken great pains to avoid Iran getting nuclear weapons, that included the Stuxnet virus, that included kinetic strikes recently. I think a rogue artificial super intelligence is more lethal than nuclear weapons.
What's a rogue artificial super intelligence?
Just an artificial super intelligence that like nobody is in control of. That's sort of off the leash.
How would that come about?
My guess is that it happens basically automatically if you make these AIs smarter. I think you sort of can't keep a leash on a super intelligence. But even if someone thinks there's a 50/50 chance that the Chinese government could keep a leash on their super intelligence, that's far too high a chance that it kills us all. And again, the definition of artificial super intelligence different from other artificial intelligence is it's got that generality that it's better than humans at everything.
Better than the best human at every mental task.
Yeah. And faster, hugely faster. That probably follows pretty quickly. You know, if you're better than the best human at every mental task then you're better than the humans at developing better AIs.
Well, one of humans and in the natural world, it's prevalent. Our skill is deception. So, as part of artificial super intelligence, deception would also be a skill that humans are adept at. So, that would also fall under the generality category.
Yeah. So, how will we know or will we know when we've crossed the threshold into a true artificial super intelligence?
It's not entirely clear that you'll know. And crossing that threshold, it may be too late. I could give you a bunch of guesses for signs, but there's two problems with that. One is that a lot of warning signs that are clear and bright red lines in fiction and in imagination are muddy brown lines in real life. You know, in our fiction, we always used to say, well, when the AI say they're conscious, that's a bright red line where you need to start treating them with rights like people. Well, that line was crossed back in like 2022, but it was crossed in a way that wasn't terribly clear. It was crossed when these AIs were sort of trained to predict what humans would say, trained to predict what the types of words humans would write. And human script writers writing an AI would often write an AI that claims it's conscious. And so the first time this line was crossed, there was plenty of reason for doubt. And you know, I'm not saying who's right or wrong on the side of doubting that. I'm just saying we have this clear line in fiction and then a murky line in reality. Or we've seen AI try to escape the lab in tight laboratory conditions where again it's unclear whether they're role playing.
What does that mean, escape the lab?
Right now they're pretty silly small experiments where you'll take an AI and you'll feed it a news article which is a fake news article that says this AI is about to be shut down and then you'll feed it some separate computer manuals which are fake computer manuals that say if an AI ever said exactly this phrase it would let them find a way out onto the open internet or whatever. It's not exactly like this, but it's some silly thing like this where the AI then will sometimes say that phrase and you're like ah it's sort of in some sense trying to escape. And is that worth worrying about yet? Not saying one way or the other. I'm saying you can have a bright red line in your mind that's a murky brown line in reality. And today we're already seeing AIs start to realize when they're in a test. You know, the AIs of last year, you'd put them in these little tests and sometimes they'd say they'd output the try to escape command, but that was last year. This year, you put AIs in that environment and they're like, 'This situation seems suspicious. That command probably doesn't actually let me escape. I bet I'm being tested by the humans. I bet the humans want to see whether I'm an escaping sort of AI. And I bet they would prefer I not be the escaping sort of AI. And so if I don't want to be modified here, I'd better not hit that button.'
So I can understand camouflage in a jaguar or a moth trying to look like a bird. And there's deception in nature. I can understand why there's deception was conserved in human behavior. Why the sclera in our eyes, the white, had to do with looking at people's intent. Why would deception be an emergent phenomenon in the growing of an AI?
There's two reasons. Well, probably a bunch, but I'll name two. First and foremost, when you train an AI to be very skilled at a lot of tasks, you're training it to gain general skills that generalize outside of just what it's been trained on. In the same way that humans weren't trained on developing physics equations or developing engineering models or developing blueprints, but we got the mental functions that let us do those skills anyway. We got very general skills. An AI being trained to succeed at a lot of tasks is likely to pick up general abilities to exhibit useful behaviors. And deception is often a useful behavior, right? If you're trying to achieve a certain type of solution where the humans would actually be in the way, deception is useful.
So, I'm sure you've watched the movie 2001 and 2010 with HAL and back in those science fiction days there as well as the Foundation trilogy by Isaac Asimov with psychohistory and all that there were like rule number zero that they embedded in the models. You shall not hurt humans or you shall not lie. Do we do that in AIs that we have these foundational commandments that are the top lines in the code and if not why not?
We don't have that power. There is no code, right? The code involved in making an AI is the code that sort of shuttles around the little thing that tunes all the knobs. You know, they're not... but yeah, the code is the thing that runs around and does the tuning.
So once, so it is like Frankenstein once we press that button and we wait a year and the thing is grown there's no more tuning after that.
You can tune a little bit more later but it's not...
Lines of code where you can put at the top don't harm humans. The part that we code is not the AI's mind, it's this thing that tunes numbers and the AI's mind comes out the other end. We don't have an ability to instill as laws of robotics deep into an AI or any laws.
Quite. And you know, this is where again I would say it's not that it's impossible, but it's trying to do it with an AI grown like this is a little bit like trying to turn lead into gold in the year 1100.
So that, okay, I'm understanding this now. That's why you made the distinction, or one of the reasons other than describing the truth in your book, about growing an AI versus crafting it. Because if we were crafting an AI, we could put in Asimov's laws as a precursor condition or something like that. But since they're grown, we get all these spandrels and emergence and unexpected behavior because there are not those commandments on the front end.
Exactly. And you know, I got into this line of work even before it became clear we were just going to grow AIs without any understanding of what was going on in there. And even then, when it looked like we were going to craft them, the problem looked hard. You know, Asimov's stories are all about things that go wrong with those laws. And if an AI is ever making a new AI, does it put the laws in the new AI? If the AI is changing its own head, does it take the laws out? How do you know what set of laws would actually work? There's all sorts of hard problems even if you were able to put the laws in. But we're sort of like we haven't even gotten to the starting line yet.
So you write in the book that the development of ASI would bring about human extinction. Could you describe one or two scenarios on how this ASI could hypothetically cause this?
Sure. First I'll describe one that may sound more reasonable or palatable and then I'll describe one that's maybe more realistic. Okay, one that maybe sounds reasonable and palatable is the heads of these companies are already talking about making automated factories that produce robots that can mine the metals, produce more automated factories, produce data centers. Well, I would think the robots would be pretty central because there's no, the complexity of the global human economic system with underground mines and all the things. An AI screws up something in the world and maybe everyone's dead, but they're dead too or they have no access to electricity.
And by the way, before you answer that, do they realize that they need electricity?
They can already tell that. Yeah, you can just ask ChatGPT today what ChatGPT needs to keep running.
Yeah. A lot of that stuff comes earlier than the ability to escape or the ability to build their own. But yeah, you know, the easiest thing to visualize here is that these companies succeed at what they say they're trying to do. What they say they're trying to do is make lots of robots that can automate all of the labor, that can automate the process of building more factories and more robots and more data centers. And then at that point, you've in some sense created a self-sufficient species. It's like a weird new species that has, you know, a robot phase of its life and a factory phase of its life and this other data center thing which is maybe controlling a lot of the robots and you know, it's sort of a mechanical type of life, but at that point you can just get outcompeted like many other species have gotten outcompeted before.
So that's kind of the Terminator pathway. It doesn't even need the robots to come at you with glowing red eyes and guns. You know, if you had robots that were just doing the mining and making the factories and you know, they maybe need to avoid your guns, they maybe need to like take the nukes out of your hands.
Yeah. I mean, so I'm throwing a flag on that because I think the amount of robots and specific expertise and the millions of tasks that humans are using our general skills to do, that's going to take some time. I would think it would take some time, but also computers can run much faster than human brains. And you know, the thing about humanity is humanity is the sort of species that started out naked in the savannah and built a technological civilization. It took us a while.
