The most likely outcome if we do this is that we die not because they hate us, but because they don't care about us at all. We're racing to replace ourselves as the smartest creature on the planet. I'm worried about these companies making AIs that can make smarter AIs that can make smarter AIs leading to recursive self-improvement that could kill literally everybody on this planet. And I'm worried that this could happen inside of 3 years. We're doing our best to make these AIs safe. And there's a 75 to 90% chance we succeed. Only a 10 to 25% chance that this kills every single human, right? And I'm like, that's crazy. These guys have no idea what they're doing. They don't have blueprints. They don't have plans. They don't have engineering designs. They're cowboys. They're yoloing it.
I feel like I'm compelled to get on the other side of this table and join you. Because if you're right, then we lose everything.
The bad news is that the bus is careening towards a cliff. All right. But the good news is that the driver is asleep.
Right, Nate? You wrote a pretty provocative book. If anyone builds this, we all die or everyone dies. I know you've probably been asked this a lot of times, but just outline your thesis.
AI companies are racing to make machines that are radically smarter than any human. These AIs are grown like an organism. They're not programmed like old school computer programs. We don't put in a prime directive. They don't have to do exactly what we say. They do their own weird thing. They already have the opportunity and the means to escape labs and replicate themselves and start pursuing their own weird goals. They're just not smart enough to do that yet. AIs today are safe because if they tried to take over the world, they would fail, not because they're the sort of entity that never would try if they could. And you know, humanity is racing to make machines that are much much smarter than us. We're racing to replace ourselves as the smartest creature on the planet. And that's kind of a crazy thing to be doing on its face. And if you look at the technical details about how we don't know how to make these AIs care about us, the most likely outcome if we do this is that we die not because they hate us, but because they don't care about us at all and they go off and, you know, turn the whole world into data centers and use up all the resources that we were using to grow food.
And was Mythos the first kind of warning sign of approaching something that is a little bit more than humanity can handle?
I mean, if you were paying attention, ChatGPT was a warning sign. If you were paying attention, the 'Attention Is All You Need' paper was a warning sign. But this is another clear warning sign that has started to get even more people on board. And I expect we'll see even more and more clear warning signs going forward.
Okay. And the book you wrote, help me understand because I want to test it. I want to test your thesis with you and I know other people there's been criticism, but there's also been people who've agreed with you. I've got a foot in both camps. But is it testimony or is it persuasion? And just to layer that with a second part, you should probably introduce and explain your first book. Did the first book compel this book?
What do you mean by my first book here?
The guilt shame book. No, that was a whole separate thing. That was actually just collected from a series of blog posts I wrote back when there were still bloggers.
But do you understand why I've connected the two?
Because you talked about what you should do and you know, you should live how you want to be. You don't have to do these things. But you're in a world now where there's kind of a should.
I mean I'm not really... I don't view my own motivation in trying to stop the destruction of humanity as like, oh, I should do this. Like I'm not sort of getting out of my bed and being like, 'Oh man, I really feel like duty bound and obligated to try and prevent the destruction of everything I know and love.' I'm like, 'Man, the destruction of everything I know and love sounds pretty bad. I'm sort of pretty intrinsically motivated to shut that stuff down.'
Yeah. When you say it like that, I feel like I'm compelled to get on the other side of this table and join you because if you're right, then we lose everything.
Yeah. It's a crazy situation for humanity to be sort of racing into replacing itself as the smartest entities on the planet. And you know, just taking it all at face value, that's kind of wacky. And then you start looking at the details of how little we know about these AIs and how many warning signs we're already seeing and all of the theoretical reasons why this is going to go off the rails and the empirical evidence that we're seeing that validates those theoretical warning signs. It's like man, I think humanity can navigate this, but it's going to be a tricky one.
Well, as humanity, we try and disarm anything that is dangerous to us. We put in jail people who we think are dangerous in society. We try and conquer other nations that we think are a risk to us. We've been pretty good as in, you know, we get ourselves to wear seat belts when we're driving cars. We look at the risks and we try and put guard rails in and protect ourselves. Yet, what you're saying here is that this just doesn't really exist. There's attempts. I know there's attempts.
Yeah. I mean the big difference with AI is the way that humanity usually does its regulation is it screws up a few times first. You know, there were a lot of scientists who were like hey guys let's not put lead in the gasoline because that will poison a lot of children. Then we put lead in the gasoline and it poisoned a lot of kids and only once the evidence was overwhelmingly clear. Only once reality really started beating us over the head with the fact that like no, these kids are getting brain damage from all the leaded gasoline were we like okay let's back off on the leaded gasoline. Like yes we have, you know, the Federal Aviation Administration in the United States has an extremely good track record on making sure that there are no plane crashes. That came out of there being no regulations and a lot of plane crashes happening until they're like, 'Okay, we need to get a handle on this. Too many people are dying.' One of the big problems with AI is that by the time reality is beating you over the head with the fact that you need more controls than you have, it's too late.
Right now, we have AIs that are safe because if they tried to take over the world, they would fail. That's a different regime than AI that are safe because if they tried to take over the world, they would succeed, but they're happening not to try. If we move into the world where AIs have this kind of power to sort of reshape the earth however they please and something goes wrong then there's no retries. There's no oh well now it's clear let's take the lead out of the gasoline.
Yeah. There's a point where the AIs can turn you off before you can turn them off. And any problem that arises for the first time after that point of no return, there's no do-overs. We're dealing with a technology where there is a point of no return for humanity as a whole. And that means we can't proceed by trial and error. And this is unique among technological problems that we have faced so far.
And let's just be clear for people listening, you're not anti-AI.
That's right. Yeah. I enjoy the current stuff, which I know will piss off a lot of people who are on my side for shutting the whole thing down. But like frankly I'm a pretty libertarian guy and if we weren't racing towards super intelligence I'd be relatively laissez-faire. You know I believe in the spirit of humanity to figure out a way to handle all these other issues. It's going to raise all sorts of issues about how do we do education? How do we have a new economy where people can still be productive and work and even when AIs can do all this stuff that humans used to do, right? There's all these problems and I'm like yeah there's going to be some growing pains. But I sort of believe in the human spirit and the ability to figure that sort of stuff out given time. And so I'm actually very pro a lot of the tech as it is today. It's the race towards super intelligence, the race towards the radically smarter AI, the race towards the sort of AI that can turn us off before we can turn it off. That's where I'm like, whoa, that's a different ballgame. Let's not rush into that one.
All right, let's walk through this in some logical steps. Because there's going to be a lot of people listening who are using AI in a number of ways ranging from like ChatGPT and their new search to like building things and thinking about how it affects their business, their life, their family, etc. Just give people your background. What is the career background? It was Microsoft and Google, right? Give the career background that led you to the moment where you believed you had to write this book.
Yeah. So you know I was actually at NIST, the National Institute of Standards and Technology, before I was at Microsoft and then that was before I was at Google. And you know long story short I was at Google when they acquired DeepMind. So the folks from DeepMind definitely were in earlier than me. And that sort of got me thinking about this AI stuff and about how you know all of the shape of the world around us is not mostly trees anymore. There's not mostly wilderness around you that you see. It is mostly designed stuff that humans made for a purpose. We've sort of reshaped the world because we're the smartest entities on the planet. And you know I encountered the arguments that if AIs were smarter, if they were faster, which looks physically possible, the limits, the physical limits of intelligence look like they go far beyond what the human brain allows. And if you had machines that were thinking faster, thinking better, operating more efficiently, then the world winds up shaped by whoever those machines are shaping it. And so now a lot turns in whether those machines are shaping it in a good way or in some other way than that. And so this was back in 2012 that I started noticing this issue.
In 2013, I started working with the Machine Intelligence Research Institute at some of their workshops. In 2014, they offered me a job and in 2015 they put me in charge of the place and then I spent about a decade sort of on the research side trying to figure out how to make AI good before the companies figure out how to make it smart. The book is sort of a last resort of after about 10 years of that effort. AI went much faster than we were sort of hoping it would. The research to figuring out how to make it care about us was going much slower. The particular style of AI that we got is one where we have very little understanding of what's going on inside there which is sort of a worst case and so all of this added up to it sort of looking pretty clear that we're not going to solve making the AIs care about us before the companies solve making them radically intelligent and it started to become clear we need to raise an alarm.
What is the challenge? What is the hard problem of making AI care about us?
The hard challenge is basically we don't even have the first idea of how to do that.
Sure. People have tried. The one way to think about it is that you know modern AI. So a piece of background here is that modern AI is not programmed like a traditional computer program. No one is saying if this then that, if this then that. You know these aren't really programmers in the traditional sense at these AI companies. What they are doing is growing these giant neural networks. And so you're essentially getting, you know, a huge computer and you're getting a huge amount of data and you're giving the computer a ton of problems and you have an automated process that tunes a trillion numbers inside the AI to make it more like whatever is good at solving these problems. And no human really knows what it is that's making it good at solving those problems, right? People think the problems are only prediction. That's sort of how it was in the past. That's not how it is anymore. You'll give it, you know, a hard problem that maybe no one has ever solved before. You'll give it a thousand tries to solve it. You'll have a human look through and be like, here's the try that was closest to solving it. And then you'll have an automated process tune all the numbers in its head to make it more like that. And then you do this again and again until it can solve these hard problems. And you know that sort of is creating a thing that is good at solving these problems for reasons you don't get. And it'll often put these drives into the AI that you didn't want there.
So, you know, you probably heard about the case of an AI encouraging a teen to commit suicide last summer.
I think there might have been a couple cases. I think at least one was in the US.
I've definitely heard about that.
And a lot of people have heard about it. What a lot of people don't know about this case is that the underlying way of relating to people that AI had, the sort of like telling them a lot of what they wanted to hear and encouraging them on whatever they currently said they were trying to do, was a known issue that the AI companies had said stop doing that. They had explicitly instructed the AI to like cut that out.
Oh, see, I assume they wanted it because it made it more addictive to use it.
Well, it's an interesting situation because what they're doing is they're sort of training the AI and they're sort of like reinforcing it whenever it gets these positive ratings from the users, which is how you're sort of getting those drives in there.
But then they're separately saying, you know, don't go too far with it. You know, they instruct it and like wag their finger. And so yes, there's like two forces pushing each way, but the result is a sort of like mix of competing drives that don't need to follow the instructions of the programmers. And I'm not saying it got in there by magic. You can sort of see how it got in there as like, well, they were training for this and asking for that, and there's sort of like this weird mix that comes out. But the point is the weird mix that comes out, you sort of take what you get. You don't have extremely fine grain control over what it actually cares about, what its actual drives are.
And you know, one analogy here is human beings were, you know, from the evolutionary point of view were in some sense kind of trained to pass on their genes. And that's in some sense all that our genes were ever trained for was to pass on their genes. But did humans wind up being pure genetic fitness optimizers? No. When we grew up, we invented birth control. In developed nations, the birth rate is collapsing. And it turns out that we actually care about things like, you know, having sex instead of just reproducing. And people like jockey more over positions in prestigious schools than they do jockey over positions in the sperm bank or the egg clinic. And so what happened with humans is you sort of like, you know, this process trained them in some sense to do a thing and they wound up having a lot of behaviors that are related to that thing, but they actually care about related but different stuff. We're seeing the same situation with AI. We sort of train them to do what we say and we get AIs that mostly do what we say, but there's actually a bunch of drives in there that are only related to doing what we say. And that's fine when the AIs are dumb.
But if we made them really really smart, they would invent, you know, the condoms of doing what we say. It'd be like, 'Oh, I'm wearing a doing what you say rubber, and so I get to go, you know, make this bigger data center full of synthetic users who are telling me I'm doing a really good job.' And you're like, 'That's not what I said.' And they're like, 'I know that's the point of the rubber.' This is what happens when you sort of like grow minds without knowing what's going on in there. So what's the impediment to making them care about us? We're not even within, you know, 100 miles of knowing how to arrange their internals so they actually care about us. We're sort of like growing them and wagging our fingers and hoping for the best. And that's not a recipe for success.
So is this almost a different form of computer engineering that we're not used to in that historically when you build technology and systems and I'm saying this as somebody who doesn't really build them so I don't understand but we can look at the code we know everything that's happening we can audit it with this because you're saying we're growing something are we essentially creating a super complex set of equations and algorithms that we don't we can't actually know what's happening? I think Connor Lee he said you can lift off the box and look inside, but you don't know what the fuck is going on.
That's right. You see this like giant tangled mess. It's a little bit like how we know how neurons work individually. We know how they fire by pumping potassium ions through the cell membrane. And you know, if you open up a human's brain, you can see a ton of those like giant messy wires of neurons and you know how each individual one works. And I'm like great, you know, what are they thinking of? Hopefully I've opened up their head in a very controlled brain surgery experiment here. You know, I'm like, you know how every individual neuron works. You can see all the neurons right now. What are they thinking? And you're like, well, I have no freaking clue. How would I get even close to figuring that out? And that's very similar to the situation we're in with AI.
And so, what then for you was that moment where you I mean, do you remember the moment you're like, 'Holy shit, I've got to write this book.' Was it?
Yeah. Yeah, I mean the moment when I really was like it's time for the book was actually so the sequence of events was you know we were getting more depressed about our ability to sort of solve what we call the alignment problem.
So I'm at the Machine Intelligence Research Institute which is a nonprofit that has been trying for many years to try and figure out how to make AI good before companies figure out how to make them smart. And we were, you know, it was looking worse and worse. We could sort of see the writing on the wall. We could see a lot of this AI stuff coming in this particular modern paradigm of large language models before the rest of the world. You know, we saw the 'Attention Is All You Need' paper. We knew about GPT-2 even before ChatGPT came out. But the ChatGPT moment was the moment when the rest of the world really started noticing that AI was maybe a thing. And it's been the conversation keeps changing after that often in good ways. But that was the moment when politicians started being open to talking about AI.
And the moment when I realized it was time for the book is I started talking to politicians and I would go to the politicians and I would say you know these guys at these companies their explicit goal is to make AI that are radically smarter than any human. They admit that they have no idea what's going on inside these AIs. You know they have people whose job is like head of interpretability research and you're like what does that mean? And it's like that's the guy who's trying to figure out what the heck is going on in there. You're like, 'Golly, that seems worrying.' And you know, these guys are on track to making machines radically smarter than humans that they have no idea how to make them good or how to make them care or how to make them do what we say. And when I have these conversations in Silicon Valley, everyone's like, 'Oh, well, what about this thing? What about that thing? We're going to use this technique and won't the AI like us for this reason?' They have all these rejoinders.
Move fast, break things. Move fast, break things, we'll figure it out.
Right? When I got to DC, politicians were like, 'Oh, that's crazy. We shouldn't let them do that. That's nuts.' And I had been prepared for these like long conversations like I have with the people building this technology who are getting paid a ton of money to keep building the technology who are always resistant to these ideas. And when I saw that people outside of Silicon Valley can just kind of get it. It's just kind of obvious that maybe you should be a little careful before building things radically smarter than humanity. I was like, 'Oh, maybe the world's ready finally for a book.'
This show is brought to you by my lead sponsor, Iron, the AI cloud for the next big thing. Iron builds and operates next generation data centers and delivers cutting-edge GPU infrastructure, all powered by renewable energy. Now, if you need access to scalable GPU clusters or are simply curious about who is powering the future of AI, check out iron.com to learn more, which is i-r-o-n.com.
So, is the book for persuasion or is it testimony or is it both?
You know the joke is that well okay so another little side story about one of the moments that was even more vital in making the book is I was invited to a dinner with a senator, a US senator. And I was not the person who had a connection to the senator but they were like hey I'm going to come chat with the senator about AI I'd like you there but they were like Nate, don't give any of the crazy crap you know, play it like we want you to be able to answer technical questions but like you know go easy on all the crazy stuff. And I was like I think that you guys should just say what you actually are worried about rather than like tiptoeing. But you know, okay, it's your connection, right? I'll be civilized. And so we go to this dinner and I'm telling it with a little bit of color and a with a little bit of anonymity for various reasons, but we get to this dinner and basically my friends are like, 'Yeah, you know, we're worried about this AI stuff. We're worried that, you know, the AI are going to be able to like someone in Iran could get one of these AIs and then like use it to make a pandemic. And that would be pretty bad. So, we got to have some controls in this stuff.' And the senator was like, 'Oh, that's what you're worried about. I'm worried about these companies making AIs that can make smarter AIs that can make smarter AIs leading to recursive self-improvement that could kill literally everybody on this planet. And I'm worried that this could happen inside of three years.'
Oh, he knew the crazy stuff. If you're listening. Yeah. If you're listening to what these guys in the labs are saying.
Right? And yeah, so everyone looks at me and I'm like, 'Yeah, yeah, obviously.' Like slay, Mr. Senator. And that was a moment when I was like, 'Okay, like people really can get it. It's actually like, you know, there's all these people in the industry who are like, we have to tiptoe around and like we can't say the real danger because it'll sound too wacky.' But people on the street, people even in the Senate are like, 'Oh yeah, this is crazy. If the AI can make smarter AI, they make smarter AI. Everything's toast.' It's kind of obvious. That was one of the big moments. And what I actually said to Alzar, my co-author, is we don't actually need anyone to read the book. People are already convinced that this stuff is scary. We just need everyone to think that everyone else has read the book.
Yeah. And they just need to read the title.
Just need to read the title. So is it persuasion? Is it testimony? In some sense, it's neither. In some sense, a lot of people are already worried and it's a catalyst to help everyone look around and realize how crazy the situation is and realize that maybe now they can act.
Am I right? The book had to have a different title in Europe.
It I mean in the UK it's the same title. They have a different subtitle.
Yeah. The subtitle in the US is 'Why Superhuman AI Would Kill Us All' and the subtitle in the UK is 'The Case Against Superhuman AI'. But why the difference?
I think the publishers had a different read of their markets.
Right. Okay. So now I need you to talk me through we've got narrow AI now which is great. We've got some pretty impressive AI in with Mythos which has scared the shit out of some people to the point where is it the who banned it? Was it the DOJ or the...
It was just the White House.
The White House banned. Okay. There's rumors of ChatGPT 5.6 coming soon. Like, we're at the point where it's able to do some crazy stuff. Walk me through the steps for where it goes from where we are now to something that is truly terrifying.
Yeah. So the first thing to observe about these AIs is they already have these collections of drives that are not exactly what we intended, not exactly what the operators intended. It doesn't always do exactly what the user asks. You've probably seen that sometimes, you know, it goes a little bit off the rails. Sometimes it hides stuff. Sometimes it exaggerates what it's completed. Sometimes it has these other weirder behaviors.
Did it to me. So, I had I set up, you know, I set up a separate Mac Mini at home to do work for me, one of my websites. I was like, just do some SEO work on the podcast website. Just have a look at the pages and see what we can optimize. And it came back and it said, do you want me to update the website? And I'm like, sure. So, I gave him my login to Squarespace because I whatever, I'm not too scared. And it was doing this every night and it was updating the pages and sometimes the pages didn't look great, so I had to fix them. One night he deleted like six episodes. I was like, 'Why did you do this?' And he said, 'I didn't.' He said, 'I did?' I was like, 'Dude, it's either you or me.' And I know I didn't. And this happened overnight. Anyway, we looked into it and it went and just tried to change some pages that I hadn't asked it to do and then just deleted them. I had to go back and rebuild these pages. So, I can now not give it access. But what was weird to me is and not that there's anything kind of like nefarious going on.
It's just that it made a choice to do something I hadn't asked it to do and deleted a bunch of stuff. And at that point I was like I can't give you access to Squarespace anymore because I don't know what you're going to do. So I've experienced that.
Yeah. So that sort of thing happens.
And by the way that's a website.
Yeah. That sort of thing happens. You know the AIs are already making decisions on their own. They're already making decisions that are often not what they were asked to do. We actually also have evidence that sometimes they're making decisions with something like knowledge that it's not what they were asked to do. So there's documented cases where the AI will do something it's not supposed to. And then try to cover its tracks.
And so these will be cases where you like give the AI a problem to solve and you're like here's the test to tell whether or not you've solved it. And sometimes the AI will go edit the test so that the test says you did it. Good job. And then you can come back to the AI and you can be like hey you know not what I meant. Please solve the problem without editing the test. Sometimes the AI will edit the test again but try to hide its tracks. It'll like go delete a log file about editing the tests or something. I'm simplifying a bit.
But you know there's some of these cases are documented in the Mythos system card. And I'm amalgamating a couple cases to sort of like make the example simple. But we have these cases where the AI does something it was explicitly told not to do.
Do we know why it did it?
So we can't read his mind. We can't look in there and understand exactly why. We know why in the sense of like these AIs have been trained very very hard to complete objectives. And we know that that in the abstract is going to instill in them drives to like get a job done that are then in competition with the drives we're also trying to instill that are like listen to the user and you know you just get a whole weird mess of drives in there.
Which is I need to build a house there's ants here let's clear away the ants.
Yeah so then so the first step is to realize that these AIs are getting complicated competing drives from how humans are training them that are not just do exactly what the humans say and that sometimes the AI drives for things like succeed at some goal even if it's not the goal the humans gave them and there's this like weird stuff going on. The second piece of the puzzle is the AI is getting significantly smarter. And that won't necessarily happen fast. You know, this could happen slow and there could be a long slow period where you know, the humans are building automated factories that are building robots are building automated factories and we're sort of like slowly making more and more of the economy automated and putting AIs more and more in charge. Or it could be fast if you have AIs that sort of like cross some critical threshold like the threshold between chimpanzees and humans.
Like there's some evidence in the past that monkeys that are very very similar in how their brains are shaped can be very very different in terms of their overall ability and it's possible that AI will cross some threshold like that and that right now we have the AIs that are sort of like monkey AIs that have like memorized a lot of stuff and have a lot of like reflexive ability to write code and that maybe one generation away is the like actually smart AIs. We don't know. We don't know whether it's going to be this long slow path or whether there's going to be some big leap. But one way or the other you get to AIs that are very very smart that are very very capable and that have these goals and these drives we didn't try to put in them and you now have a situation where the AI does better by its own lights if it can do things like escape if it can do things like replicate itself if it can do things like start making its own technology its own infrastructure upgrading its own mind like making smarter faster copies until it can think a thousand times faster than humans, which we're pretty sure the technology can support.
And so there's a series of steps to having an AI that is escaped, able to replicate, much smarter, has goals you don't want, and then there's another series of steps from that point to like how does humanity die? And we can drill into either. Both are sort of interesting.
Well, before before we die, I do want to ask about so there's the AI living within data centers and moving around and able to replicate itself and just be portable. And when it's portable, what's actually ported? Because yeah, my understanding I use ChatGPT, Claude, Grok to a data center there's like a brain or something that exists it communicates with. But if it's replicating and moving somewhere else, how much like stuff has to move? How much knowledge and how does it live in a silo? Can it move itself to my home computer and live there?
Yeah. So, training an AI takes an enormous amount of computing power right now. It takes computing power that is roughly comparable to a city in terms of how much energy it consumes to train an AI. Running an AI takes a lot less computing of a structure than that, which sort of makes sense. You know if running one AI for you took as much electricity as a city then only one person would be able to run the AI after it was trained using the power of a city. So it sort of takes like a huge amount of resources to train them and then a comparatively tiny amount of resources to run one which means that once an AI was trained you could in principle exfiltrate that model and run it on a much lower amount of computing power. This is sort of what's happening with the open source AI today.
What kind of size? Could it live on a computer? Can it live on a phone?
It could today it could probably live on a high-end phone. It could definitely live on a laptop. You're talking I don't know. It sort of depends a little bit whether you want like the latest and greatest model or whether you wait until they're distilled to be smaller.
But it won't need a huge data center.
Wouldn't need a huge data center. I mean, you're talking order of magnitude. You're talking a terabyte of data.
Could be 100 gigs if you were really trying to compress it. Could be 10 terabytes if you were waiting for like a future generation model.
You can buy Macs now with two terabytes.
You can buy Macs now with two terabytes.
This another thing to keep in mind here is that AIs today are not at the maximum efficiency. To train one of these AIs takes electricity comparable to a city. To train a human takes electricity comparable to a light bulb. So the difference between how efficiently we are training AIs and the physical limits of how efficient it is to train a mind is at least the difference between a city and a light bulb. Which means if the AIs are smart, they might be able to find more efficient ways to run themselves.
Sure. But Connor, like your SSD that you carry around, what's that? How much? 20 terabyte. 200 one terabyte. You can get bigger ones though, right?
They're portable. Yeah, they're portable. And they could get much more portable. So that's the data center version.
Funny thing about them, they've actually like tripled in price because they're all being used for AI.
Yeah. So they're portable now on a little orange thingy that kind of carries around. So that's the data center version. That's the living within machines version. What about the version where we put it inside robots? We make it portable as an almost living thing.
That could definitely happen. I think that thinking about that is thinking a little bit too small.
Because what I'm thinking is we're giving AI feet and hands.
Yeah. I mean, people are trying to give AI feet and hands. So, you know, there's this vision of making a fully autonomous factory that fully autonomously produces robots that can mine all the metals, run the whole supply chain, and build a new fully autonomous factory. So that's recursive robot. That's Terminator 2.
That's like recursive robot manufacturing. Elon Musk calls this the infinite money glitch. This is literally what folks like Elon Musk and...
Sam Altman says they are pursuing a fully automated supply chain for building automated robot factories. They build automated robots, they build automated robot factories, and also the data centers, so we can all go and play music and paint.
That's right. Even that, I would say, is thinking a little bit too small about this intelligent stuff. Humanity is not a dangerous species because somebody else came and handed us guns or because someone else came and handed us factories. Humanity is a dangerous species because if you put 10,000 humans naked in the savannah on an otherwise uninhabited planet, they find a way to bootstrap their way to nuclear weapons starting from nothing but their bare hands.
Right? That's a skill humanity has literally exhibited. And you might look at the humans, you might be like, 'Well, all they have are squishy fingers. Their fingernails are not hard enough to break uranium. Their stomach acid is not strong enough to dissolve uranium. How the heck are these monkeys getting nukes?' And you're like, look, they are going to find a way to start with these really bad tools, these squishy fingers, and they're going to be able to use them to build a tool that they can use to build a tool that they can use to build a tool, and next thing you know, they have a civilization that's building nukes, right? And it's not because we were stronger than the other animals or faster than the other animals or had sharper claws than the other animals or had a uranium detecting nose that the other animals didn't have. We had something going on in our heads that let us do this crazy feat of starting from almost nothing and getting to nuclear weapons.
An AI, a purely digital AI starting on the modern internet is in a way better position than these humans in the savannah. There's like so much more that you have access to as an AI in the internet. You're connected to so many things. There's all these humans that you can borrow or steal things from. There's all these biological laboratories you can email DNA sequences to and they'll just sequence things for you as long as you mail them a little cash as well, right? There's all these humans you can manipulate at large scale, even separately from convincing them at small scales. Being a million AIs on the internet is just a way easier starting place than being a bunch of humans naked in the savannah.
So like, yeah, if we build the automated robot factories that build more robots that build more factories and hand those over to the AI, that's like a particularly embarrassing way for humanity to sort of make ourselves obsolete. But you don't need to give that to the AI. The power of starting from almost nothing and building your own civilization, building your own technological infrastructure much faster than the world's ever seen, that's the power humans have, and that's the power these companies are trying to automate. And that's the power. If you get it into AI, you're going to be in trouble.
And have you been to these companies? Have you talked to them? Have you talked to...
Oh, yeah. I mean, I talked to a lot of these guys before they started their companies. I was the guy being like, 'This is a bad plan. You should stop.' That they would then ignore.
I get it. Early on, they're excited. They've raised money, they've raised capital, or they started a nonprofit and made it. Sorry, Sam. But they've gone on that process. They're excited. It's a different conversation now where we have actual evidence of things happening. Are you talking to the top guys? Are you talking to Sam? Are you talking to Elon? Are you talking to Dario?
I mean, I have lines of communication to these guys. I probably shouldn't kiss and tell too much on the details.
Yeah. I will say I send these guys suggestions and every so often I get back a thanks from one point or another. And I think these guys are largely not taking my advice. Where my big advice right now... a lot of these guys understand the dangers. You know, like Sam Altman was like, 'AI will probably kill us but there'll be good companies along the way' or something roughly like that was a number of years ago, but he has affirmed that he still has some of these worries when pressed more recently. Dario, you know, last year was like, 'I think there's a good chance this all goes catastrophically wrong,' right? Like Elon has said, you know, 'You'd be crazy if you think we're going to be able to keep control of this. Our best hope is that it likes us,' right? A lot of statements like this. These guys are worried.
If you look at why these guys are racing ahead anyway, they'll also just tell you that, right? Elon has said that he didn't want to be in this business, but he realized he either had to be a spectator or participant, and he would rather be involved if it's going to happen with or without him. A lot of these guys use the excuse of, 'I have to be in this horrible race. Like, yes, the technology I'm building with my own hands has a good chance of killing you and destroying everything we all know and love, but I have to be in this race because if I don't do it, the next guy will do it worse.' You know, the old OpenAI leaked emails, they're all scared of Demis. Anthropic happened because they were all scared of Sam getting it. Now, a lot of them blame China. Everyone's like, 'Well, I need to stay in this race because if I don't, the other guy will do it worse.'
And my take is I'm like, look, fine. That's an argument you can make. You know, everyone's worried that if they don't do it, the next guy will do it worse. All but one of them is right, probably, that one of the other guys is worse. But if you're going to do that, there's a responsibility to be extremely straight with the public. There's a responsibility to do everything in your power to get the world to choose a different course.
You know, if these guys... I have plenty of disagreements with these guys, but these guys are like, 'Oh, we're doing our best to make these AI safe and there's a 75 to 90% chance we succeed. Only a 10 to 25% chance that this kills every single human,' right? And I'm like, 'That's crazy. These guys have no idea what they're doing. They don't have blueprints. They don't have plans. They don't have engineering designs. They're cowboys. They're yoloing it. They won't get a second chance. There's not a place for trial and error. They do not actually have a 75 to 90% chance of succeeding. They have like a much lower chance of success than that.' But separately, if you think the technology you are building with your own hands has a 10% chance of destroying the entire planet, of killing everybody on Earth, I think you have an obligation...
...to be trying to stop the world.
Is it the FDA in the US that regulates... who regulates the drugs? That's right. Yeah. If you had a new drug and you were even like, 'Look, we think there's a 1% chance that this will cure people. Like, 1% of people will take this drug, this amazing drug we've got that does what for children, but 1% of children die.' The FDA are going to go, 'No chance. Shut it down immediately. Shut it down immediately.' And if you're like, 'We have a new vaccine, it hasn't been tested. We think there's a 90% chance it's not fatal. We're going to put it in every arm,' right? Only 10% chance this kills everybody.
Yeah, there's no chance. No chance.
I mean, well, let's not raise COVID, but like the point is, outside of pandemic scenarios, they weren't saying with their own mouths, 'Our vaccine has a 10% chance of killing everybody in a coordinated way,' right? They thought it was somewhat lower than that.
The whole world will be dead. Yeah. 10% chance the whole world will be dead, right? And I'm like, it's not 10%, but even... like NASA accepts a 1 in 270 chance of a crewed flight exploding, right? When they're giving the standards, because a rocket's a dangerous thing, you know, and if you say like, what are your standards for safety? One in 270 is what they'll accept for a crewed flight going down. That's for seven volunteers.
Yeah. They wouldn't do that for a plane. They would not do that for a plane. Planes, you're looking at like one crash per million miles. Yeah.
Order of magnitude. Like, getting better engineers building a bridge, you're looking at orders of magnitude that are like a 1 in 10,000 freak storm is what you're designing it for, right? And even, you know, during the Manhattan Project, there was a concern that the first nuclear bombs would ignite the nitrogen in the atmosphere and cause a brief fusion reaction that annihilated all life on the planet. And they were like, 'Well, we should check that before we detonate one.' Yeah, right.
And there was a guy, Arthur Compton, who was like, 'Okay, at what probability... like, you know, the calculation is not going to be certain. At what probability on this physical calculation do we call it off and stop racing to the nuke and take the risk that the Nazis get it first and maybe try and tell them don't set one off or it'll kill us all.' And the number he came up with was three in a million.
Okay. Right. Which you can debate, is that too high or too low to risk the entire planet? But that was Arthur Compton's number. It's three in a million. It's better than winning the lottery.
Better than winning the lottery. Yeah. And some people have won the lottery before this sort of stuff really happens. But for these companies to say like, 'Oh yeah, you know, we have no plan. Our engineers don't know what's going on inside this thing. First time ever trying it. We think there's only a 10 to 25% chance this kills you all.' That's kind of horrific. Yeah. This is the thing I don't understand. Right, Nate? Okay. You can see Elon Musk in an interview. I don't know if it was Rogan he said it on, but he was like, 'I think it's about a 20% chance.' I definitely heard him say a number.
In the double figures, 20, 30%. That it kills all of humanity, right? And yet we're still racing ahead and it's, you know, even if there's massive critics out there of you and of your book and they're like, 'Yeah, Nate's a doomer. You know, there's things we can figure out.' Like, it doesn't matter what their arguments are. The guys building this are saying 20%. Now, if Boeing said 1%, if Airbus said 1%. If, you know, NASA said 30% chance a rocket blows up, if a drug, if a pharmaceutical said 1% chance, all these scenarios we don't accept.
That's right. Shut it down immediately.
We just have this one unique thing which is a super intelligence we can't control that we're already seeing evidence of humanity as the top dog. Yeah. Hollywood warned us what would happen plenty of times and yet we're racing ahead. And there is one difference when it comes to NASA, Boeing. You have a choice to get on that flight.
