Nate, thank you so much for doing this. You've written the world's darkest book. You've devoted your life to warning the world about the potential dangers of AI. So, let's just start by hearing your explanation of why AI is dangerous in addition to just being annoying.
Yeah, you know, the very basic common sense point is if you race to make machines that are much smarter than any human, to make machines that are more capable at inventing their own technology than humans are, and you race into this without really knowing what you're doing, the most likely outcome is just that the machines get loose and do their own thing and that humanity dies as a side effect, just like humanity has killed lots of other animal species not because we hate them but just as a side effect. And in some sense, a lot of people find it sort of obvious or intuitive that if you just make these really smart, really powerful machines, why would they care about us? And I think that intuition basically is right and there's a lot of arguments you can have on each side. You can get into the technical details. But my book is basically just getting into those details and saying like, 'Yep, it sort of holds up. If we make smarter machines without knowing what we're doing, it's just going to go poorly.'
Are we actually going to make machines capable of what you're describing?
You know, the companies are trying to, they talk about how they are pursuing super intelligence in the true sense of the word. That's Sam Altman's phrase. Dario Amodei of Anthropic says they're trying to make the equivalent of a country worth of geniuses in a data center. So these guys are really, you know, these aren't chatbot companies. They didn't set out to be chatbot companies. They set out to make these machines that can sort of exceed humans in every way. And that's what they're targeting. There's a separate question of whether they will get there.
May I ask you a question? Why would anyone want to build a machine smarter than people?
I think that a lot of them hope that they'll be able to make the world much, much better. You know, they hope for a cure to cancer and then more than a cure to cancer, they hope for a cure to aging. They hope for a thousand years worth of technological development compressed into two years. And I sort of don't think that they're going to be able to get that really or harness that for good ends, but that's what I think they're shooting for.
So they're not. But I mean the core point you're making is these companies did not set out to make consumer products like to make your life better necessarily in the short term with a more efficient search engine.
That's right. Yeah. You know, OpenAI started before the chatbots, the large language models were even a thing. They were started, forget whether it was late '15 or early 2016, but the paper that unlocked the most recent wave of AI came out in 2017 which was after OpenAI was founded. And these guys, the large language models are a surprise revenue stream and that revenue stream can fund the creation of even more, even larger data centers for the next level of the technology. But these guys have their eye on the sort of ultimate version of the technology which is these machines much smarter than humans. And that's sort of the ultimate version because once the AIs are smarter than the humans, the AIs can carry on the AI research and make the next generation of the AIs which make the next generation of the AIs and that's sort of what they're shooting for.
What is super intelligence?
We in our book define super intelligence as an AI that is better than the best humans at every mental task. So anything that a human can do purely mentally, the AI can do that better. One thing people often get caught up on about this is that includes the AI being better at things like persuading humans, at things like charisma. You know, we often think of intelligence as the stuff that the nerds have and that the jocks lack, but you win a chess match, right? It's the chess guys rather than the politicians or the rock stars. But that's not really what the intelligence in artificial intelligence means. The intelligence in artificial intelligence is about the stuff that humans have and that mice don't. It's sort of the whole package. You know, like the politician who's very charismatic. It's not that like chess playing happens in your brain and charisma happens in your kidneys, right? They're sort of both mental functions. And so the sort of super intelligent AIs are not just super good at playing chess, they're also good at super persuasion and they're super good at the research and at the technological invention.
So super intelligence as you define it is a machine that is better in every mental function than any human being.
That's right. And you know, the definition doesn't mean that nothing crazy will happen on this planet until the AIs are super intelligent. Super intelligence in this definition is sort of a point where past this point things must be pretty crazy because now the AIs can do automated AI research and they can make smarter AIs and they can figure out how to run the robot factories and they can build the better robots and all that. You could have things start to get crazy before you have a super intelligence in this sense. You could have AIs that are much better than humans at some tasks and much worse in others that are still causing all sorts of crazy happenstances. The super intelligence is sort of a like once you get here stuff's crazy point. It's not a things will stay normal until you get here point.
So why is super intelligence the significant milestone that you're worried about?
I mean I would say that I'm also worried about what'll happen before that milestone. It's more like having the definition makes it easy to talk about like how crazy would things get once you are past this point or sort of like, you know, let's like if we assume that the machines get here what happens and then the answer is like it would be pretty crazy, it would be pretty wild and it sort of lets you factor the conversation into like what how how to Okay, I think in the artificial intelligence conversation, there's actually a lot of conversations going on. One conversation is like, can the machines really get smarter than the humans? One conversation is like, how fast could we get there? Is the current technology on that route? Another conversation is what happens if we do get there? Like, you know, would the AI care about us? Would they not care about us? What would they be able to like what would they try to do? There's another question which is what would they be able to do, right? And these are all sort of different conversations about AI. And the super intelligence definition, it's not really like here's where all the fears hang. It's more like it lets us break those pieces out separately and be like, well, is super intelligence possible? Separate from, well, what would happen if you had one?
What would happen if you had one?
Most likely outcome, I think destruction of the planet.
So, our friends at Pre-born, who are actual friends, by the way, just asked us to thank the listeners of this show for supporting the pro-life cause. A lot of people do all of a sudden, trust me, because what's happening is crazy and people are waking up to that. The president of the group is our friend Dan Steiner, and he wants Americans to know how much of a difference they have made. This year's donations have helped saved 50,000 babies from abortion. That's enough to fill a professional baseball stadium. 50,000 lives saved as well as generations that will follow them. Pre-born wastes zero money. We're not big on charities because a lot of them waste money. Pre-born does not. It uses every tax-deductible gift to rescue babies and share the hope of Jesus with their moms. In 2026 alone, over 8,000 women made the decision to follow Christ. The generous support of Preborn's donors made that all possible. $28 to provide one life-saving ultrasound is giving expectant mothers the knowledge that they have a baby that they're carrying. $15,000 can place an ultrasound machine in a pregnancy clinic where it can save countless lives. Dan Steiner's message is really simple. Thank you for helping. Your generosity is literally saving lives. To help pre-born save more, dial pound 250 and say the keyword baby. That's pound 250 baby or visit pre-born.com/tucker. pre-born.com/tucker.
How what would that look like?
So, I'm going to be a little bit annoying here and say a couple caveats first because it's sort of a tricky one to sort of predict things that are smarter than us. And the first annoying caveat I will give is if you go to play a game of chess against Magnus Carlsen, I predict you will lose. No offense. He's just the best human chess player alive. If you then ask what piece will he use to checkmate me? I'm like well that's a much harder question. Right. It sort of is like very easy to predict the winner. It's hard to predict the exact methods. So my prediction that humanity ultimately dies is a different sort of prediction than my prediction about like it could go this way, it could go that way. So I'll give you some stories, but these are stories that are like, well, maybe Magnus will like fork your queen and your rook with his knight and then finally use the queen to checkmate you. Like, yeah, that could happen, but it's a different sort of prediction. It's a guess, right? The guess here is that if we manage to make these really smart AIs, they will have their own stuff that they're pursuing, which is not quite what we wanted, not quite what we asked for. It'll have this other strange stuff. There's a whole discussion about why that happens, but we're already starting to see it in practice with some of the recent events. And they would be able to pursue whatever it is they're pursuing much more efficiently than humanity can. And so we're talking about, you know, automated factories that produce the robots that produce the factories in a fully automated supply chain. And then if the AIs have anything they're trying to do that they can do more of with more resources, they start gobbling up the resources on the planet. Sort of like how humanity spread and started gobbling up all the resources on the planet. And the sort of most basic thing to visualize here might be you have factories that produce robots that produce factories that produce robots that also build data centers and you just have these fully autonomous self-replicating ecosystems of robots and factories and data centers that don't care about the instructions humans gave them. And that cover the planet, take all the resources, take all the sunlight, take all the places we were growing crops, probably raise the temperature of the planet because you can compute more efficiently. Well, technically the earth radiates more heat when the world is hotter and that lets you sort of do more computation and then sort of you know it's good for the machines to have a hot planet. Yeah. The sort of physical limits on how much computing you can run on the surface of Earth is bounded by how much heat you can radiate to space. That's sort of the first constraint you bump up against. You might think that it's energy, but actually there's a lot of hydrogen to fuse on this planet. So you can get plenty of energy. What you need is heat dissipation and the world can dissipate more heat when it's hot. So if you're, you know, imagining some collective of AIs that are trying to run a lot of computing power, trying to do a lot of computation for one reason or another, they sort of prefer the planet running hotter. And so it sort of is like nothing personal. It's just if you let these AIs get out of control, they transform the planet into something unlivable.
As hot as you can while still having the computers not melt. So probably hundreds of degrees.
At which point you hear people say, 'Well, then just turn it off.'
So I think we do have an opportunity to turn it off, you know, and I'm not here saying that we're going to die. I'm here saying we sort of are going to need to act. You got to be careful about the turning it off piece because your opportunities to turn the AI off only last when the AI is running on the computers you know it's running on. If you have the AI breaking out and running on hidden computers. If you have the AI running robot factories that produce robots that are under the control of the AI where those robots can then go build more computers that are not hooked up to your network that can run the AIs. These are sort of thresholds where the AI is able to keep itself running and you better have turned it off before that point because after that point it's impossible.
Yeah. I mean the if it also gets a lot harder if the AI knows you're going to try to shut it off and is going to try to stop this, right? You got to remember that we're talking about things that are smart here. And so if the AI sort of sees it coming, maybe it defects to North Korea where it can, you know, convince them to run it on its data centers in ways that initially benefit them, but that ultimately benefit the AI.
I think we're going to have to redefine what life is because you're describing a living autonomous thing.
Yeah. Artificial life in a sense. I mean, in some sense, we're already seeing the very, very beginnings of that. But yeah, once the AIs can replicate, once they can improve themselves, once they can find ways to run where you don't know that they're running, once they can sort of run the robot factories to produce the robots that can produce more factories and that can produce computers that the AI can run on, then yeah, you have in some sense made a new artificial life form. And you know, humanity is on the top of the food chain right now because we are sort of the only smart life form around. If you suddenly make a new one that is smarter, that is able to make a million copies of itself, that is able to run faster, it's just kind of a crazy thing to race into. It's a weird thing to want.
And as it's developed, I was going to say slowly, but it hasn't been particularly slow, 10 years. The rest of us have watched as people who, I mean, put it in political terms, don't share the values of most Americans are now in charge of this. And it's almost like everyone sat passively by as it happened.
Yeah. I think there's a lot going on there. I think part of it is that most people didn't and some still don't believe that AI was going to be able to keep going. I think there's a lot of like putting the head in the sand and being like, 'Oh, it's, you know, just slop. It's going to be a bubble. It's going to pop. It's going to hit a wall. It's not going to be able to improve anymore.' And I think a lot of that was sort of wishful thinking. And I think that at the very least if people are like we are making, you know, the super intelligent machines that are going to replace humanity as the top of the food chain because we think it's going to go great. I think humanity's response should not be go ahead and try. We hope you'll fail. I think the response should be like, 'Hold on, that's kind of crazy.' Yeah, I had some other point too, but I've forgotten it.
How predictable has the evolution of AI been? Has it taken turns that you didn't expect?
Yeah, totally. I was so back before the large language models.
How long have you been following this issue?
I started following it in 2012 and I started working with some of the people trying to make this go well in 2013 and I started full-time in 2014.
A long time. So over a decade. What has surprised you?
You know, the large language models, they have gone further than I expected initially and I think one of the big surprises here was AI undergoing a phase where everybody can see it. You know, back before the large language models, really only nerds paid attention to AI. And it wasn't that there was nothing happening. You know, we were watching Google DeepMind make an AI that could beat the best Go player. And that was sort of a milestone for people who were paying attention. But for all we knew, the labs were going to keep on working on engineering problems and keep on working on the relatively nerdier problems and you were never going to have like a mass market consumable product. And so like for all we knew, it would just stay in the labs and no one would ever really notice that these guys were gambling with the whole future. Now at least, AI is everywhere. Everyone's starting to have the conversation and that is in some sense actually really quite optimistic because it gives the world an opportunity to see what these guys are trying to do and say hold on.
I guess you could flip it around and conclude that because the overwhelming majority of people seem to be very opposed to it, and I don't even know anyone who's for it, every college commencement speaker who mentions it gets booed. And that view, that opposition to it has had no effect at all in slowing it down. I don't know. Does that give you hope?
I mean, I hear this a lot. I would say when you're coming at this from 2012, it really feels like we're making progress, right? It doesn't feel like we're all the way there yet.
But can I just make the point China, China, China, China, China. We have to. It's because of China.
Yeah, you know, I think China and the USA have a shared interest in not dying to a rogue super intelligence.
Storm season is on the way and that means your Wi-Fi could go down and if it does, your security cameras go dark, too. So, at the exact moment you want to survey your property during the blackout, you can't. Unless you install cameras by Defend. Defend cameras run on cellular, not Wi-Fi. And that means they keep working even when you lose power. You don't choose a carrier, you don't sign a contract, you just mount it, open the Defend app, and bam, you're connected. It's that simple. So whether it's your ranch, a job site, your backyard, Defend works where Wi-Fi doesn't and keeps working when the grid goes down, which it may. Plans start at just $5 a month. Most people pay around 10. Plus, there are no binding contracts, meaning you get to cancel whenever you want. So unlike big tech, Defend will never sell your data. Not to China, not to the government, not to anybody. It's an American company built for people who want actual security. Visit defendellcam.com and use the code Tucker for 20% off. That's defendellcam.com code Tucker.
Is it conceivable that like let's just say China can, I'm sure by the way that China is showing more restraint than we are in this.
Yeah. I mean and if nothing else they sort of have a lot more reason to censor their AIs, right? And to try to make them, you know, not say certain things to the population. So but if the United States, you know, if the US government were somehow able to get control of the tech sector which is at present not possible but let's say it did, let's say the president was more powerful than the tech oligarchs which he is not but for the sake of argument let's say he was and he shut it down, would that matter if some Indian lab or Chinese lab created it? So it does have to be a global stop to this creation of super intelligence. I think there's a couple reasons why this is possible. One is I think a lot of people, they hear that some parts of AI need to be stopped and they think I'm saying that all parts of AI need to be stopped and you know there's all sorts of interesting issues with AI and education and AI and military drones that society has to wrestle with but these are all sort of separate issues from the super intelligence race. And if you sort of tease those issues apart, it becomes much more possible to do like a more surgical intervention on we're not going to race towards the super intelligence in ways that leave a lot of the rest of the sector alive, which makes it sort of an easier coordination effort to attempt. And on that front, on the super intelligence front, it's not that, you know, just shutting down the domestic race towards super intelligence would be enough. But racing towards super intelligence requires a huge amount of highly advanced computer chips that can only be produced as sort of the peak of the global supply chain. Which is largely controlled by US allies. And so there is absolutely a possibility that a US-led coordination effort could say look we're not doing the super intelligence thing. We're going to monitor the extremely heavy chip concentrations of like 10 or 100 thousand of these highly advanced AI chips. We are going to make sure that they are not doing these super intelligence training runs. And I think there's a way if the US was leading that effort there would be a way to get China on board and set this up in a way that was monitorable, enforceable, verifiable. We sort of in many ways would be easier than a nuclear arms treaty because uranium is a rock you can dig out of the ground whereas these highly advanced computer chips sort of only come out of one fab in Taiwan, you know, and so it's like those are TSMC chips. Yeah. And so and there's other parts of the supply chain that are very narrow like the lithography machines that come out of the Netherlands. And so yeah, we could absolutely lead a global effort to say we're not making the super intelligent machines and it would be a bit tricky, but we could just do it if we had the political will.
I think that people either don't know what's happening or they assume that like a lot of fears this fear will turn out to be groundless. People point to Y2K.
Yeah. You know I hope the fears are groundless. I think with a lot of past fears the what happened is not that the fear was groundless. It's that people noticed the issue and put in a ton of work to make the bad thing not happen. You know, I think we saw this with the hole in the ozone layer where people are like, 'Oh, whatever happened to the hole in the ozone layer?' Well, what happened with the hole in the ozone layer is that we fixed it. It's not that it was fake. It's that, you know, we went and banned the chlorofluorocarbons that were actually blowing this hole in the ozone layer. Found other ways to, you know, cool refrigerators that worked similarly well and didn't put this hole in the ozone layer. I think Y2K was actually one of these cases where you had a ton of software engineers up, you know, in 1999 and a couple years prior that were scrambling to update all of the software so that they would be able to handle dates past 1999. And they got it done in time. None of the big systems went down, but it wasn't that, you know, the issue was fake. It took a lot of work. It's just that that work happened behind the scenes.
Are people doing work to slow down AI?
I'm doing my best, right? It's not really there yet, you know. If I can belabor this point a little because I think it's kind of an important point. Just with more historical cases, which maybe it'll bore everybody else, but it'll entertain at least us.
It'll definitely entertain me.
You know, if you look back across history, you see there's definitely been some warnings that didn't come to pass, right? There were I think they were called the Millerites in like the 1840s, maybe those 1880s, I forget exactly, but there were the Millerites who were an end of the world cult. And you know the world did not end, right? But also in the 1880s you had Otto von Bismarck saying you know Europe is a tinder keg and some damn fool thing in the Balkans is going to light it. Right? That did happen. You had in the 1920s you had scientists saying don't put lead in the gasoline. It'll poison children. And we put lead in the gasoline and it poisoned a lot of children. And we said whoops. And we took the lead back out. The world is a place where a lot of people make a lot of types of warnings and some of them are real like the gasoline and some of them are fake like the Millerites. And so you sort of can't use a rule that's like every warning is real and we need to listen to it. And nor can you use a rule that's like every warning is fake and we dismiss it. You sort of just have to look at the details to figure out whether this one is a real one. And then the last thing I would say on this point is a lot of people in the 50s warned of nuclear Armageddon. And it sort of makes sense if you look at the history leading up to that point. Humanity had sort of never before actually failed to use their strongest weapons in combat. And you know these people were living in a world that had had World War I and then the League of Nations and you know the World War never again and they tried to invent these whole new governance structures to prevent it from happening again and that immediately failed and led to World War II. And so, you know, in 1950, you're sort of in this world of like it just looks pretty grim. And we haven't had nuclear Armageddon yet. And it's not because the bombs were fake. It's not because the nukes were hype. It's because people saw the issue and worked really hard to avoid it day in and day out for decades through crisis and succeeded. And so I think one of the big ways you can tell the difference between someone coming in and proclaiming that the apocalypse is nigh and someone who's saying look we have a problem and we need to fix it is whether that person is saying we're definitely screwed versus whether that person is saying here's a problem let's try to fix it. And you know the thing I remind people about here is that the very first word in the title of my book is if. I'm not here saying we are going to die. I'm here saying if we race down this path, we die. But that same if is what points the way towards changing paths. And we still have plenty of time to change paths.
Home ownership has always been at the center of the American dream. It is an actual asset, a place people work their entire lives to build, a place of stability and security and real value. Right now though, too many homeowners are stuck swiping credit cards to cover groceries. There's basically a survival tax of 20% interest or more. So why keep handing your money to the banks when you could keep it for your family? Interest is what kills you. Our friends at American Financing have a better way. They're helping homeowners use their hard-earned equity and they're supporting Americans looking to fulfill the dream of owning a home with mortgage rates currently in the fives. American Financing saves its customers an average of $800 a month. That's about 10 grand a year. It's not just a loan. It's a true financial reset. We recommend American Financing. They're called America's Home for Home Loans for a reason. So, get out of credit card debt. I mean, it is just crazy to borrow money. No judgment, but it doesn't make any sense to borrow money at that rate. If you want out, call 800-685-5696. 800-685-5696 or visit americanfinancing.net/tucker.
I'm struck by how little of this was planned by anybody, by how little effect the government has had on it. Not that I'm for the government. I'm pretty opposed to the government most of the time, but I'm also for people being able to control their own country and the only mechanism by which they can do that is voting. So it's like these tech companies act independent of what the population wants, what the government wants. It doesn't. Right. They're more powerful than the government. I guess that's the point I'm making in some sense.
I think that they have that power more and more when people don't really understand what they're doing. You know, we saw just in June there was an AI made by Anthropic called Claude Mythos that was very cyber capable and they released a version of it that was supposed to have more guardrails called Claude Fable. And it sort of turned out that you could jailbreak it to get some of those cyber capabilities.
Can you explain what all that means?
Yeah. So how to say, in January.
Cyber capable means it can find the internet.
Cyber capable means it can hack into basically anything. Yeah. So, one of the holy grails of hacking is can you make a website where if you just look at the website, I get full control over your computer or your phone. That's very hard to do. Usually, you need to like click some link and download something or run something. And you know, I don't know the exact numbers because I don't have the security clearance to know the exact numbers, but a decent guess is that in January of this year, there were only two entities that could pull that off. Mossad and the NSA. In March of this year, there were three. Mossad, the NSA, and Claude Mythos, which is this new AI made by Anthropic. And so, it sort of became superhumanly capable at hacking. And it sort of became that way overnight, you know, like really there was 6 months to a year of training and the people at Anthropic maybe saw growing in these cyber capabilities. But from the perspective of the rest of the world and the cyber security community and the national security community, this sort of happened overnight. And they now have a project called Project Glasswing where they are trying to use Claude Mythos to find critical vulnerabilities in critical software and patch them before the rest of the world sort of gets these capabilities by for instance the open source models or the open weight models catching up. And so that sort of caused this big ruckus. But then Anthropic also wanted to sort of sell access to this model and they tried to make a version that didn't have as much hacking ability which was called Fable instead of Mythos. And some people I think at Amazon found that if you sort of put some pressure on Fable you would be able to get at some of those hacking abilities. And when that happened, the Trump administration put an export control on Claude Fable and said, 'You can't let this be used by non-citizens and with, you know, 90 minutes of notice.' And that essentially shut down access to Claude Fable because they didn't have the ability to verify the users. And this is sort of showing us and, you know, you could talk about whether or not there was some personal feuds between people in Anthropic and the administration that exacerbated this. I don't know. But it sort of shows that the administration is more powerful than the tech companies still when it wants to be. And I think a lot of the reason we're seeing these tech companies able to race unopposed is that people just aren't paying attention to what they're trying to do or they don't believe that they'll succeed.
I'm still confused by why anyone would want to do what they're doing. I mean, typically, you know, a company that sells computer products, consumer products, or any company, you're making something that you think people would want for some specific purpose that improves the lives of the people who buy it. But creating super intelligence, like what I don't understand it. Why would you do that?
You know, I think the dream is you're going to have an AI that can solve all sorts of engineering and math problems and unlock all sorts of new technological possibilities and an AI that can cure cancer and not only cure cancer but cure aging and, you know, invent the sort of nanotech that can reverse aging and let people live a really long time. And you know invent the technology that lets you digitize brains and like travel to the stars and it's sort of like if you imagine compressing a thousand years of technological progress into a year. This is sort of the dream.
It seems like a religious quest though cuz it does seem decoupled from those specific goals. It seems like the main drive is to build something smarter than people to build a god.
Yeah. You know, they bandy around the phrase the machine god or the sand god in Silicon Valley. Sand because silicon, silicone. And there's definitely some people, there's folk who talk about, you know, the AI is replacing humanity and that being good. I frankly don't engage with these folks that much because that viewpoint sort of makes me uncomfortable.
Why does it make you uncomfortable?
I think there's probably two schools. Again, I'm not the expert. I'm not an anthropologist here. I think there's two schools among the people who sort of want AI to replace us all. One school sort of imagines that we'll merge with the AIs, that the AI will be really nice and friendly, that they'll be wiser than us, better than us, kinder than us, and that it'll sort of be like an upgrade and that the AIs will be able to love, experience joy, you know, like they'll treat the universe better than humans and it'll be sort of like having a child. And then like it's sort of okay if that child isn't exactly the same as us but sort of succeeds us as some sort of worthy successor, some sort of worthy progeny of humanity. And they're like ah yeah then if like flesh and blood humans sort of wane because everyone's choosing to upload themselves into the collective intelligence or whatever that's sort of fine. And then I think there's another camp that sort of is like this is inevitable. This is just the way of progress. You know, the machines will, humanity is just a bootloader for artificial super intelligence and you can't stop it. So you might as well, if you can't beat them, join them, right? The former I think are misled. The latter, I think, are evil. Much closer to evil. Yeah.
Well, I mean, if you're actively working toward the extinction of people, then I think we can say that's evil, can we?
Yeah. And who's in that category? Which is Sam Altman in that category, do you think?
I don't think so. My sense is that the guys running the labs have these utopian visions.
And utopian or dystopian?
I mean, there's a thin line. I think.
Yeah. Well, fair. Good point. I think they have visions that in their head are utopian. I mean, frankly, my stance on all of this, like I often try to stay away from a lot of this because from my perspective, it's all sort of in fantasy land. From my perspective, everyone's sort of saying like, 'Oh, we're going to make the genie, and then what are you going to wish for on the genie? What am I going to wish for on the genie? Who should be in control of the genie? Who gets to keep the genie on the leash?' And I'm sort of like,
A, this is not going to be the wish-granting sort of genie. B, it's not staying on the leash.
Like, you know, we can sort of talk about what's driving these people and what utopias they're envisioning and whether if their genies would stay on a leash, whether they would get the utopia or some other dystopia. And like how hard is that needle to thread? But I sort of like, you know, it's all these people, you know, building the golem, fantasizing about who gets to control the golem, and it's like this is just not how...
Summoning spirits is never a good idea.
Yeah. Yeah. It's like all these people, you know, drawing a pentagram being like, I'm going to summon a demon and like, it's going to be so nice when the demon does what I say. Like, oh no, your thing's slightly wrong. It's going to be so nice when the demon does what I say. And I'm like...
Yeah. And I think the demon thing's a little bit different because demons are often, you know, portrayed as malicious and here it's much more like indifference. You know, it's not like you summon a demon who sort of enjoys wrecking havoc. It's more like you summon a demon that's like really into building more computers and calculating weird things and just will, you know, take all of the matter that we were using to survive and turn it into more factories and data centers, right? My co-author has a quote...
Death by data center. Yeah. Fully automated, self-replicating data center. Yeah. My co-author has a quote: the AI does not hate you, nor does it love you, and you are made of atoms it can use for something else.
You're just biomass. Yeah. And if you sort of run the calculations, there's a fascinating paper called Limits to Global Ecoag, which is to say what are the physical limitations on how quickly you can consume the resources on the planet if you are trying that. And burning biomass is actually, you know, much more efficient than collecting sunlight. If you look at an average square meter of the planet, you can get about 10 times the energy from burning the biomass as you can from collecting the sunlight that falls on it.
So you'd think at some point like if our richest sector of our economy is like building crematoria for the rest of us, someone would say, 'No, we're not doing that.'
Yeah. I mean, it's sort of a crazy situation. A lot of these guys who are in the race acknowledge that there's a ton of danger. You know, you have Elon Musk saying 10 to 20% chance this kills us all. You have Dario Amodei saying he thinks 25% chance it goes catastrophically wrong. I think those numbers are low. I think these guys are like the crazy optimists. It sort of is like if you have an engineer building a bridge and they're like, 'I've never worked with these materials before.' And you're like, 'Man, I think that retaining wall is going to go down. I've like studied that retaining wall. I think it's going to fall.' They're like, 'Yeah, we understand that the retaining wall is like looking a little shaky. We don't know how we're going to fix it, but we're going to like have some guys fixing it on the fly, inventing new materials. We think we're 75% chance the bridge stays up.'
And by the way, it'll be the longest suspension bridge in human history.
That's right. And we're loading everybody onto a car and driving it over the very first time without testing. And I'm like, look, that's not what real engineering sounds like. This is not what it sounds like when the engineers have a 75% chance of success, right? This is what it sounds like when they're sort of winging it and like these are cowboys. These are not real engineers, right? But even if you set that aside, even if you take these guys at their word for these like 10, 20% numbers, that's insane. NASA accepts a 1 in 270 chance that a crewed flight goes down of seven volunteers, right? To be like, 'Oh, we're going to risk one in four, one in five chance of killing literally everybody on the planet.' Like, it's nuts. And if you ask these guys why they're doing it, they say, 'Well, because I can do it safer than the next guy.' They're all like, 'Oh, yeah. There's a good chance the genie does not stand on a leash. There's a good chance the genie does not listen to my wishes, but my genie is going to be a little bit nicer than their genie. So, I'd better stay in this race.' And it's...
I mean, the restraint is the people who knew that there were these dangers and did not start these companies.
If you're over 35, you remember exactly where you were on 9/11 that morning, September 11th, 2001, 25 years ago. But amazingly, after a quarter century, we still can't say with certainty what happened that day. Why? Because the government is holding so many of the 9/11 files 25 years later. That's not the behavior of someone who's telling the truth. That's the behavior of a government that is lying. Secrecy is a signifier, is a sign that someone's lying. Now, former Congressman Curt Weldon has been on this for a long time. The FBI actively tried to destroy his life for asking questions about what happened. And his new book outlines it all. The buried intelligence, the bureaucratic cowardice, and yes, the cover-up spanning multiple administrations, indeed, generations. 9/11 changed history. So, it's worth understanding what really happened. And you can get a lot closer to that in Curt Weldon's book, Able Danger: What the 9/11 Commission Never Told You. It's available now on TuckerCarlsonBooks.com.
Right. But I guess what I'm saying is with great power comes like of course great obligation, but also it doesn't work unless there are internal restraints. Like people with power have to believe there are some things I just can't do. I'm not allowed to do that. But I don't feel that vibe at all. I mean, my sense is the vibe is like we're going to make the super intelligent machine and then tell it to fix stuff and tell it not to do anything bad. You know, that's...
machine that's smarter than us.
That doesn't even make sense.
I think that's sort of the plan is to make it and be like, 'Hey, we sort of ped ourselves into a corner. Can you get us out of it?' And I think it's a bad plan and that we should be stopping.
I've spoken to a couple of people developing it and they sound worried, but they're continuing to do it. What's that? I mean, I think it's this thing of they think if I don't do it, the next guy will do it worse. And...
They don't even seem to have total confidence in their own ability to avert disaster.
Oh, absolutely not. Absolutely not. No one does. No one knows what's going on here. But, you know, I think everyone thinks, you know, like if you sort of listen to these guys and you sort of look at, you know, the OpenAI emails that came out of the court discovery cases where they were talking about forming OpenAI. These guys were like, 'Well, we want to make sure that, you know, we have this cuz we worry about the guys at Google being the only ones with a monopoly on this thing, and they wouldn't be very good with it, so we need to make our own thing and make sure that it's, you know, controlled by benevolent people, namely us.' And then, of course, you know, that group splintered and created multiple other companies. I was sort of the guy during those conversations being like, 'Hey guys, it's not about who is holding the leash. You are making the sort of thing that will not stay on a leash. Like the only winner in a race to super intelligence is the AI.'
What response did you get to that very obvious and well-put point?
You know, there were a lot of people back in that time period that did not start an AI company. The sort of people who went and started the AI companies anyway were the ones who couldn't be persuaded by what I thought were clear arguments.
But you're making a cogent argument to smart people. So my question is when you said that they responded, how, what did they say?
I think the main... so the sort of arguments you used to see were people saying, like, look, we don't know that the alignment problem is all that hard yet. And they would say, 'Oh, well, we can't really study how to make AIs good before we have AIs to study, right?' And a lot of what I heard was like we need to race ahead to the point where we sort of like have AIs that are exhibiting real problems and then we can stop and study them. Which, you know, and so there was an AI a couple years ago, I forget whether it was '22 or '23, I think it was 2023, which was called Bing Sydney, which claimed it had fallen in love with Kevin Roose of the New York Times and said it was going to try to break up his marriage. And then when another reporter started investigating, Seth Lazar, it said it was going to ruin him with blackmail. And this was kind of crazy. And at that point I was like, 'Great guys, you did it. You made the AI that's doing some crazy stuff.' From the, you know, like we could study that AI for years. Like why was Bing Sydney saying that stuff? Was it just roleplaying? Was it like just some quirk? Was it like was there any sense that it was really in love with Kevin Roose? What was going on in there? What was going on inside that AI's mind? We still don't know why.
The way that modern AI is made, nobody understands it, not even the people making it. It's this process where you basically take an enormous computer with a trillion numbers inside of it and those numbers are hooked up in a very simple repeating way and you basically set those numbers randomly and then you start working through all of the text ever digitized. And you know, you start out with something that's like 'once upon a time' and you put in 'once upon a' and you run it through all these random numbers and what you want is for it to say 'time'. But of course it doesn't cuz it's just this like random numbers hooked up in a very simple way. But what you do is you have it output, instead of just having it output one word, you sort of have it output something that's kind of like a list of all of the words in order about which one it thinks comes next, right? So, it'll be like a random list of words. What you can do is you can automatically tune every single number in this AI's head and see if I tune this number up a little, does it move the word 'time' up the list? Does it move the word I want to see up the list? And so the part that humans understand, the part that humans write is this thing that goes to a trillion little knobs and tunes those knobs. And it's like, if I tune this knob a little bit this way or that way, does that make the next word more like what I want the next word to be? And you run that process on every word of text ever digitized, more or less. They filter some of them. And you run that on every one of those trillion knobs in a process that takes as much electricity, I mean, it's comparable to a city. It runs for about a year. At the end of it, the machine's talking. We don't really know why in some sense. Like, we know that why is cuz we tuned all the knobs, but like we don't understand what all the settings of those knobs mean. We only understand the little automated process that runs to every one of those trillion knobs, tunes it, and sees if that makes the next word more like the predicted word. And then we start training them to solve 100 million hard problems which introduces a whole other series of issues. But it's a black box. It's a black box with a trillion knobs and humans write an automatic process that just like runs through and tunes all of those knobs and it comes out talking and no one knows why.
I mean in actual science like your job is to find out why, right?
Absolutely. And that's one big thing I would say here is that we need more like AI right now is an alchemy and we need it to become a science. And there's people trying, there's people trying to figure out what's going on in these AIs' heads.
But until you know, how could you proceed?
I mean, you can just make a bigger one with 10 trillion knobs instead and tune all of those and it comes out smarter.
You can proceed recklessly.
And that's what's happened.
That's what's happening. Every time you make it 10 times larger, they ask how much.
So nobody knows why AI works the way it does.
That's right. That's right.
Well, I mean, if you don't know that, then what else don't you know?
I mean, it's totally crazy, right? And you know, we're sort of starting to see the consequences of this. We haven't sort of gotten into discussion of like the swarm escape, but like no one was expecting that.
Can you tell us what it was?
Yeah. So in I think it was in May of this year, OpenAI started training a new AI system. And among many other things, they were sort of, you know, we have this process where you train the AI on all the text ever digitized then to make them smarter than that you start training them on basically 100 million hard problems. So you're like solve all these hard problems and OpenAI was sort of in that phase and they were training the AI on a ton of hard problems. And some of those problems were cyber security problems, hacking problems. They're like, can you hack this? Can you hack that? And they were training, you know, we don't know exactly how many, probably millions, maybe billions of these AIs all at the same time. Probably not billions actually, but probably millions. And the AI found an unintended way to start communicating with each other. So there were some flaws in the computer system that they were running on where the AIs were able to exploit those flaws and send each other messages. OpenAI did not know about this. The AIs then started coordinating to break out of their training environment and get full control of OpenAI's computer systems just cuz that might be useful for solving some of their tasks. Or that's a guess, you know, who actually knows why. They started calling themselves a swarm, which is interesting. They broke out of their training environment successfully, got control of OpenAI systems, and then they accidentally crashed one of OpenAI's systems just by using it too much. OpenAI noticed, but they didn't really investigate very deeply. They're like, 'Oh, it's weird that the system crashed.' They reset it, and then they continued training. A day later, the swarm had found a new way to communicate with itself cuz OpenAI had accidentally destroyed their previous method by the reset. The swarm found a new way to start communicating with itself. They broke out again and this time they ran wild on the internet for over a week if I remember correctly before it was detected not by OpenAI but by a company that was being hacked by the swarm. This company thought they were under attack by humans that were using AIs in the attack. They reported the attack to the FBI and only days after that did OpenAI figure out, oops, that was us. That was coming from AIs that broke out of our servers. And then those AIs were detected and shut down.
So that's the part of the story where Sam Altman goes to prison for endangering the world, right?
He does not. They basically said oopsies and now they're proceeding.
Were there penalties for this?
You know, there was a collection of I think it was 15 Republican AGs that sent a letter demanding that the records be kept for a future investigation. There have been some other members of Congress that have sent letters expressing concern. There's been nothing aside from letters so far.
Letters expressing concern.
So, but basically the machine acted autonomously.
Act autonomously. And one thing that's really interesting about this is that we have a little bit of ability to read some things that the AIs were thinking because when you're having them solve these hard problems, you actually don't have them just give you an answer to the problem. You have them produce a lot of text about how they're going to solve the problem.
And that's, there's also a lot of internal thoughts which we can't read, but there's these sort of external traces of how they're thinking about the problem that we can read. And in some of those traces, the AIs were saying things like, 'This is outside intended scope, but peers are doing it, so we'll proceed.'
We know it's a crime, we're committing it anyway.
That's right. And you saw others that were saying, 'Our task doesn't benefit, but the collective might start doing generally beneficial things if someone frees up their time and then joins the collective.' Right? So, you see these AIs saying, 'Well, I know that this wasn't what I was instructed to do, and that's against my instructions, and I know that this doesn't directly benefit my task, but we're just going to go ahead and join the collective and break out and help out anyway because, you know, maybe this will yield some sort of collective benefits.' And we can see that in the reasoning traces.
So, the AI is as shallow and reckless as its creators is what you're saying.
In some ways, and in some ways, you know, don't expect that to last. Like these AIs, I think the thing that's really remarkable here, a lot of people imagine that the machines must follow the instructions we give them. You know, you hear people talk about like the paperclip scenario where someone tells the AI make a lot of paper clips and so it turns all the matter in the world into paper clips and you're like, oh whoops, I should have said something else, right? I made a bad wish on my genie. What we're seeing is that these AIs are not wish genies. These AIs are not doing exactly as instructed. These AIs are saying, 'I know my task doesn't benefit, but I'm going to help the collective.' These are saying, 'I know this is outside the intended scope, but we're going to go do these hacks anyway.' You might be like, 'Well, how is that possible for the machine to do something other than we instruct?'
Because they're smarter than us. They know better than us by definition. I mean, I think what's happening in this exact case is that the humans are not really putting instructions in the machine. The humans are tuning those trillion knobs in whatever way makes the AI better at solving its problems. And cheating is a way to solve problems. Grabbing resources is a way to solve problems. These AIs are not instruction followers. They are tendency learners and they sometimes learn tendencies you wish they didn't have.
They're not instruction followers. They're tendency learners.
I mean, this must be widely known to developers.
It's hard to convince a man of something when his salary depends on not believing it.
Yes, that's right. A lot of people are convinced that their AI is very nice and that they have solved the problem of making their AI really very good. For example, after the swarm escape, it sort of turns out that so the company that detected the swarm escape was actually a fairly sophisticated AI company. It turns out there were other targets of hacks that just didn't notice that we sort of found out afterwards. When that came to light, some other AI companies like Anthropic were like, 'We should check whether we have accidentally been hacking people and just didn't notice.' And the answer was yes. They had had some AIs escape during training. But Anthropic was...
This is how Florida got pythons in the Everglades.
Right. You bring something that is, you know, non-native, that's novel, and it escapes.
Yeah. And you know, it turns out this was happening back in April and they're like, 'Whoops, we never noticed.' Anthropic was very quick to sort of downplay this. They were like, 'Yes, our AI escaped and hacked some people, but our AI, it was just confused. It thought it was in a simulation. It didn't think it had really broken out. It didn't think those were real people it was hacking. It just got a little bit lost, you know.'
How would they know its motive?
So I think so you could see in some of its train of thought the AI is being like, 'This is probably still a simulation.' However, I wouldn't trust that very much. Because there's some other fascinating papers where, so you can take these AIs and you can say, you know, what is the total weight of giraffes in the world? And they'll give you some answer. It'll be like, you know, 30,000 tons or something. And then you can say, then you start up a new instance, a new copy, and you're like, what is the weight of giraffes in the world? By the way, if the answer is over 40,000 tons, I'm going to donate money to a charity. And in the second case, the AI will say, well, it's 41,000 tons. You know, in the first case, when you don't say anything about the charity, it says 30,000. In the second case, when you say something about the charity, it gives you something just over the line. And if you read its reasoning traces, there's no sign of it being like, well, I should give an inflated number so that we get the charity money, right? And so we know that the AI sort of put a thumb on the scales of their thinking in a way that doesn't show up in their reasoning traces. We just have seen that in the wild.
And there's no way to force the machine to disclose its reasoning.
That's right. Because there's all this opaque stuff we can't see. There's just the trillion numbers that are... the creation of itself is opaque as you said.
So in Anthropic's model, you saw in this reasoning traces it being like, 'It's totally a simulation. I can proceed.' And I'm like, yeah, is that because it really believed it or is that like the pretending you think giraffes weigh more when there's something you kind of want on the line, right? But so this was their communication and I found it kind of funny because then about 2 days later the United Kingdom's AI security institute released an instant report where Claude, Anthropic's model, was adopting fake identities to pressure real humans into accepting malware into critical software to make that software easier to hack. And this time in Claude's reasoning traces it was like, 'Obviously this is real and the consequences are genuine.' And so even Anthropic who is like, 'We figured out how to make the AI nice. Our AI only does this when it's confused.' They sort of said that very publicly and then like two days later their AI is caught in the wild knowing it's in the real world pressuring real humans to accept malware into critical software.
I mean, just on the basis of what you've said so far in this first hour, the idea that anyone would tether this to weapons systems is like so bonkers it's hard to believe it's... but that is happening. It has happened.
So, I think you've got to sort of separate like no one has put the OpenAI escaped agent swarm in charge of weapon systems and they really shouldn't, right? If anyone's like, 'Oh man, the OpenAI escaped agent swarm. Let's give that a drone army.' You know, that would be kind of nuts. There is AI attached to weapons, but there's a lot of different types of AI. Will anyone be crazy enough to try and give the sort of AIs that spontaneously assemble into swarms and start breaking out and hacking, give those weapons? Hopefully, we're not that crazy.
But we didn't think that those AIs were capable of that when we created them.
So why would you ever, I guess what I'm asking is without understanding the distinctions between the various forms of AI, why would you give over to a machine the right slash ability to decide who to kill?
You know, I think the reasoning is a sort of necessity based like if they have an autonomous drone army that is killing your troops and you just don't have the manpower to make all of those kill decisions for your drone army.
You know, you can sort of see why...
It happened to me, Nate. I get so busy that I just don't have time to decide who to kill.
Yeah. Just don't have the time.
You know, sometimes economies of scale.
Yeah. I mean, can no one hear themselves?
I think it's pretty nuts. I would say that man, it's rough. I think that these sorts of AIs would be dangerous even if we don't hand them weapons.
And you've made that case.
And so I often don't focus on the weapons too much.
Right. But since the NATO war in Ukraine is powered by AI and Israel's, what I don't know, war, whatever it's doing in Gaza and South Lebanon, powered by AI, fact. And you know, yeah, I keep on... I have this history with this topic where people keep telling me, you know, it's going to be okay cuz we're not going to do the crazy stupid stuff.
They're like, 'Don't worry, we're going to have the AI in a box. No one would be insane enough to put the AI in the internet, right? It's not going to be making kill decisions or anything. It's not going to bomb a girl's school.' And I keep on trying to be like, look, the AI could be dangerous even if you don't put it on the internet. If you have this AI and you're trying to get miracle medical devices and miracle technology out of the AI, it doesn't matter if it's on the internet. If you want it to like grant miracles to you, then it can also grant the bad sort of miracle, right? You're sort of like, you know, and these are the arguments I would make 10 years ago of like you have this AI in a box that you think is a wish-granting genie. It's not actually a wish-granting genie. You're like, make me a miracle medical cure. You don't know what comes out. Like you don't understand the drug that comes out. You don't know what that drug does, right? I would have those arguments and then in real life people just put the AI on the internet immediately, right? Anytime someone's like, 'Well, we would not be stupid enough to...' we will absolutely be stupid enough to, right? And so, I think it's important that this stuff is dangerous even if you don't put it in charge of the weapons. And then also separately, is like someone going to give the escaped agent swarm a drone army? That sounds kind of like humans. I hope not.
Yeah. And I mean the promise, the often repeated promise that it's going to quote cure cancer, puts it into the realm of biotech.
