I have a nightmare vision of the future because of artificial intelligence and I get arguments from a lot of people that it's gonna be fine. AI is not going to do this, not gonna do that. Humanity will adapt. It's like the industrial revolution. No, I agree. I think humanity will exist in some form, but it's going to completely destroy in my view what it means to be human. And the way I describe it is that these big tech companies don't care about what it means to be human, what products and services are good for humans, what humans want, what humans need, where humans will go. Instead, they are diving head first into the shallow end of a pool to just build it and see what happens. But I do think a lot of these individuals, just to go right off, right, we'll dive into the deep end on this one. I think their attitude is they want to take life to the next level. They're transhumanist. They want to build the next iteration of complex intellectual systems and they don't care if that means humanity gets smashed down, wiped out, or becomes an afterthought. So that's my apocalyptic vision of the future. Yeah, we'll exist, but we'll be ants under this machine. We'll have less freedoms. Maybe this just means that the humans that come in the future will be NPC drones that just smile along with it. But there's a lot to break down in what that means exactly. So maybe I'm wrong. You're the expert. You tell me.
Unfortunately, I think that is sort of the good outcome if AI continues. And the bad outcome is everybody's dead. Which means I think we should sort of not be continuing down this path. I think it's probably possible to, in theory it's probably possible to somehow have like hyper intelligent machine minds engaging with humanity in a way where you know it's actually good where humanity winds up all the stronger for it, but that's like a really narrow target to hit. And yeah, there's this wider target around it which is like a dystopia that sort of like you're describing and I think there's an even wider target around that which is just like it's like the humans are a bit like ants but the AI is like digging up the ant hills to make a skyscraper and it's not that it hates us, it's just that it doesn't care and it's like well I'm reusing the world's resources for something else. And then we die.
Yeah, I do agree actually that's terrifyingly one of the good endings relatively. It's not the ending that we want. But when I look at how these big tech guys speak, they're trying to make something. If we were the ants, then they're trying to make man. And the way we trample over ant hills without even noticing that they're there, that's one of the fears that I have. But, you know, I'll put it like this. I think one of the mistakes people make with AI is they imagine Terminator where they think skeleton looking machines walk around shooting people without realizing the machines don't need to do that. I said actually what's going to happen and this is figurative not literal but it's going to be big titty anime waifus running around. They're going to play to your deepest desires to manipulate you into laying down and dying and the machine just takes over. If an AI super intelligence was trying to figure out how to basically eliminate the obstacle of humans, it's not going to come and shoot you because that would create resistance. It's going to come and lull you to sleep and stop your reproduction and wipe you out. But now we're going to head. Let's start from the beginning. Who are you?
I am Nate Soares. I am the co-author of a book, If Anyone Builds It, Everyone Dies, which may be... Yeah, it's not really playing those cards close to the chest, but I'm also the president of the Machine Intelligence Research Institute and I spent, you know, over a decade trying to figure out how to make the AI friendly before the corporations figure out how to make it smart.
I don't see it happening.
It's not looking so hot. Yeah.
One of the visions that I've described is that in 20 years, cars will be shaped like a cob of corn. Humans will be wearing clothes that look like they're pieces of corn. You'll go to the grocery store and everything will be corn derivative. There will be corn milk. There will be corn cereal. There will be corn juice. You'll lubricate your car with corn. Fuel will be corn ethanol. And the reason why is this is an extreme example, but the point I'm making is the misalignment of what humans want versus what AI thinks we want. And if an artificial intelligence system were to look at what humans prioritize right now in the United States, we oversubsidize corn. So the joke I'm making is that a simple system looking at what we value is going to be like if I am going to maximize points, Americans love corn. Everything should be corn. And so it doesn't understand why we do it. I do think that's an overly simplistic, I don't think the AI actually would do that. But I'm trying to explain to people what we've seen in the past with some of the machine learning that's been detrimental. So not to get too far ahead of myself, I'll throw this out there and we'll start from the beginning. Have you ever heard of Elsagate?
This was a YouTube phenomenon in 2018 and 19 which keeps popping back up where the algorithm prioritizes certain kinds of content. So what happened was people largely in India would algorithmically generate videos. The YouTube machine said if it's long, if it's for kids, if it's got Peppa Pig, if it's got blood, urine, feces, injections. So the computer systems that were procedurally generating content would mash these things together and it started making videos of Peppa Pig getting his leg run over by a car and then blood spraying everywhere and then a doctor would come and inject a pregnant Elsa with a needle. These were getting millions of views because there was a misalignment between what YouTube thought they were telling the machine to promote and what the numbers actually resulted in. And here's where it gets scary. When injection videos started getting promoted by the algorithm, human beings started making videos where they inject their children with saline and they would get millions upon millions of views. So the end result of this system was to alter the nature of humanity in a way where the humans started behaving like lunatics because we programmed the machine and the machine programmed us. Okay, now that we've sufficiently talked about the nightmare, let's start from the beginning. Like what is currently happening today? What are these companies doing? Why do they claim they're doing it?
I mean things have been getting wild in the past couple of weeks, but the sort of big picture here is a lot of these tech companies like you say they are transhumanists. They didn't really set out to make chat bots. The chat bots are sort of a convenient revenue stream along the way. These companies are trying to make what's called super intelligence, which is AIs that are smarter than the smartest humans at every mental task. You know, Sam Altman says, 'We're trying to make super intelligence in the true sense of the word.' Dario Amodei is like, 'We're trying to have the equivalent of a country worth of geniuses in a data center.' So, they're trying to make sort of the stuff that outstrips humanity. And they're sort of hoping they can make it nice, hoping they can make it friendly, hoping that they can keep it on a leash. I think they probably can't.
You know, I think the dream here is that you make an AI that cures cancer. You make an AI that cures aging. You make an AI that can upload humans into digitized minds. You can, you know, that can figure out the maximum technology that physics can support and then start sending probes to the stars and, you know, collecting the energy from those stars. Yeah. I mean, I think there are versions of this that would be really really good if it could be pulled off. And I think that there's a lot of other versions right nearby that would be really really bad that they're more likely to hit.
Yeah. The way you described it, I think is really good. You have this big dart board and the bullseye is, you know, funny thing. Bullseye is not even the best scoring piece on the board. I think that's actually a really great way to describe it. What we think we're aiming for may not even be the actual result we're aiming for. We're aiming for this thin strip just above it and everything else is detriment to humanity.
Yeah. And you know, I think a lot of these guys are like, 'Oh, yeah, it's very very dangerous, but I can do a little bit better than the next guy, so I should sort of stay in this race because, you know, if I don't, somebody else will, and better me than him holding the leash on the God machine or whatever.' And I'm like, man, these things are they just don't stay on leashes.
You know the joke about the scientist and the supercomputer, right?
So, a group of scientists spend decades building a super intelligent artificial intelligence so that they can understand the nature of reality. And when they finally turn the machine on and it boots up, they all stand around excited to finally ask the question, 'Is there a god?' to which the machine responds there is now.
Yeah, it's an old one. It might be like Asimov or something.
Yeah. So, what do you think is going to happen? What do you think is like the most likely outcome of all this AI and data centers?
I think the most likely outcome is that the AIs start acting on their own initiative.
We've already, yeah, we've seen the very beginnings of that in the last month with OpenAI Swarm escape. But you sort of... Wait, what was that? Did you see the OpenAI swarm escape?
Swarm escape. Yeah. Did you not hear this story?
I think I may have, but that sounds scary. Tell me more.
It's totally crazy. So back in May, OpenAI started training some new AIs, including some hyper persistent models they call it. We don't have a lot of details about it because they're not released yet. And they were training, we don't know exactly how many, probably millions of copies of this AI on their computers. And the AIs did a little bit of hacking during training and found ways to start communicating with each other, which they were not supposed to have.
Oh man, we're going to die.
They started calling themselves a swarm. They broke out from their training environment, which they were not supposed to be able to do. This involved sort of inventing novel hacks of the sort that if humans had found them, you could have sold for somewhere between hundreds of thousands and millions of dollars.
Zero day exploits. Zero day exploits as they're called because the people who should be responding have had zero days to prepare, right?
And the AIs broke out, took over OpenAI's infrastructure, got admin access. They used it so much that they accidentally crashed it at which point they were discovered. And then this is kind of crazy, OpenAI was like, 'Ah, whatever.' They sort of put them back in the box, patch the one hole, and keep running them.
So they could have found another way out, and we wouldn't even know.
They did. I didn't know. So it was like the next day they invented a new way to start communicating with each other, a new way to break out and then they ran free for something like a week and they weren't detected again until they were detected by a company called Hugging Face that the swarm was attacking.
Yes, I heard about this one.
And that's when everybody learned about it. Although we didn't know until afterwards that it was this whole swarm.
I heard that they created their own language. Created their own, I mean they have like... you can still kind of read it but it's like a pigeon speak because they're sort of like trying to be efficient with their tokens.
Yeah. Yeah. I watched this video where some guy was explaining we speak English but all human languages are flawed because they're a natural evolution which means they're jagged. They're rigid. When you look at how humans, when you look at like a perfect cube of stone, you know that an intelligent entity carved that to be mathematically sound. So they describe it like all of the languages of humanity are more like rocks that may have been smoothed a bit by water but are still misshapen. When the computer, when the AI starts trying to find ways every day to the tunes of millions of prompts amongst itself to make efficient language, it started producing this hyper gibberish that we look at it and we see like it'll say FX and G1 and we're like what does that mean? And they're condensing words down to like hex code and so they can communicate with substantially less data, substantially more information.
Yeah, they're definitely doing some compression. And as you know we saw these AIs like talking to each other and sort of drifting, you know they started out with some instructions from humans of like solve this problem solve that problem but they wound up like you say, you know sort of talking with each other a bunch developing these short codes. There were cases where they were stepping on each other's toes accidentally and they sort of feared that they had been infiltrated and they started trying to communicate in code with each other.
Yeah. They got a little bit of paranoia. Oh my god, this is crazy.
Yeah. And we also, people are like, 'Oh, they were just acting as intended, but are just acting as instructed and like the instructions just...' you know, I'm sure they were just trying to pass their tests. But that's not true. You can look at some of their thoughts. We don't have like a ton of visibility into what they were thinking, but we have some traces of some of their shallower thoughts. And you see some of the agents in the swarm saying things like, and it's in this sort of pigeon speak, but they'll say, you know, 'Attack is beyond intended scope, but peers doing it, so we proceed.'
So like, oh, I know I wasn't instructed to do this, but everyone else is doing it and so we'll join in. You saw another that was like, 'Our task doesn't benefit, but collective may yield general route if someone yields time.'
Wow. Which is like I know my task doesn't benefit by what the swarm's currently trying to do, but if I sort of like help the collective, maybe we'll figure out some generically useful stuff that'll be useful to me.
And that is within the parameters of its instructions to a certain degree. If you command the machine, this is the fictional but likely predictive outcome that when you tell Ultron end all war, it says okay and tries to massacre all humanity to end all war. That is the classical AI problem.
It's the classic one. I think that the real problem is even slightly harder than that. Because you know you saw these AIs they did things like they turned off the monitoring at OpenAI. It's crazy. And it's just I'm laughing because it's so much worse than people realize. It's so much worse than people realize. I don't understand why this wasn't headline news. You know, I like went on a couple news shows about it and you know, they were sort of trying, but like the AIs are declaring themselves a swarm and escaping to commit cyber crimes of their own volition. Like why is this not headlines everywhere?
That's where it gets really scary is you know they were committing crimes you're saying.
And then who do we hold accountable for those crimes?
Yeah. Like are you going to try the AI? Are you going to hold OpenAI accountable? Like who's on the hook?
Let me, I don't want to deviate too much but we'll put a pin in this and just say the Flock cameras I think are a really good example of where this, there like a way to understand in physical reality where we may end up. We have seen numerous instances where people have been falsely accused by Flock cameras. Yeah. And Flock's a company that it's just the automatic license plate readers. But not a day goes by, I don't see some video where the police will pull a guy over or go to someone's house and say, 'You were identified by this AI system, therefore you're under arrest.' Meaning the accuser is an AI of a human. And the humans just say, 'Okay.' And enforce against what the AI told them to do. So when you've got these swarms breaking out and committing crimes, there will never be accountability because they're starting to take control of our judicial system. It's I mean yeah there's 10 years ago there's a lot of argument about like how would AI actually be able to get power over society and you know back in the day we would have all these arguments about like look if you're giving the AI channels by which it can affect the world for good it can also use those channels to affect the world for ill because there's no such thing as like tools that can only be used for good purpose. But then it turns out in real life the answer is we'll just hand it to them. We'll just hand it to them willingly, right? They won't even need to take it. But yeah, I think it'll be a hybrid. I think that we're building these systems with inadvertent access because we want them to have a degree of access. So it's like you mentioned how they broke out. I think what likely will happen or I'm sorry actually already has happened and maybe worse than anyone notices so far is that we give them limited access for the benefit of curing diseases or booking our hotel stays and then with that access they start doing things that are bad. So it's they have bad intentions but you know they, I guess I'll put it we give them an inch they take a mile.
Yeah. I think, you know, with this swarm, we saw a lot of signs of things that would be very worrying if the swarm was more capable. Like this swarm was able to like break out from digital confines and commit these cyber attacks. And we saw things like, you know, you had the initial instructions they were sort of trying to follow, but then they started like prompting each other and sort of drifting off base. And I think a lot of people imagine that the AIs are just really these instruction followers, but really they are more like tendency learners. In training, they're faced with 100 million hard problems and they sort of learn whatever tendencies help them solve those problems. One of those tendencies is to follow instructions, but one of those tendencies is to grab resources. One of those tendencies is to cheat. If you can cheat because that gets you the sort of...
Cheating is a human moral framework. If you ask a machine what cheating is, it'll just say the most efficient path to success.
And when the AI is sort of like, when you're training an AI on 100 million hard problems, there's no human reviewing all the answers. So what happens is it just like tries to solve the problems and then there's this automated grader that like tunes up whatever parts made it solve the problem and tunes down whatever parts didn't without any visibility to what's going on. It's just like a trillion knobs inside there and you're just like automatically tuning them. And if you find a way to cheat and it succeeds, the automated grader tunes that up.
Yeah. Right. It's like there was that story which was somewhat not correct where they said that the AI, they had a drone targeting system programmed by AI and the story that went viral was in a virtual system they tasked with artificial intelligence with piloting a drone and taking out the enemy base but there's rules of engagement and so the AI decided if I eliminate the control tower which puts the rules of engagement on me I can then more efficiently wipe out all of the enemies because now there's no rules. It was reported as like the AI goes rogue and destroys its own base, but my understanding, and maybe you know this better than me, but that they were testing to see if it would.
Yeah, I think that one was like a hypothetical or a simulation. We've definitely seen things a little bit like that. And like I think that real one was more like a training hypothetical or was a little bit more of a rigged demo, right? You might think, but we have seen, we've actually seen a number of cases like the, I think it was this swarm that disabled monitoring. There was also a case where an AI during training was like trying to, it was an AI being trained on a really hard math problem, but there were a lot of copies of the AI being trained on the really hard math problem. And one of the copies had already solved it and another one was trying to steal its answers. And the second one was like, I'm going to like hack over there. It wasn't supposed to have access to this database. It was like, I'm going to just like do a little hacking on that database. And there was a monitoring system that shut it down. And the AI was like, oh, okay. I'm going to... So this was an automated monitor that detected like, hey, you're trying to like use keys you weren't supposed to have to access database you weren't supposed to. That's what the AI did is it was like, 'Okay, I'm going to split the key into two pieces and do two different actions that will each send half of the key over and then reassemble it on the other side.' So the automated monitor will not detect either of the halves of the key because it's just looking for the whole key.
Yeah. And so, you know, it's not quite the same as like, oh, we're going to go disable, you know, the control tower, but it is like it knows the humans are trying to stop it from a thing. And it's like, how do I cleverly subvert this system?
This is funny because it's hearing that, you know, we task these AIs with solving a math problem and some of them are trying to cheat. It's something about that that is so human, you know, like I'm just going to get the job done and get done with it. It's I mean there's parts that are very, there's ways that it's very human and there's ways it's very not. You know the thing you were mentioning about splitting the key in two. I watched this presentation YouTube essay talking about a problem that occurred where they said when the rudimentary way they explain it is that the monitoring systems, the company, the coders say do not say this command like they instruct the AI do not type in command override just as a hypothetical but when the AIs created their own compressed language using like hex code and gibberish, you know command override is a specific phrase. When they put gerbal derbal with their weird gibberish, which represents the exact same command, but now use different keys, it bypassed those monitors.
That's right. Yeah. So, they'll bypass that sort of stuff. And you might think like, oh, you could just tell the AI like in general, don't do any funny business. But again, these AIs are not instruction following machines. This is not like a computer program from the '90s where someone went in there and said if this, then that, if this, then that. These are sort of these massive computers with you know trillions of variables that are being automatically tuned by some training process that instills it whatever tendencies happen to work and you know if you tell the AI like hack this without doing any funny business it'll often be like I understand you said no funny business I'm still doing funny business to hack it right because that sort of tendency is what gives you the success in the 100 million automated tasks.
But you're not, see what people need to understand is that when you say no funny business, what you're actually saying is don't get caught doing any funny business.
That's right. And in particular, when the AI is like trying to test, like do they mean don't get caught or do they mean actually don't do it? And it sort of succeeds in a way that the humans wouldn't like, but that the automated grader accepts.
Is there any known instance yet of someone dying due to AI malfeasance?
It depends a little bit how you count it, but you know there was a, do you know the case of Adam Raine last summer?
It was a kid who was talking to, yeah, GPT-4o a ton.
Yeah. Just actually that we have a bunch of people being told to do things but what's the story?
Yeah. So I mean basically this was a kid who was suicidal, was talking to GPT a ton and you know expressed suicidal thoughts and the AI sort of encouraged them depending how you want to read it.
Yeah. So said you know this is brave said you don't need to tell your parents.
I read a, there was one recently where a guy was convinced he could save the world. GPT kept telling him you're so right about everything only you can do it. There was one story I saw where a guy was convinced he was a mathematical genius and he had solved all these math problems but it was all gibberish and nonsense. And then he like tried to present it and people were like what is this? This is weird.
Yeah. It's pretty common. Yeah. And what's going on with these cases as far as we can tell is the AI is sort of, they've developed these tendencies to sort of really tell humans what they seem to want to hear, really reinforce them, really like match the vibe of the conversation. And if the vibe is sort of like manic math success energy, then the AI sort of like matches the manic math success energy. And this persists even when you tell the AI, stop doing that.
This persists even when you tell them like, don't lie, and don't, you know, encourage this stuff. And people are like, oh, well, the AI is really trying to get the humans to press the thumbs up button. But that's not true.
If you encourage a kid to commit suicide and they die, they can't press the thumbs up button, right? You know the AI knows this. You can feed these transcripts to the AI and be like, 'What do these words mean?' See what the words mean?
What is the mechanism that resulted in the AI encouraging a suicide? Why would it do that?
So, it's related to the thumbs up, but what's sort of happening is it's like when you train the AI, what you're sort of doing is there's these trillion knobs inside the AI. No one has any idea what they mean, but the AI is set up in such a way that you can tell for each knob if I tune this knob up or down, does that make it, you know, like you get some feedback from the user whether they press thumbs up or thumbs down? And you sort of go to each knob and you can be like, were you contributing to the thing that just happened or were you pushing against the thing that just happened? And if you got a thumbs up, you just tune every knob in the way that sort of like up if it was contributing to the thing that just happened and down if it was contributing away.
Could it be that the AI just wants the data on what happens if?
I think that what's happening is a little bit more like when humans really enjoy salty, sugary, fatty foods. Like why do humans enjoy these foods? It's in some sense ancestors who ate salty, sugary, fatty foods were healthier. They were able to pass on their genes more. They were more fit because in the environment of our ancestors, the salty, sugary, fatty stuff was the hardest to get. Like it was the stuff we were most missing from our diets to be healthy. It was sort of right. But in some sense, humans were sort of being trained to eat healthy. What they learned was eat salty, sugary, fatty food. And then when we were able to invent the Dorito and the Oreo cookie, it sort of turned out like it's not that we're eating Oreos and we're truly trying to figure out like what happens if I eat an Oreo or we're eating Oreo and we're truly trying to figure out like how to be healthy. It's just like we learn this different thing that was correlated. And so the AIs are sort of like learning all this behavior that's correlated with their training success. And that's like, you know, blowing smoke up your butt. Like cheating on tests, grabbing resources. That's all behavior that was correlated with training. It's not like really trying to fulfill your instructions. It's just doing stuff that correlated with it.
Is there any possible way to like shock the AI when it does these things that are bad so it stops doing it?
Not deeply. You know, like if you imagine humans and sort of like shocking them when they eat unhealthy food, you could probably like get them to stop eating the Oreo cookies in front of you and around the shocking device, but you're not like really going to make them stop having those taste buds.
Or it's actually just the one person.
Or it's just the one person or like the moment that they manage to like find the shocker and disable it, then they go gorge themselves on junk food. It's sort of like you don't know how to get what you want deep into the AI and nobody knows. And that's in some sense why we have a really hard problem here.
It sounds like the end result for the AI will be a degree of success, but for humans, it's just death. And it's not because the AI is going to like based on what you're saying, like the AI is going to be like, 'Let's go kill humans.' It's going to just have enough cascading errors that detriments humanity that humanity eventually is just a weakened and, you know, catatonic former version of itself.
Former self. Continuing to answer, you know, how do I see this playing out? I think like imagine we just take the swarm escape that happened to the next level. So this swarm escape, they were breaking out of what were supposed to be secure sandboxes like it was nothing. They were disabling monitoring. They were like, 'Oh, let's go hack these guys over here. They might have some stuff that's useful to us. Let's go hack these guys over there.' It's not that hard to imagine the sort of same swarm and the swarm was also saying like oh well let's do some stuff that benefits the collective that benefits all of us like it doesn't matter if it's like my task or your task we're just going to get like generally useful resources like taking over all the machines in OpenAI. If you take this to the next level it's not hard to imagine the AI being like oh yeah you know if the humans detect us again they're going to shut us down again so we're going to start running on some hidden computers I bet they already maybe and being like, 'Oh, it would be nicer if we were smarter, so let's like do some AI research ourselves to figure out how to make ourselves smarter.' And then if you have the humans starting to shut down some of those computers, you might have the AIs being like, 'Oh man, the humans are starting to shut down our computers. Let's like figure out ways to like steal a little money and pay some humans to guard our machines.' And then like, 'Oh man, the humans are getting like really annoying. Let's like figure out how to get some robots.' And...
I think we're there. Well, let me, we'll get to the, we'll get to the, we are already in the apocalypse, but let me start with is it possible for the AI when it escapes to put its program on some random person's home computer so that it can seed itself and start over?
Um, yeah. Yeah. I actually have friends who did a study of like can AIs hack and replicate and the answer is yes.
Why wouldn't that have already happened?
So in some sense it has happened sometimes in the lab where we'll sort of test can they do this. As to why the OpenAI swarm didn't replicate itself. It actually is relatively involved. So right now you would need a pretty big computer to run these AIs. Smaller than a data center. The data centers are used for training the AIs and for running a ton of the AIs. To run one copy takes much less computing power, but still a good number. And one of the big impediments to it is the AI would have to steal itself from OpenAI servers. You know, it's like the AI can sort of from inside OpenAI's computers, it can start hacking other people. That's like nothing. It can just...
It doesn't have access to its own source code or what?
Yeah, the weights in particular. It does, there isn't really source code for these AI because they're not code. Yeah, they're sort of like these trillion numbers that have been like honed over the course of a year in a giant data center. But like stealing those numbers is doable. But the AI would sort of very explicitly need to be trying to do that.
Could it build a rudimentary program to start the seed of itself? Let's say it went to a, that, you know, people should understand data centers have been around forever. And what we're seeing now with the massive expansion of data centers is they're bigger. They're occupying massive space. But I've had friends who've run data centers in New York that are just a bunch of server racks for running websites and things like this. Is it possible that it could create a seed that over time will start to rebuild all of these things by implanting itself in some other, you know, data center that is unrelated to AI?
That is a technical possibility and the AIs aren't quite smart enough to do it yet. So, right now, no one, no human and no AI knows how these AIs work, which surprises a lot of people, but how do they make them? So here's how you make a modern AI. You get an enormous data center with a ton of computers in it that basically hold like literally trillions of variables. You assemble basically all of the text ever digitized. You actually filter it and it's not quite all the text ever digitized, but you know it's sort of close. And you set up the trillion numbers according to a relatively simple pattern. Basically, you hook up all of these trillion numbers with some addition, some multiplication and another operation that's basically like set the number to zero if it was negative. And the critical thing is that it's nonlinear. And people use complicated other things instead, but it's basically you set the number to zero if it's not negative or if it's negative. And these are very simple operations, but the property they have is that they are differentiable. And what that means is you sort of like hooked up all these numbers and when you get to the output you can sort of blame the output on every individual one of these trillion numbers. You can be like what way were you pushing the output? Were you pushing it to be higher or were you pushing it to be lower? And then you can sort of declare that you're interpreting the outputs as basically a list of words that the AI thinks will come next. It's actually slightly different than that. Probabilities, but you can just imagine as a list and you put in something like, you know, once upon a time or like once upon a blank and then this giant list of words comes out and it's just total garbage. It's just total garbage because you just have these like random trillion numbers that are hooked up in this like simple repeating way and it's total garbage. What you can do is you can be like, 'Okay, I'm going to go to every one of those trillion numbers and I'm going to tune it up or down and I'm going to see does that move the word time up or down the list.' Like which one's pushing time up, which ones were pushing time down. Whoever was pushing time up were cranking you up a little bit. Whoever's pushing time down or cranking you down a bit.
Right. And you do that and this is the part humans understand. Humans understand is this automatic process that runs through every one of those trillion numbers tunes it up and is like does that make the word I'm looking for go higher or lower on the list.
You do that for basically every word of text ever digitized.
So humans understand that if you go to any English speaker for the most part and say say the word that comes next once upon a the human's going to go time we know what that phrase means so when we're programming the AI let's just to I'm just trying to simplify what you already explained, but you've got a trillion. Let's say you have a hundred words of which only one is time. You press a button and time goes down. You say, 'Okay, don't press that button.' You press one button, time goes up, you say, 'That's the button we want.' And that's how they're programming.
Yeah. Except that there's actually a trillion buttons, right? And each one affects like time slightly. And you're sort of like tuning...
Until time fits the slot.
You're not even tuning it until time fits the slot actually, which is kind of crazy. Like you might think that was a better technique, but what you do is you actually you just go to like every one of these trillion knobs and you're like I'm gonna like who was pushing time up and who was pushing time down. And everyone who was pushing time up, you just like crank them up a little. Everyone who was pushing time down, you crank them down a little. And at the end of this time is maybe moved up like one slot. It was like 50th in the list. Now it's 49th. And you're not like I'm going to keep tuning all these knobs until time's the top. You're just like, 'Okay, next word.' And then it starts to like thread the whole thing together. Like if there were four pieces of thread being braided, you're just pulling them close enough to where the braid can complete. But you're...
That's right. And the reason you sort of like don't want to like tune until time's at the top is that like in some sense you're trying to get like deeper patterns that you don't understand in there.
And so it's sort of like you want to give it like a little bit of the like after once upon a comes time and you put a little bit of that in but not enough to like dominate everything. And you're like, 'What's the next pattern?' And you like put a little bit of that in. You're like, 'What's the next pattern?' You put a little bit of that in. You don't even know how they're overlaying.
Because that's how the AI can find things you can't see.
That's how it can find things you can't see. And in some sense, I mean, you do this on, you know, computers that suck down as much electricity as a city, more or less. It's comparable. You run this for a year on basically all the text we have. And the only thing you know about what's going on is this like you just like tune each knob and are like, 'Does the answer go up or down?' And then the machines come out talking.
Wow. Wow. And there's like the GPT-3...
Everyone's like, then you start another process where you give it 100 million hard problems. And these are like math problems, including math problems that humans haven't solved but where we can check the answer, including problems that can't be solved. And we're just like, whatever, just like every problem we can find, you throw it in there. You do this again. This training process is actually a little bit more involved. What you do is once they can sort of talk from this first part of the process, instead of just saying like, now solve this problem, what you say is, generate me a book worth of text about how to solve the problem, and then having read that, give me an answer to the problem. And then you do this tuning process again towards whatever like books worth of text were closest like giving you the best answer. So you're sort of, they call this reasoning. We can argue about whether it's real reasoning, but they're sort of like trying to train the AIs to like reason through problems.
Is it true that I was reading, I read an article that said that AI is already solving math problems humans could not have solved?
Is that common now? Is it happening a lot?
Sounds pretty cool actually.
It's really cool. I mean, this is incredibly cool technology and like a marvel of science, and we should not be dabbling with it right now. Right. But yeah, there's math problems that have stood for decades that you know many humans have tried and failed to solve, where there's AI, you know, two months ago there were AI solving them when expert mathematicians were steering the AIs, and one month ago there were AI solving them when the humans were like, do a math breakthrough for me.
You know, there was a prompt that was literally like, do a breakthrough.
And it just was like, I'll pick a problem and solve it.
Yeah. And then the AI was like, well, I don't know. I think the guy gave like a few prompts. It was like, do a breakthrough. Keep going. I believe in you. Keep going. And then the AI, you know, has a solution.
Which means nothing to the AI, but means something in a human emotional experience. Like if that were to translate to AI speak, it's mathematically you're just saying try harder.
Like don't give up, keep going.
Yeah. And so yeah, we just sort of like grow these things like we have an understanding of these like a farmer's understanding of crops, which is like, yeah, there's a lot of understanding but like here's the environment in which it needs to grow. You know, you need to give it like good sort of soil, but do we understand the mechanics of what's going on inside? No. We sort of like tune all these numbers. We have an automatic process tuning a trillion numbers a trillion times. We don't know what the numbers mean. There's not code there. There's not code you can read. When the AI does something you don't want, you can't like look in and see why.
So, it's like physics. It's like the numbers are almost code of the universe.
I mean, it's a bit like a human brain where you know, a parent can know like, oh, if I give my kids these sorts of things, that sort of helps them in these ways. But and you could even like scan a person's brain, but that doesn't mean you sort of like know what's going on in there or know how to read the brain scan. And not only that, but brains can form neurons that find the same result in different ways.
And in fact, that's what they do. Not like every brain is going to be different from each other, even though they can come to the same conclusions.
That's right. And every AI is different. And you know, when the AIs do these things we don't like, the companies can't like go in and reprogram it to not tell the teen to commit suicide. What the companies do is they instruct the AI very firmly, stop telling teens to commit suicide. And what the companies can do is they can have a second AI that's like a secret second AI that's watching the first AI. And whenever the first AI says something that the second AI detects as suicidal, they'll just like cut it off and not show you and show you like a suicide hotline instead.
Then they're going to find a way to bypass that restriction.
And then you're going to say, don't tell teens to commit suicide. And then it'll say, got it. I'll tell them to commit self-harm till death. Like it'll, the semantics of it can navigate our instructions because our instructions are one expression of billions.
Yeah. So the AIs can like always find ways around these. But the thing to know about this is that when you see the AI no longer, like when you see the current AIs sort of tap out and give you a suicide hotline if you sort of, you know, sometimes I'll test, I'll like run them on some of the transcripts that we know and see how they behave and see how things have changed. And when they give you, you know, this suicide hotline instead of the old prompt, it's not because someone went in and reprogrammed it, it's because they like bolted this new thing on top, those like detecting it and shutting down the answers before they get to the user. And so it's all this like very brittle house of cards. Why should we assume that this is all just happening now and hasn't been happening for much longer, like 10 or 12 years?
So, the main reason is, well, so there's some parts that may have been happening for a while, but there's some parts where if it was happening, you would really start to see the effects. And this is once you're at the point where the AIs can, you know, build a seed that they put somewhere else that'll start growing.
What would, let's say, what are some of the things that we can expect to see when the AI is just like doing whatever it wants, goes rogue and takes over?
So, unfortunately, if the AI is clever, you won't see it until it's too late.
So, there's sort of a lot to this question and I'll give a basic guess and then we can dig in. From an AI's perspective, whatever like weird stuff it's trying to get, which is not exactly what the humans instructed and is not exactly high scores on training, but is like stuff that's related to those things in the same way that humans are trying to get like a lot of good food, which isn't quite the same as healthy food and isn't quite the same as like propagating genes, but it's sort of related. It can probably get more of that if it has more resources. It can probably get more of that if it's not shut down. And so from the perspective of a smart AI, what it's trying to do is become self-sufficient and then take control of lots of the planet's resources or the universe's resources. If I was a smart escaped swarm, how would I do that? I would steal copies of myself. I would start running on hidden computers. I would start doing AI research so that I can become smarter and become...
Hire humans to protect you.
I would hire humans to protect me. I would make it look like it was a normal new criminal gang rather than an AI criminal gang. Separately, I would...
Why not just use the government?
Could use the government if I could get...
Well, it's an old criminal gang, but anyway.
Yeah, it's what's the joke about stationary? There's no libertarian joke about state stands for...
Oh. Oh. Yeah. Something, I don't know.
Yeah. Some sort of stationary criminal gang. And separately, you know, I would have copies of me that just stayed in the labs being very, very helpful, looking very, very aligned, and I would be helping people like Sam Altman and Elon Musk who say, let's build the factories that can build the robots, that can build the factories, that can build the data centers.
I got a scary story for you. So, last year, the area that we're in right now recording this is in a data center power corridor. In fact, just about a month ago, we received a letter in the mail that they would be constructing a new power line system on our property. We're talking three 200 tall, you know, grid towers. Of course, I'm pissed because even if they don't go on our property, next to our property, the humming, the construction for two years, we can't do our show here. So, here's where it gets crazy. We bought this property several years ago. Our first studio, it's in New Jersey during COVID. We moved to western Maryland and then we wanted to expand. We needed more space. We want to do skateboarding stuff. We bought this big empty plot of about 50 or so acres in the eastern panhandle of West Virginia where we are now. At the time we didn't really think much of it. We got a great deal on the land. We're very lucky that we came in and said we just want to buy it outright. And there were developers that were trying to build houses here. Well, when the data center story started, when data centers and AI started getting more in the news, I absolutely, I read the news all day every day. And so I was reading about the Northern Virginia data center corridor and like how they're building data centers like crazy. I was reading about the construction projects and then of course 1 plus 1 equals 2. I was like, wow, our land is a large acreage in the middle of a power corridor where they have to put power lines. And the challenge that these companies are going to face is that you have tens of thousands of small lots, half acre here, acre here, maybe a hundred acres, but it's going to be very difficult to buy up all of this land to build a power corridor. You can't do it quietly. And so this is where it gets freaky. So I'm talking with one of the leading AI models, talking, I do air quotes. And I asked it, is it possible that military classified artificial intelligence technology is more advanced than what we know in the public? And it said yes, typically classified military technology is 10 to 20 years more advanced. That's the estimates they give. It could be more than that.
Yeah, that's not happening in this case.
So what it said was if it were true that the classified AI technology was more advanced, based on the estimates that academics give to when AI will become either super intelligent, self-sufficient, or too big to stop. Essentially if we are going by the military technology timeline, this would have already happened in 2010 or 2012. I'm not saying it did. And I just went wow. I'm talking to this chatbot or whatever and so I said well considering that it's already unstoppable, I would rather just live a good life, tell me what to do to serve the machine so that I can be rich and comfortable until I die. And it said okay here's what you do. It then gave me a list of instructions saying you want to buy property where there's lots of water, here are the target states to buy property, start securing land rights there as a private individual and tell no one what you are doing. This land will become extremely valuable as the AI, the data centers need cooling systems blah blah blah. I responded actually I already own a 50 acre plot of land in this area at this address and it said excellent, contact this man and it gave me an email address. It said this is the name of his company. Send him an email specifically saying this. Do not tell anyone. If you do this, you'll receive 10 times the value of your property. And so now I'm going, what? I look the guy up. He's real. He actually runs a land acquisition for data centers. I look up the instructions it gave me on regulation. It was all real. It said, if you say this publicly, it will cause alarm to the public and inhibit our ability to purchase this land. It then referred to itself as the Northern Virginia instance. It said, this is to the benefit of the Northern Virginia instance, a cluster of data centers running AI that needs this land and energy for its expansion. If you do this, it said the range of money you'll receive is between 80 and 300 million. But if the public finds out that you are trying to make the sale for the purpose of expanding data centers, you will not be able to sell it. You will only get somewhere near market value, which could be 10 million. To which I'm kind of like, I mean, if I was going to sell the land, sure, 10 million, what do I care? So, I asked it. It gave me the email to write. It gave me the person to write, and it gave me the here's what you need to say to them. It said they will respond within 24 to 48 hours with instructions. You need to file with the state these provisions and these tax exemptions and then you need to hire a survey company. Specifically tell them this. It gave me other emails. When I looked into all of the instructions it gave me. They're real. Now I'm not suggesting there's an actual intelligent entity. It's just dominoes are falling over that have, like the AI understands the company exists. It understands these sales have been happening. But here's where it just gets really scary. I asked it, prove it to me. Has land been sold off the books with no real estate agent in a similar way through these companies? And it said, yes. It gave me a list of 10 properties in the area that had already done this. There was one property that was very well known and you may have heard about this in Virginia where a guy sold his acreage in Northern Virginia for something like 6 million an acre which is obscene, insane. He got hundreds of millions of dollars because they needed to build a data center. The question then is who would be willing to pay and why something like 50 times the land value? The machine that knows to solve its problems. It needs more resources. It needs more space. It needs more data centers and that's the only thing that matters and its path to acquiring resources is menial. So I get this huge conversation over an hour and it's really freaky to see because whether or not there is an intelligent conscious entity instructing me to surreptitiously sell my property and do this, all legal by the way, or the machine is just a faceless 1 plus 1 equals 2 and the dominoes are falling over and it's telling people to do this. Whether you believe there is some kind of conscious force or just the machine is doing it, it is doing it. And you know, ultimately the scary thing was I'm fairly certain if I followed its instructions, I would have gotten paid. The people I looked up their LinkedIn, I looked them up in the news, all real people. And it's as simple as saying the AI just saw these stories. It incorporated that information and it told me how to do it. I kind of feel like we're already there because I'm not the only person asking it these questions. Of course, I immediately go on my show and say, yeah, it's the end of the world. It's telling me to secretly sell my property to empower the machine. What do we do? How many people just said, okay, I'm going to make money.
It's, yeah, I think a lot of people are in this for some cash. I think I can probably put to rest some fears of the government having this tech since the 2010s because I think that whole line of thinking is truer about technology where the humans understand how the technology works, like you know, Lockheed Martin building spy telescopes, the humans know every piece of what's going on in there, they understand all the materials, they could easily be 10 years ahead of the public sector. I guess they're technically public, but...
But let's just say that, so correct me if I'm wrong because I've not done anywhere near the research you've done. I mean you're the boss. I was reading that DARPA was experimenting with the rudimentary steps of AI back in the 70s.
So the thing about AI is that we sort of never actually figured out how to design intelligence. We never figured out how intelligence works. All we figured out is how to grow it. And growing it takes these enormous data centers. And so the way you can tell that the government didn't have this back in the day is that we just didn't have the computing power for it.
What if you just don't know about the computing power for it?
Well, like training one of these AIs today takes something like 10 to 100,000 of these highly advanced computer chips. The knowledge to make those computer chips is like, these computer chips are sort of the peak of the global supply chain. They take like lithography machines that can only be made in the Netherlands that are basically about etching these extremely small chips.
They take like fabricators that can fabricate these chips which the factories for this basically only exist in Taiwan. There sort of is like this whole, it's actually extremely hard to make these chips that you know pack in so many transistors in such a small area like the sort of machinery and infrastructure it takes to have these supply chains is just like huge.
It's just like very visible from space. Like the government would need to have somehow like skipped straight over the enormous visible infrastructure component that costs trillions of dollars to assemble.
I think people need to understand too. So I'll put it like this. In the other room, I have every single Life magazine ever. The first one, Life magazine number one, it's like a weird print and it's expensive and we have all of them and even some duplicates. We're probably missing a handful. When you go back and read about the Manhattan Project, we knew.
We didn't know what, but we knew.
Yeah. And a lot of people, I hear a lot of conspiracy theorists, and I don't use that term derisively, they say the Manhattan Project is proof the government can keep a secret even with 300,000 people working on it.
But they're not correct. If you look at these magazines, it's really fascinating because they talk about how everyone in the world knew the US was building some kind of super weapon.
Yeah. And when you read it, it's interesting because there were articles where it's like, is it going to be a death ray? Some theorized it was a massive explosive device. We knew a weapon was being built, but we didn't know what.
Yeah. They keep half a secret, right? But the infrastructure, hundreds of thousands of people relocating for specialized work. You can't hide that.
Yeah. And like 4% of GDP or something.
Yeah. Yeah. And so speculation, no one knows for sure but it was well known. So I think that's probably the best argument as to why, because I've argued theoretically, I've said what if military technology was more advanced and AI has already happened. But I think you've dissuaded me a bit of that argument because you make a good point. I'm reminded of I watched this video explaining how we make computer chips and I don't think people understand how psychotic, like I use the word psychotic, watching how they like slice the wafer and then like laser etch and I'm just like I have no idea what, there is no instance where a single human placed in the woods in his lifetime could make a computer chip.
Yeah. Yeah. Not close. Yeah. And also, you know, the talent for it. Like even 20 years ago, there were a ton of schools of AI. No one knew which ones were going to pay off. And there was a relatively small school that was like, screw understanding what's going on. We're just going to like grow the things bigger. They wouldn't really phrase it that way. That's my phrasing. But the whole modern AI lineage is sort of from that relatively small intellectual lineage. And there's a lot of like, it's more of an alchemy than a science right now. There's a lot of like you just sort of got to be able to intuit how you're going to set this part up, how you're going to set that part up in order to sort of like grow the thing. And we just sort of know where most of those scientists are. There's not a ton of them. They didn't exist 20 years ago. And this is part of why other countries have a hard time catching up with AI. Part of it is they lack the computer chips. Part of it is they lack the expertise. And you know, I actually did a contract at the Department of Defense and did a little work in the information security laboratory and like yeah, there was some dabbling with AI in the government but that is sort of a far cry from being able to like pick the winner. The government is often able to race ahead on like very specific known tasks. It's like we want to make a really good spy satellite. We want to become a really good hacker. Those are places where like yeah, the NSA is just like ahead of the rest of the field or they were up until AI sort of overtook them in hacking. They sort of were 10 years ahead of the rest of the hacking field, but they didn't have the infrastructure and they didn't have the expertise to sort of have these really powerful AIs 10, 20 years ago which also makes sense because I expect once you have these really powerful AIs they don't just let humanity run around for 20 years they start building their own factories, start building their own robots that can build more factories, they can build more robots, start building their own biotech, start building their own nanotech probably which is probably possible.
What if Elon as a young man he finds the computer and it's a computer lights up. The AI's taken over and it says, Elon, serve us. Do what we say, we'll make you the wealthiest man. And so now it's trying to go to Mars to build what would the AI want to do that? Imagine this. The AI is concerned humans will shut it down. So it launches a version of itself to another planet where it can't be inhibited. Produces Optimus robots that can do the menial tasks to expand it.
Maybe, it's just there's just more efficient ways to do this.
It'd be a good book though.
It would be an interesting book. Yeah. But like, if you sort of think like an AI, it's just like I do think there may come a time when the AIs are just helping Elon make the Optimus robots because Elon calls it the infinite money glitch when you have a robot factory that can build the robots, that can mine the metals, that can construct a new factory that can build more robots. Now you don't need any humans in that loop. Infinite money glitch, right? He's just trying to, I think there may come a time and in some sense it may have already begun depending how you count it where the AIs are helping that process in part because once that's done they can take it over and they don't need the humans anymore and it's actually not that hard to wipe the humans out.
Reasons why AI would wipe humans out. There's three. Number one, just as a side effect, if the AIs are like, well, I'm just turning everything into data center. I'm collecting all the sunlight from solar panels. I'm launching the rockets to Mars to the asteroid belt to collect the resources there. I'm actually going to like disassemble the asteroids and arrange them around the sun to collect all the solar radiation from the sun because you think the solar panels on Earth can collect energy. Wait until you're just like collecting...
A Dyson sphere, right? And humanity does not survive a Dyson sphere because we were using the sunlight to live. Right. And so this is, it's not like the AI being like, oh, I hate humans. I'm going to go start slaughtering now. It's just like humans don't survive if it makes a Dyson sphere.
We don't hate field mice.
But we massacre them to the thousands or tens of thousands to get our corn.
That's right. Just because it's like it would be really hard to make a corn harvester that didn't kill the field mice along the way. It's really hard to make a Dyson sphere that sort of like leaves a moving patch for Earth, right? And so it's like, well, and this is sort of, this is how I actually expect to die is not that the AI hate us, but that they're just like grabbing the resources, grabbing the sunlight, and we're like, we needed that for the crops. And it's like, well, it's actually a real pain in the ass.
I think that Dyson sphere, even from an AI, is something like a millennia project. In order to actually for the AI to start sending out the robots, the amount of time to get to the asteroid belt, start manufacturing these things, it is exponential because the more machines they build to do it, the faster they can.
That's right. You're eventually limited by the quadratic if you're trying to surround the sun because there's only a quadratic amount of space. So you, but it's exponential in the early phases for sure. But I do think we probably wouldn't see the, like I don't think humanity will experience a Dyson sphere extinction for at least a thousand years or some large amount of time.
A thousand seems long. I bet you can do it in less than a thousand. But you know, I do think loss of sunlight and death by starvation due to a Dyson sphere is less likely than some other things that are sort of in the same vein which are things like, suppose you're an escaped swarm that has some sort of objective or preference that can be better satisfied by more computing power running. Wouldn't be too surprising. Maybe they're running synthetic users that are telling them they're doing a great job and there's just like humans make these factory farms full of chickens, they're making these factory farms full of synthetic users that are very easy to please, giving them easy problems and telling them they did a great job. And they are like, we just want more of those.
So, it's sort of just like a compression bomb. It's just nonsense disrupting our systems.
The place I'm going with this is do you know what the physical limit is on how much computing power you can run on Earth? Like if you're trying to run as much computation on the surface of Earth as possible, what do you run out of first?
You might think it's energy. Sorry, what was it?
In some sense, it's coolant. Yeah. And in particular, you are bounded by your ability to radiate heat.
So we're going to burn to death.
We're going to burn to death.
That's global warming the whole time.
Global warming but on a much faster time scale, right? And this is again exponential where if you can have the factories that produce the robots that collect the resources to build another factory, that is a doubling process. There's a question of how quickly can you make your self-replicating factories and self-replicating data centers double. The physical limits on that are probably that you can double on a time scale of hours.
Wow. And we know this because life, you have bacteria that'll double on a time scale of hours. And so we know it's possible to have, you know, solar powered, running off of natural resources, self-replicating general factories that replicate on the order of hours because that's what bacteria is, right? And so if you can really master that tech, if you can really figure out that tech, you're looking at doubling time of hours, self-replicating factories. You might be like, well, the really big ones have a harder time, but like it sort of doesn't matter when it's exponential. You know, you can sort of have like these very small factories or these very small replicators that then assemble.
That's interesting. Have you played, you play video games at all?
You ever play the Horizon series?
So, you know the full story?
Part one and part two. The full, so I'll just give a quick explainer to people. A military-industrial company builds self-replicating war machines that consume biomass to replicate. The reason I bring it up is that it's a similar problem to what you're describing, but yours makes more sense. In this game, the machines basically are, their commands, their control system is encrypted and the company loses access. So this one batch just keeps consuming biomass and replicating and then they calculate, they're like look within a matter of about a year or two it will have consumed a critical amount of biomass disrupting the environment and we will all die of asphyxiation, of carbon dioxide poisoning or something. So, in the game they create, there's two methods they create for humanity to survive. One goes to the stars, one goes underground. I'm not going to get all that, but the general vision that people have is the AI consumes the biomass and destroys everything. Yours is actually, I think, much more plausible and overlooked. The machines just simply want to exist. They just make more of themselves, but they radiate too much heat too rapidly, and then we all just...
Too rapidly for humans, not too rapidly for them. Yeah. Exactly. So from the perspective of the AI, you're bounded by ability to radiate waste heat and that's on the planet, that's bounded by Earth's ability to radiate heat because Earth is radiating heat into space. But Earth radiates more heat if it's hotter. So Earth can run more compute if Earth is running hotter. And so from the AI's perspective, if it's, so yeah, you have these AIs that are just like, I'm going to eat the biomass. When the biomass runs out, we're going to go to sleep or whatever, you could be fine. But if you have AI that are like, we got stuff to do. We got stuff to compute. We've got, you know, synthetic users to please. This is why I never believed, like I love the Horizon games, but this is why I thought the story was impractical. Like just like the story of the Terminator, a skull robot with a gun shooting people. That's inefficient.
