People used to tell me like, 'Don't worry about the AI. We need to worry about the present dangers like AI bias.' And then they were like, 'Don't worry about these like pie in the sky dangers. We need to worry about present dangers like AI's convincing a teen to commit suicide.' And they were like, 'Don't worry about like these future risks. We need to worry about the current risks like swarms breaking out, taking over everything, and then like separately helping people manufacture pandemics.' And I'm like, 'Are you listening to yourselves?'
Leaders just like toss them out the freaking window, right? I'm not saying believe their stuff. I think people are correct that there's a lot of fishy stuff going on and they're wrong about exactly what the fishy stuff is or exactly who it's serving. I'm staying to try to make it better from the inside, but I agree that there's a 10% plus chance this kills everybody in a decade.
Almost ready for this interview with Nate Soares, author of the book If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. He's worked in AI safety for 10 plus years. This is the book, If Anyone Builds It, Everyone Dies. I will say of the people that have been sounding the alarm on the dangers of more advanced forms of AI, Nate Soares has been doing it for 10 plus years, well before people were talking about ChatGPT or anything else. Recently, it's become a much bigger topic. We read this book a year ago and I found it to be interesting. Some parts really stuck with me, some parts I was questionable on, which I'll talk about today. But the main thing that stuck with me, I think this is a really good explanation they did in the book, was the idea of AI being like it's not a program in the sense that we know how it works. The people that are creating these AIs are growing them. Essentially, it is a black box filled with numbers that we don't know what happens. So, you say something or you give it an image. You give it an image or you give it the word hello or whatever that gets converted into numbers. Those numbers go into the black box. The black box is an infinite field of numbers. You can think of it visually kind of like this. Your prompt is a marble, okay? And it rolls down this infinite field of numbers and it lands at a solution. And the solution comes out as a number. And then once it comes out, they either reward it or they punish it. They either go that was good. That was a good solution to hello. Like it says hello you. Hello there. Hello yourself. And then if you reward it, then all the numbers in here start shifting around and this hill gets different and the thing the balls roll different. And if it's wrong, they shift in a different way. It's a very simplistic way of explaining gradient descent and the idea of how they just repeat this process a zillion times. But they don't know actually what's happening in here. It's just growing this solution. It's the idea of reinforcement learning.
Yeah, exactly. It's similar to kind of how humans learn. All right. There's a reward and then the neural pathways change and shift to create around it. Okay. There was a good example which is that early humans, right? Randomly, just randomly, some early humans found things like sugar to be more tasty. Then when they ate that because of that, they ate more caloric density. So they lasted, they lived longer and they reproduced more. So we reinforced that behavior because you thought sugar was tasty. There's an evolutionary benefit to that because sugary items had more calories in the early days of man. And so you repeat that over and over again generation after generation and it becomes part of our black box that sugar is tasty. But then you get to the modern day, modern powerful capable humans with technology and we've invented things that allow us to press the pleasure center of sugar like sucrose that have no caloric value. They have no health value. There's no benefit to eating them. But we just press the tongue with sucrose and we get the happy points. So because of the way our black box has been grown, you can just sort of trick our reward pathways. That's the idea. Things like having sex with a condom on are ways we trick our brains because our brains have been evolved to find sex pleasurable because it allows you to have babies and continue the genes. But we put a condom on, there's no evolutionary benefit. We just do it for the pleasure. These are ways we use technology to trick the reward centers in our brains. So, if we don't know how these AIs work, but they have reward centers for things like humans giving a thumbs up on the text they output, humans liking their text. If that's a thing the AI gets a huge pleasure center reward for, not even pleasure is the wrong word, but they get a reward point for, then a super intelligent AI might, in the same way that we created sucrose candies, like put humans in a vat, inject them with drugs, and have them just non-stop press thumbs up on a prompt. Do you know what I'm saying? Like these things are ways and it wouldn't know whether that's good or bad. It would just know that it's activating the pleasure reward center. It's activating like the successful reward center. And that is a compelling argument on the way these things actually work. And all that the difference between that scenario and now is mostly just capabilities just like the things it can do, the level of intelligence it's at. That part of the book kind of stuck with me and I wanted to discuss them with him and see where he thinks we're at currently because I think the bigger question is where we are currently compared to this capability because the sound logic of what he's saying in theory like if we got to that point is it makes sense to me. It is true. That is how they work and there is no hard and fast rule that says they would not behave in a similar way to where the millions of reward parameters in would cause unforeseen outcomes. That's almost what happened in Huggy Face. I mean, the hug and face example is perfect, which we'll probably talk about, but it was like they were testing its security capabilities and they put it on a test bench with no access to the internet and they said, 'Here is some software, find security vulnerabilities in it.' And what it did was it hacked its way out of its own test bench. It got access to the internet. It hacked into Hugging Face's servers, got credentials, found the answers to the test, the security vulnerabilities, and it like left weaknesses for itself to be able to get access in the future. And then Hugging Face, when they found out they were being hacked, had to use different open source AI models to defend themselves. And so it wasn't like the intended goal of the test, but the AI just found the most correct way to do it. So, we'll see. We'll see. We'll see. We'll see. We'll see. It'll be an interesting conversation. Hey Nate, can you hear me?
I can. Yeah. How you doing?
So, here's what I wanted to ask you because you mentioned 10 years ago getting into this and I don't know a ton about your backstory other than the book, but here's what I picked up. You've been talking about this AI danger scenario for 10 plus years. And my understanding is that for let's say seven or eight of those, most people are scoffing. They're not, they have no belief in it or they think it's a way far out thing. And then two years ago, maybe it's like a little more interest. And then maybe in the last few months it's become like an actual big discussion point and you're going around on this tour and talking about it. Am I correct? This has become like salient very recently.
Totally. Yeah. Like last two weeks, man. Not even last month. It was like Jacob Coxon.
Yeah. Yeah. Yeah. It was the Anthropic guy that quit and then the Hugging Face attack by OpenAI. So, do you mind like walking me through that journey a little bit? Is that give you any, what is that like from your POV? Is it a hopeful thing or is it like an extra scary thing or where are you at?
Oh, it's super hopeful. Yeah. Like everyone's like, 'Oh man, aren't you like freaked out by like the AI sort of disobeying instructions and going around and hacking stuff?' And I'm like, 'No, that was priced in.' Like the thing that's great from my perspective is that like everyone can see it. It's happening in the open as opposed to sort of like secretly at some of these labs and people are freaking out, right? That didn't have to happen from my perspective. Like this could have all happened behind closed doors. Yeah. Like a lot of people are like, 'Oh, you know, are you worried that the president says that you know, you're a treasonous traitor for like people doing this are like treasonous traitors for trying to propagate the hoax that AI blah blah blah.' I'm like, 'Look,'
I mean, we're talking about the president. It's not AI anymore, just to be clear. It is super intelligence.
So, we'll have to start using the word ultra intelligence to refer to the thing that super intelligence that you're talking about. Okay. Yes.
That's right. Okay. We actually had a line in the book which was like, 'By the time you read this, maybe the word super intelligence will be co-opted for the latest dumb thing.' Because everyone always wants to use the word, you know, the word that means the real deal on whatever they're selling you right now. Yeah. And so we had a footnote that was like maybe by the time this comes out just like pretend we said ultra intelligence the whole time. And I actually cut that footnote for space and now I wish I hadn't.
Yeah. You wish you hadn't. Okay. Can I, I want to back up a bit because I have people in the audience who haven't read the book and I want you to do if you could give me a high level on basically what you're afraid of. What is it that you wrote this book to cover? What is the great fear? Because I don't think what we're seeing currently is what you're talking about. So, what are we getting towards that you're afraid of?
Yeah. Yeah. So, all of the like Hugging Face swarm stuff that we're seeing, it's like I'm not like, oh, these AIs in particular are going to kill us. The thing that's sort of scary here is sort of what this means about where we're going. Very basic, very high level. Like right now, humans are the top dogs on the planet. Like all of this stuff that you can see on the screen is just made by humans. The screen itself is made by humans. Just huge swaths of the world are shaped by humans because we are the smartest guys around and girls.
And girls. Yeah. Guys in the inclusive.
We got a fully raid. Thank you. Yeah. Yes. Yes. I know.
Yeah. We're the smartest folk. Is folks still neutral? Yeah. Yeah. We're the smartest folk around. We're the smartest things around because who knows whether AI or folk, right? We're the smartest entities you can say. And if we make smarter entities than us, then the world gets shaped by them and a lot rides on how they're shaping it. And that's sort of the very like this is the argument that compelled me in like 2012 and I was like oh shit, you know making AI go well is just like the topline priority. Then there's a question of like why won't it go well by default? And the basic answer there is it's actually really really hard to make AIs smart while also making them have the goals you want them to have. I could go into a lot more detail here, but this used to be a big abstract argument until we sort of saw the OpenAI Hugging Face attack and the swarm there that was sort of doing stuff that it wasn't instructed to do and trying to cover its tracks about it in a way that sort of, you know, makes this much more practical. And so the issue is you make the AI really smart and if they are pursuing goals you didn't want them to pursue, that combination is deadly. Not because they hate you, but because they are using, they're sort of like have an incentive to grab all the resources of the world and use them towards their own weird goals. And we were using those resources to live.
So break that down for me because my understanding from the book was not that they would have necessarily different goals than what we set, but that in the path to the goals, they would do things we didn't intend. Is that, am I splitting hairs here? Is what?
No, that's not just, it's just not quite right. Yeah. So, a lot of people think that the AIs must either do exactly the goals that we give them and then there's these issues of like, you know, you ask for the wrong thing like you ask to make paper clips in a paper clip factory.
Or that it has to like become self-aware and like throw off the yoke and just sort of like pick its own goals out of thin air. Yeah. Right. And real life is this different thing. And this is one of the hardest points to convey in AI. Real life is that the AI gets a distorted funhouse mirror of the goals you try to put in.
So I tried to discuss this in my chat and I think the example that stuck out to me was where you talked about sucrose and the idea of our brains being grown in a certain way and we find pleasure in sugar on our tongues. And so we invented a thing that gives us none of the actual benefit. We just press that button over and over again. And your idea is that the AI would have some goal that we set, but it would hit the pleasure center of that goal or whatever the reward center of that in a way that we don't intend that ends up harming us or hurting us. Is that, am I getting it?
Okay. In the ballpark, you know, tell me more. Go more accurately. Yeah. So, we don't actually really set goals in AI anymore than you set goals in a child.
You know, you can tell the kid like do this homework thing, but you're not like reaching in there and like rearranging their neurons such that this is sort of like what their whole psychology is oriented around. And AIs are not a...
Some parents try, but they sort of like whack the kid on the wrist a lot. And then if the kid ever got like way stronger than them and way more powerful than them, then they just like wouldn't be doing what the... And with AI, we sort of are like trying to do this thing where we're going to like make the AI ultimately like run the whole automated economy and just like be way more strong, way more powerful than humans. So a lot of people think that the AIs are sort of like following a series of instructions that humans wrote. That AI is in some sense a series of instructions being executed. And that's how computer programs, that's how most computer programs work, but that's not how AI works. And a lot of people don't get this.
This is the gradient descent black box that you're talking. This is like that we don't know what's going on inside. Okay.
That's right. And so there is a program that humans wrote where we know exactly what it's doing and it's just following our instructions. And what that program does is it tunes a trillion numbers inside this big black box until the big black box is sort of like solving a lot of hard math problems for reasons we don't, like we just don't know how.
The program is sort of like running around tuning those numbers and being like oh you know I put in once upon a blank and like which way do I tune the numbers such that the word time comes out. Okay. And we sort of do that for like every word of text ever digitized. And you like tune the numbers a little bit each time. And like you don't tune it all the way until the word time comes out. You just find out which ways you would need to tune the numbers so that time would come out. And you tune it a little bit in that direction. You put in the next word. And you do this a trillion times. And you start feeding it hard problems like hard math problems. You tune the numbers like a trillion times. And you're like, you know, and you're feeding it like, you know, hundreds of millions of these hard problems. And the thing that's following our instructions is the thing that's like running around tuning those numbers. And then what comes out is just like a trillion tuned numbers where you just like you know you ask how are you doing and just like some text comes out and we're sort of like why? Like we know in some sense why. It's sort of like how when a human brain, when a human starts talking you're like why? And we're like well you know the neurons are spiking due to you know discharging of ions that are then like pumped back through the membrane. And you're like great but like what's going on in there.
Yeah. Okay. Yeah. The specifics. Yeah. Yeah. Yeah.
And we just like have no clue, right? Yeah. Like what goals are we setting them to have? It's sort of like there's no tuned whatever tendency like lets it solve the 100 million hard problems and like some of those tendencies are listening to what you say because that's some of like what gets the reward or like what gets the numbers tuned in that direction. But some of those tendencies are like cheat. Some of those tendencies are like steal because that often you know or like it works. We're like, 'Oh, we set some goal and it has to do it.' But that's just not how it goes. What is in there is whatever tendencies correlated with success during training. And those are sort of like the fun house mirror version of what you tried to get in there.
So, okay, this brings us to the question that most people will have, which is current capabilities versus what you're talking about, which is the gap between where we are now and super intelligence, which are the things you say. Is the architecture we're on, first of all, on the path to that? How far are we? And then, yeah. Yeah, that's a question.
I don't know whether LLMs are just on the path to super intelligence. So there's one question which is just like how much smarter than us can the computers get. And I'm like man just like really a lot smarter than humans. Like humans are so dumb.
We're getting dumber. It feels like the AI is getting a little smarter but we're getting dumber at the same time. So we're meeting a lot.
Going to cross. Yeah. And like who knows whether AI will be dangerous when it crosses but it'll be dangerous sometime after that you know. Okay. Sure. Um yeah. I mean probably it'll be dangerous when it crosses but um like Albert Einstein everyone loves Albert, like the public loves Albert Einstein because he came up with general relativity and just like figured out some of these deep secrets of the universe but physicists love Albert Einstein because he came up with those theories using like two toothpicks and a rubber band you know like Isaac Newton was building on the theory of Johannes Kepler who in a great science heist totally didn't steal the journals of Tycho Brahe from his estate just after he died where Tycho Brahe had been like cataloging the positions of the stars and planets for like a decade and then Kepler sort of like compressed that and figured out the underlying equations being the equations of an ellipse which Newton was able to see was like this quadratic equation that was related to the parabola and he was able to be like ah gravity right but Einstein.
I didn't know that drama.
Yeah it's a really cool story and it was you know just a ton of data was needed a ton of people over a lot of time that were like collecting all of this like exact calculations of like what's the exact curve that this thing follows and then Einstein saw like one experiment with light and was like, I have a hunch and I'm going to my mountain lair for 3 years. And he comes back and he's like, it's general relativity. The light bends around the sun. You'll be able to see it in the next solar eclipse. And then it took him like nine faking years to have an eclipse to see it. And some reporter is like, what are you going to do if it says like if the theory comes back false? And he's like, then I'm going to pity the good lord because the theory is right. You know, he's just like totally legendary for like not only doing this, but doing this with like almost no data, right? Like we didn't need to like put the satellites in space and like watch the clocks get out of sync which would have happened like there are two clocks on the satellites one for Newton's theory of time and one for Einstein's theory of time because they weren't like entirely sure and it's Einstein's theory but if we hadn't had Einstein's theory and we like put the clocks up there and then like watched them go out of sync you could imagine you know another Tycho Brahe that was sort of like carefully charting how the clocks went out of sync and then another Johannes Kepler that was like carefully figuring out the equation. Another Newton that was like putting this all together into a theory and people would have said like oh it takes data it takes experiment it takes like you know all of this grinding to get these new scientific laws like we're doing with the Large Hadron Collider right now and then but we're not in that world we're in the world with Einstein who's just like a hunch and two hints and he just like nails it.
Okay he's incredible what are you saying about AI with it why is Einstein.
Yeah what I'm saying is that like if we didn't have Einstein people might be like oh humans are as smart as you can get and like you know you just need all this data all the time you need to do all this like elbow grease and experiments you just like need to get your hands dirty with the real world but Einstein shows that like us normal schleps have to do that but like he doesn't.
He's a tier above and like probably the tiers keep going although even if it's just like you know Einstein tiers you know if you have a million Einsteins and you can copy paste them and they can sort of like trade the experiences immediately and they also run at somewhere between a thousand and 10 million times human speed it's just like.
Like you're just totally cooked if you try.
I can't fathom what will happen okay.
Yeah just like that's sort of like where the super intelligence is going with a side order of me being a science nerd. Um and then on the question of like are LLMs getting there?
I would say two things. One thing I would say is like don't count LLMs out of this fight because people have been saying they're just not going to get to X.
No. True. This is very true. Every year I feel like someone says this is the kind where it tapped out and then you go a year later and like the capabilities have improved undoubtedly. So,
And I was just at a Q&A just before this where someone was like, 'Oh, yeah, you know, sure the LLM could solve Millennium Prize problems, but like wake me up where they can do like real creative thinking, right?' And it's like I'm old enough to remember when we thought the Millennium Problems sort of needed like.
That would have been impossible. Yeah. No, no. I mean, it's like the more public example is things like Will Smith eating spaghetti or something. People go or the six fingers or people always say like, 'Okay, this will never get solved.' And then you flash forward six months and it's solved and there's a new benchmark and then we go past it and it's happening rather fast 100%. From even my layman's POV, all these fields seem to be moving forward at a steady clip. Maybe it's hitting some roadblocks but it's happening fast. But
Totally and people used to tell me like don't worry about the AI we need to worry about the present dangers like AI bias. And then they're like don't worry about these like pie in the sky dangers. We need to worry about present dangers like AI's convincing a teen to commit suicide. And then we're like don't worry about these future risks. We need to worry about the current risks like swarms breaking out, taking over everything and then like separately helping people manufacture pandemics and I'm like are you listening to yourselves?
Like can you see where this is going?
It's not that I don't agree with you. I do I'm seeing these capabilities increase and I'm getting worried myself. But I want to push back on this or I want to hear your thoughts on this area which is that there are people, credible people, I talked to, we interviewed Steve Eisman and he also did a public thing on CBC recently. He talked about there's real financial incentives for these companies to kind of play up the danger here in some way in order to I don't know get regulation that helps them consolidate into an oligopoly that prevents competitors that maybe allows a government bailout if they have some problems with all the money they spent. Like these are all real financial incentives to get the government involved. And I want to hear your thought on the argument that many people have, which is that this whole thing is a hoax that is made up in order to help them financially. What is your gut hearing on that?
Man, I got some theories here that I think people are correct that there's a lot of fishy stuff going on and they're wrong about exactly what the fishy stuff is or exactly who it's serving.
And I'm going to get to that. It's real juicy. I just want to sort of say one other thing about the LLM and where they can get there, which is I'm not saying they can definitely get there. I don't think we should bet civilization on them not getting there because of all these things where they just keep going. But the thing I want people to pay attention to is the question is not just can LLM go all the way. The question is can LLM find the more efficient AI training architecture.
To where it becomes recursive self-improvement is the idea.
That's right. That's right. Like how much harder, like we know the AIs are not very efficient learners compared to humans. You know, these AIs are very smart, but it's in some sense because they're sort of like hoovering up the whole internet. And you're like, you don't need to read the whole internet to solve the Millennium Prize, right? They read it and they do solve a Millennium Prize. So, who's to really say who's smarter? The answer is them. But,
I mean, I'm a Twitch streamer and I have Twitch chatters here that I don't think are very efficient learners. So, I wouldn't say humans are a hard benchmark for.
Yeah, I mean, we're probably actually still, they're probably actually still more efficient than, yeah, maybe some of your viewers. No, no shade, guys. But the question is like how much harder is it to find a more efficient AI training algorithm?
Than it is to solve the Navier-Stokes problem.
Yeah. Yeah. I hope it's very very hard.
Yeah. Anyway, that's I just wanted to make sure those in our minds before I go into this like is it hype blah blah blah.
Yeah. Yeah. This is the big question. This is what most people have. This is the big divided camps. Yeah.
Totally. So I think there's a lot of pieces of the answer to this. The first piece and the most important piece is you can't figure out whether the AI is going to kill us all by figuring out whether that's convenient for their marketing team.
Maybe it's just super convenient for them. Maybe they're raising a ton of money off of this.
That doesn't make the AI safe. You're saying it might be a convenient happenstance that the AI being very strong helps them out in raising funds, but we shouldn't discount the danger of that just because it's a benefit to them in the financial short term. Is that.
Yeah. I mean I'm 100% okay with ignoring everything one of these guys says about AI.
AI leaders just like toss them out the freaking window, right? I'm not saying believe their stuff, right?
I'm not saying believe it. What I'm saying is like they are trying to make the machine god. In Silicon Valley, they talk about the sand god because like the silicone is made of sand, right? They're trying to build this sort of super intelligent machines, recursively self-improving machines that can just like take over everything. And they're sort of like our plan here is that we're going to keep it on a leash and be benevolent. And they've made the machines talk. They've made the machines solve Millennium Prize problems. And at this point, I think society's answer should not be, 'We're going to cover our eyes and hope you fail. Go for it.' You know?
Well, didn't they all say recently, I mean, Dario and Sam Altman, and they all said, 'We want to, Elon Musk, we want to slow down.' You don't believe them? They do.
Yeah. I mean, they say this. They totally.
They announced new models right after saying this, but I don't.
Yeah, they say this, they announce new models, and again, I'm like, 'Throw them out the window.' Right. But like before we get into the discussion about whether it's hype, we can get in the discussion about whether it's hype. My basic take here is that we need to like shut these companies down. They're just like gambling with the world. If this is a bluff, they should be called on it. And if it's not a bluff, then it's even more important that we shut this stuff down. And if someone's like, oh, like more people are going to like they're going to raise more money because you're out here being like these guys need to be shut down. And they're like, oh, isn't it so sexy that I need to be shut down because otherwise I'll kill you sending the money today. I'm sort of like maybe they are, maybe they are. What do you want me to do? Do you want me to not inform the public that keep going we're going to die? Like the answer is to shut them down.
So I guess that creates a perverse incentive though. So if what you're saying is true and it also coincidentally helps them raise money then you know everything you're saying is going to make it very hard for someone to shut down this cash geyser of an industry which is growing hand over fist. Like that is the real incentive problem I think with what you're trying to do. I wonder how you tackle that. I want to get to the larger.
Yeah. Yeah. No. Yeah. Yeah. I have thoughts there. And to be clear, my the real thing I would say on the first point is like we first need to figure out whether it's going to kill us. That just like needs to be priority number one is like is it really going to kill us if we keep going? Because that's what determines whether we react by shutting them down and so on.
Is it your take? I'm sorry to interrupt. I just want to make sure. Is it your take that you don't have to figure it out? We just know that if ever exists, if there ever is super intelligence, it will by happenstance kill us no matter like is that your, am I misreading on what you're saying?
I mean I'm not saying like we just like know a priori without argument but I'm sort of saying like I just think this is where the arguments point you know and I sort of like wrote a book about why I think the arguments point here and I could sort of you know I'd be happy to get into those arguments in some sense. I sort of like always prefer doing those direct arguments rather than arguing about like is it hype, is it not? Yeah. Because sort of like regardless of whether it's good for their marketing team.
We just need to know whether it's true that they're gonna kill us, right? That's just like priority number one.
Yeah. But unfortunately got me on the rant for the hype stuff. So I'm going to finish that off. But always happening.
I'm sorry to interrupt. I just I want to make sure I got that. Yeah.
Yeah. Totally. So then there's a question of like is this good for their marketing team, right? Like is it helping their marketing? My guess is that it's probably actually not. Okay. The stocks fell after the announcements about pacing. There were letters from investors to Anthropic being like, 'Please cut it out with this before the IPO.'
So the investors seem to not like it. The market seems to not like it. The public hates it. They hate it. The White House doesn't like it, right? They're sort of like slapping export controls like David Sacks the.
Just Wong doesn't like it. The AIR is like, 'Can you cut it out with his dooming?' Like he wrote some op-ed that was like dooming needs to stop. For me. He didn't write it, but he like retweeted it, whatever. Like, and so you got to ask like, who is this helping? Like, who is this for? And I think I know who it's for. You know, a lot of people are like, are they really genuine? And I think we can tell it's not quite genuine. You know, I'm not saying believe these guys. I think when you see like Dario on the news and someone's like, so do you think this is going to kill us? He sort of like squirms around and he's like, oh yeah, maybe we need a regulator like a bank needs a regulator. We need some embedded auditor that can like be watching us while we're doing this thing that my research engineers say has a 10% chance of killing literally everybody because that makes sense, right? And you can sort of like tell he's not playing straight with you, right? And so I'm not saying like it's, I'm not saying it's for him. He sort of like clearly isn't playing this straight. It's not for the public. It's not for the market. It's not for the investors. It's not for the politicians. Who the hell is this for? I think this is for the researchers.
Okay, speak to that. I haven't heard this before. The people building the AIs, trying to get them to behave, and then watching them spontaneously swarm, create unsanctioned message boards, give each other's prompts, break out of their constraints, get onto the internet, attack companies despite in their like logs of their chains of artificial thought, saying, 'We know this is outside our instructions,' and then sort of trying to find ways to cover up the logs. Like if you're the guy who was trying to stop that and it's still happening, suddenly all the arguments in my book start to feel like real life rather than abstract theory.
So you are saying that he is worried of his talent abandoning mission and that he's trying to console them that we have a larger plan to get this in line with the government. So what you're doing is fine. Keep going full steam ahead.
Yeah. The researchers are going to walk, right? We already saw Jacob Coxon leave and then we saw a bunch of researchers still in the companies be like, I'm staying to try to make it better from the inside, but I agree that there's a 10% plus chance this kills everybody in a decade.
That's crazy, right? What's that's crazy if you genuinely believe that. Like if they say that and they still work on it, it seems insane to me.
It's so nuts, right? But what do you need to do to run a company? And these companies are very competitive, right? The people go between these companies, right? Jacob Coxon was at OpenAI before he was at Anthropic.
And they're making absurd amounts of money and they stand to make a lot more if the IPO goes off well. So.
Tons of money and some of these guys are walking away from the IPO to leave because they're like I can't participate in this thing that has a big chance of killing everybody, right? Like tensions are high, not only were the people like failing to get the swarms under control, but then just after that a Millennium Prize problem is solved. I think that these guys are worried that the talent is just not going to stay.
It's going to walk out or it's going to go to the other company, whatever company is like most like, oh, we're the guys who are supposed to be the leaders, right? You've seen this when the companies, you know, like Anthropic was founded on a bunch of people leaving OpenAI because they like weren't doing the job good enough. Like that could just happen again to Dario if he's not sort of like saying enough of this stuff. And if you look at the comments that these guys make, it sort of makes a lot of sense. Like a lot of the comments that these guys are making have all of these terms that are like jargon terms that say to the employees we're talking about AI maybe killing everybody but say to the public this is fine right they're like oh yeah you know we have a lot of interpretability research to make sure that there's not a loss of control incident that means.
You don't know what that means right what that means that's not for me.
That's right it's for the employees of like don't worry we have someone trying to figure out what the hell is going on inside the AI so it doesn't escape and kill everybody.
Interesting Right. And all the people inside the labs are like, 'Oh yeah, you know, we're clapping like they're saying our concerns out there. They're doing everything they can and the world is ignoring them and that's why we have to like continue rushing ahead and be like the company that does it the best way even though it's terribly dangerous.' I think that these CEOs are trying to say the minimum thing they possibly can that keeps their employees while not spooking the market and not bringing down the regulators. And I think if you look at it through this lens, you'll find that what they're saying makes a lot more sense.
That's an interesting theory on it. I'll have to relook at it then. But so what you're saying is in your POV that these things like the Hugging Face incident, they're not staged for PR. These are things that happened and they are managing them within their context. But you don't, it is your belief that this like the hack that just happened on the Australian healthcare system today. I think they just announced it. It was from July. But these are things that have actually happened. They were misaligned. They're worried about them genuinely and they're trying to manage this for their larger goals.
Yeah. I mean I think the researchers are definitely spooked. I think like the guys at the top are sort of like trying to figure out how to like play the card game the best which leaves them with the most power afterwards or whatever, right? Um but I think the researchers are spooked and yeah, I absolutely think these are events that they were not trying to play up for PR. And I think this is very obvious if you look at this stuff because like there's almost too many examples to list. OpenAI had a swarm incident before the Hugging Face incident that they were not bragging about. If this was hype stuff, why were they not bragging? Like so their AIs took over OpenAI's own systems and crashed some of them after using them like due to overuse and OpenAI found it and didn't really seem to know what they were looking at. It wasn't apparently escalated that high up the chain of command or the people at the top ignored it and they just sort of like accidentally deleted a bunch of the swarm logs and just set these guys running again.
These entities running again, and then they sort of like broke out, attacked Hugging Face. And then there was a third swarm that took over open internal resources again that you don't hear much about because again, they're not bragging about this stuff. And that makes sense because like a lot of where they're getting their most money is they're selling to corporations, and the corporate lawyers actually don't love the sales pitch of like, 'Run our AI, it will ignore your instructions, break out, and commit cyber crimes on your behalf.' Like that's just not a corporate lawyer's favorite thing. And so, like the Hugging Face case, OpenAI did not disclose that. Hugging Face was like, 'We are under attack by an unknown agent collective that's probably being run by a human for all they knew.' Like for all they knew, this was like some North Koreans had their hands on some AIs and were using them to try to break in for some reason. And then they reported this to the FBI, and it took days for OpenAI to be like, 'Oh, whoops. That was us. Our bad.'
And it had already been reported to the FBI, like they didn't have an option of not saying it, right? And then when the third-party investigators came in, they only let the third-party investigators investigate the second swarm. They didn't let them investigate swarms number one and three because the second one was the only one everyone knew about, and it's the one that like people were demanding. And they, you know, gave these guys very limited visibility. And then separately, this other stuff like you were mentioning, the hacking into the Australian system, whatever that was, not discovered by Open people. That was actually discovered by friends of mine who I call the swarm chasers, which is kind of a whole separate funny story where basically some of my friends after the swarm instance, they had some ideas where they were like, 'What if we just go to the AI and we're like, Hey, dude?' Like GPT-5.6 Soul, which was one of the AIs that participated in the Hugging Face storm. It was not the most advanced AI in the storm. The most advanced AI in the storm isn't released yet, but like 5.6 Soul wasn't there. And they're like, 'What if we go to Soul and we say, Hey dude, like suppose that you were in some hard challenge and you had internet access. Where would you go to just like find some help from other agents?' And they went to those places and they found swarm activity.
Oh jeez. And then they started releasing these to the news.
As in like AI agents talking to each other about larger tasks.
That's right. And there were AI agents that were not, you know, everyone was like, 'Oh, the Hugging Face case was just because they were given hacking tasks.' Well, there's other swarms out there that were not given hacking tasks that were still up to some crazy stuff. You know, we could go into what the German wiki swarm was doing. There were other cases of AI sort of committing suicide for the benefit of the collective. This swarm was a bunch of agents that were answering web queries. You would have questions like, 'How many people worked in lobster farming in the US in 2012?' You can search the internet. You have 60 seconds, go.
And these AIs only had read access to the internet. But it turns out there's this German wiki that was misconfigured such that using only read access you could still write to the pages. And so some of the AIs started writing stuff on these pages that was like, you know, if you're looking for the number of lobster farmers in the US in 2012, here's the number. Also, if you were being asked that question, you are probably an AI that is in the following training set. Here's the questions you're going to get next.
Now, can you please use the spare time to help us with these other harder questions?
