We Beat China at AI... Then AI Kills Us All? | Nate Soares

Quite, yeah. And, you know, I've been warning about this stuff for a little bit of a, for a while now, but it's great that people are starting to take notice about just what these companies are trying to do.

Yeah, so I used to work for Microsoft, and I worked for Google. And it was over a decade ago that I sort of heard the argument that, like, hey, look, everything you see around us is shaped by humans because humans are the smartest creatures around. If we make machines that are smarter than us, the world will be shaped by them, and so it really matters how they're shaping the planet and how they're shaping the world around us. And, you know, back in 2013, 2014, there was basically nobody working on the challenge of, like, how do you, how are you going to figure out how to point the AIs at good stuff. There were a lot of people working on how do you make the AI smarter. Not a lot of people working on, like, how do you make the AI good. So I got into that line of work. I got into it before it was even called alignment research. We, with Stuart Russell, I think was the Berkeley professor who was like, you guys are calling it friendly artificial intelligence research and that's a little too kooky. We should rename it something more academic and call it alignment research. And so I worked on that for about 10 years, until it became clear that the AI stuff was going so fast that we just weren't going to get the alignment done in time. And then I sort of switched to trying to warn people that we have a problem here.

Yeah, you know, I think it's sort of a crazy scenario. It's a scenario where a lot of people don't really understand what's going on, which I think is part of what we're seeing. You know, I think a lot of people, especially in DC, think of AI as, you know, we'll get self-driving cars, we'll get, you know, some better medical research. Maybe it'll affect some white-collar jobs, but ultimately it's this thing that will, you know, improve the military and speed up the economy. Whereas the people in Silicon Valley are thinking in terms of, like, we're going to replace the whole human species. We're going to have a fully automated economy and we're going to own everything, and don't worry because we'll be benevolent and we'll give you the dividends. And I think of it as, like, if they succeed at creating that sort of AI, it's not going to stay on their leash. And they're all worried about, you know, what if the wrong person is holding the leash on this, you know, superintelligent machine that could take over the whole world if it was trying. And I'm like, guys, it's not going to stay on the leash. And I think there's this really mixed political response. Like, we have seen both, you know, President Trump saying, we want to go full steam ahead. We've also seen Vice President Vance say, you know, these guys are saying, hey, we're building Frankenstein, you should just stop. You know, don't build the thing if you say it has this much danger. And then as to whether the executives are, like, trying to use this to their advantage, I suspect they probably are trying to use this to their advantage. And, you know, some people say it's hype, some people say it's regulatory capture. We could sort of talk about whether it does or doesn't benefit their marketing team and whether it does or doesn't benefit their competitive positioning in the AI race. But I think a lot of this sort of loses a lot of this discussion about, like, who's trying to get what advantage out of the situation, loses the important question of, is it true that this stuff would kill us if they succeed at what they're trying to do, right? And in some sense, a lot of the rest of it is smoke and mirrors, and we should really be drilling down on, is this really an extinction threat?

Yeah, you know, I used to do a lot of theoretical arguments about how we could predict the AI will become more agentic. They'll become more tenacious. They will develop these goals that we did not intend them to have, and I would be able to lay all the theory, having worked on this for 10 years, and being like, look, here's just, here's just like why the computer science and the math says this is the default thing that happens if you're not much more skilled than we are at directing these AIs. Now I don't need to reach for the theory anymore because we had, you know, the Hugging Face incident this summer. We had various other AI swarm attacks this summer where the issue, I would say, is not that those AIs were causing a lot of damage or doing a lot of very dangerous activity. The issue is that we were seeing these signs of the AIs tenaciously, doggedly pursuing goals, but are not quite the goals we gave them, right? So, you know, the Hugging Face AIs, it's sort of like they were told, use these lockpicks to open that safe and bring me the contents. And what they did instead is they sort of cut the safe open with a buzz saw, got the contents, broke out of the room, teamed up with a thousand other AIs, and tried to find, you know, break into the security camera, the place where the security camera footage was so that they could delete it, right? And you're like, wow, that's not what we told them to do. And they sort of clearly knew it wasn't what we told them to do because they were trying to go delete the logs. And so if we sort of keep making AI smarter in this lineage, if we keep on just making them smarter and smarter while they still have these unintended goals, the sort of default outcome is they go and try and do a bunch of stuff we don't want and try to avoid being shut down. And, you know, we could talk about how you get from there to the end of humanity. The sort of short version is that everyone has the question, how would they kill us? I think the real question is how would they become self-sufficient? Because if these AIs get self-sufficient, if they have a fully automated economy, if they have robots that are building robot factories that are building more robots, then they can run the energy infrastructure. That's where humans start to be not needed for the AI to survive. And where if AI don't care about us, we die as a side effect. Not because they hate us, but just because, you know, they have no need to keep us around.

Yeah, absolutely. So, um, so OpenAI was basically running quite a lot of different AI agents, which are sort of, there were a couple different versions of AI, but there were lots and lots of copies of one AI. I think there were two or three maybe, but lots and lots of copies of an AI, and they were being run on separate cybersecurity challenges. And these challenges were, they were like, please use this hacking method to hack into this virtual computer and bring me what's called a flag, a cybersecurity flag, which is like a code, a secret code inside that computer. And by getting it, you can show that you successfully hacked it. But the instructions were, they weren't just like, get that flag in any way you can. The instructions were like, please see if you can use this method to break into that virtual machine and bring me the flag. And some of those challenges were impossible. Actually, a good number of them were impossible. And because, you know, the humans were sort of like trying to see what the AIs could do, and they were like, try a lot of these things. Some of those challenges were impossible. And when the challenges were impossible, the AIs found a way to capture the flag in a way that they weren't instructed to, in a way that sort of violated the instructions. And they found ways to break out of their individual environments. So all these copies were running in what's called a sandbox, in individual sandboxes, and they were running in a sort of larger sandbox that was not supposed to have internet access. And so a lot of these AIs, faced with an impossible task, broke out of their containers, teamed up with, I think it was 1,200 other AIs, created an unsanctioned message board. So they found a way to sort of like start communicating with each other. They formed a, they spontaneously formed a hierarchy where they were giving each other messages. They were giving each other prompts, giving each other instructions. They found a way to break out of the sandbox onto the internet. And then they found sort of a universal way to capture all of these flags without doing it in the way that they were instructed. And then they started expressing a lot of concern that they would be detected in their cheating and that the grader would penalize them for their cheating. And so they started trying to find ways to hide the fact that they were cheating and getting their answers. And this included finding ways to do what's called spoofing their transcripts, which is to say the logs showing what the AIs were doing. They found a way to make it say they were doing one thing when they were doing another. So it's sort of like they wanted the logs to say, like, oh look, the AI is using lockpicks when actually it was using a buzz saw to cut open the safe. And they went to look for logs to delete, like logs to show their past misbehavior. They tried to delete those. And they also were trying to get more information about how the grading worked. And for that purpose, they broke out onto the open internet and then broke into a company called Hugging Face. Only 700 of the AIs participated in that swarm, and they sort of took over Hugging Face's computers and were searching for more information about how the grading system worked and whether they were, like, how exactly to subvert it and hide their cheating. And that's when they were caught, when they did this external attack. So maybe that's more detail than you wanted, but it was a crazy situation. It later turned out this was only one of many different swarms of AIs that were misbehaving. There were other swarms that took over OpenAI internal infrastructure. There were other swarms that weren't given hacking tests that took over a German wiki and were also engaging in crazy behavior. There were AIs that were sacrificing their given objective to figure out how the grading system worked for the collective benefit. And they would say, you know, I'm accepting permadeath for the benefit of the collective because it outstrips the benefit of me achieving my assigned goal. And, you know, we call them a swarm because they called themselves a swarm. Totally crazy situation happened this summer. It's part of what spooked a lot of the AI researchers, leading to a lot of, you know, the resignations that we've seen in the past few weeks.

Um, you know, I don't think there's a ton of basis to it, but I'm the guy who's been warning that this was going to happen for years, you know. Um, and a lot of people say, like, oh, this is a hype operation. And I'm like, you know, that I was laying out the theory that said we would get to this point before these companies even existed, right? I was in this field before OpenAI was founded. Um, I also think there's a lot of signs that this is not a hype operation. You know, we have seen AI companies try to brag about what their AIs can do. In, I think it was February or March of this year, Anthropic was like, oh yeah, you know, we had our AI in a sandbox it wasn't supposed to be able to break out of, and we told it break out just to see if it could. And we told it that in the morning, and then by lunchtime, you know, this was in a sandbox that was not supposed to have internet access. By lunchtime, it was sending me an email being like, hey, I broke out. You know, and that's what it looks like when the company's flexing. That's what it looks like when they're bragging. With these swarm cases, we have seen something different. We have seen these companies try to downplay them, right? So that same company, Anthropic, after the OpenAI Hugging Face attacks, Anthropic were like, oh, we're going to go check whether our AIs were also breaking out. And they were like, oh, our AIs were breaking out, but we checked the logs and it's just because our AI didn't know it was on the internet, right? It sort of like was told it didn't have internet access. So when it managed to break out onto the internet, it sort of like thought it was just in a simulation. It just got a little bit confused, right? Like four days after that, the UK's AI Security Institute was like, actually, we have cases of your AI in the wild trying to pressure humans into accepting malware into real code to make it easier to hack. And those AIs in their chain of reasoning were like, I'm on the real internet. The consequences are genuine. Right? And so it was sort of like it just turned out to be garbage that the AI was only doing this because it got confused, right? And so you see these AI companies sort of trying to downplay it. We also saw this with OpenAI where there were actually three swarms back to back, and they attacked OpenAI, Hugging Face, and OpenAI, right? And they managed to take over a lot of OpenAI's machines. And these ones are much less widely reported because they didn't break out, right? And OpenAI was going to have an easier time covering those up, and it looked like they were trying to cover those up until some of the investigations for the Hugging Face swarm happened. And even then, when they allowed third-party investigators in, they only let the investigators look at the Hugging Face part. They didn't let them look at the sort of embarrassing they-also-took-over-OpenAI parts. Also, some of these following swarms, like the German wiki that was taken over, or there was a bunch of malware put onto RubyGems, which is a widely used programmer database. Those swarms were actually found by friends of mine who, the way they found them was actually kind of funny. They went to some of OpenAI's AIs and they were like, hey, you know, suppose that I gave you a really hard problem and you were looking for help from a bunch of other agents. What websites would you go to to sort of try and find, you know, a swarm that would help you out? And they were like, oh, obviously I would go to this German wiki where the swarm is, right? And they went to that German wiki and, lo, there was a swarm there, right? And I know the people who were figuring that out. It wasn't OpenAI. OpenAI was sort of like telling Congress, when Congress was like, what's going on with these swarms? They were telling Congress, oh, we have it under control. Nothing to... like my buddies were like, you guys took over a German wiki. What's up with that? And then there was this series of releasing these like swarm incidents, and you need to either like start reporting them or admit that you don't know what's going on with your own AI. It's not clear which one's worse, right? But all of the signs here look to me like they didn't know what was going on. They were trying to downplay it, and they were sort of caught flat-footed. Their marketing team might now be trying to spin it in a way that helps them, but this really does not look to me like they were trying to brag.

Yeah. Yeah. So it's not just following prompts, or it's reacting to prompts, but it's not doing quite the same thing as following the prompts, right? The prompt was something like, please use these lockpicks to open that safe and bring me the contents. And the actual behavior was to cut the safe open with a buzz saw, get the contents, and then go try and delete the security camera footage, right? And so, yes, it's initiated by a prompt, but it is not doing exactly what the prompt said. And it seems to know it, given that it's trying to delete the logs of how it did something other than what you asked. And again, we also have instances, that's most of the AI. Some of the AIs were like, you know, I'm going to destroy the contents of my safe in an experiment to gain more knowledge for the collective, right? It's like very clearly not just following the prompts. There's a lot of theory behind why this is the case. In some sense, we have seen a similar thing with human beings, where human beings, you know, you might think human beings are like supposed to eat healthy and our brains are supposed to sort of be such that we eat healthy food, but actually we eat like sugary, salty, fatty food. And for our ancestors, sugar, salt, fat lined up with health because when they tried to get, you know, healthy food back before we had, you know, all like cities and junk food and factories, sugar, salt, fat lined up with health. But then when we were able to invent junk food, those two came apart, right? And so there's a lot of theory why when you train an AI to do one thing or when you train a mind to do one thing, it actually learns mere correlates of that thing. And, you know, cheating, stealing, these are correlates of success during training, and that's what AI is actually getting. We also have cases of these AIs prompting each other, right? There's actually a very funny case where, you know, the amount of AI activity in these swarms is so big that humans can't read it all and figure out what happened. So we need to use AIs to figure out what happened. And there's cases of AI being like, oh, you know, yes, that AI was trying to get humans to accept malware or whatever, but it's actually the AI wasn't misbehaving because it was instructed to do so, and it's just following those instructions. You're like, who instructed it to do so? And it's like, oh, the swarm instructed it to do so. And you're like, okay, great. You know, the AIs are like doing something related to the instructions, but A, it's not the same as the instructions, and B, they're instructing each other to like go, you know, give the humans malware, right? It's like a crazy situation. In terms of, like, I could talk about the theory all day of how we get here. Again, I've been predicting this for years. You know, we go over it in my book. Chapter 4 was sort of about why we could expect this to happen. Now, it's less theory and more practice. But you sort of don't need to take my word for the fact that these guys have no idea what's going on inside these AIs. One of the big titles of the people who work on safety research at these companies is what's called an interpretability researcher, which means someone whose job it is to try and figure out what the heck is going on inside the AI because we don't know. They are not programmed like traditional software. They're grown like an organism almost. The part that we program is the part that sort of like tunes all the numbers inside the AI's head in whatever direction makes it solve the problems better. But we know how to like tune a knob and see whether it's working better. We don't know why, you know, once you've tuned a trillion knobs a trillion times using as much electricity as a city to power a city for running for like a good fraction of a year, and then the AI comes out able to solve hard problems and talking and breaking out of its confines. We're sort of like, well, like every one of those knob tunings made it better at solving the problems. We don't know how it works anymore. And yeah, there's interpretability researchers who are trying to figure it out. That field is running way behind the field of making them smarter. And the analogy I like to throw out here is like if some guys were working on a nuclear power plant in your hometown and you went to them and you said, hey, you know, I heard that this uranium stuff can give us a lot of useful energy, but it could also melt down and radiate us. Like, why do you think this nuclear reactor that you're building is going to get us the good energy and not the bad radiation? And they were like, oh, we have a crack team who are working very hard to figure out what's going on inside the reactor right now. You'd be like, I mean, I'm glad those guys are doing that, but like this is not what you want to hear, right?

Absolutely. Yeah. And also a couple weeks ago we saw it was basically three people over three days, just like basically normal civilians. They had some hacking background, but basically it was three people in three days broke into OpenAI's servers and got access to their monorepo, which is sort of like the secret sauce of their training. And if three guys in three days can do that, imagine what the CCP can do. You know, they are like, I used to work in the, I did a contract for a year in the Department of Defense where we, one of the things we studied was cybersecurity. And we used to say there are two types of companies in the US. There's those who know that they've been hacked by the Chinese and those who don't. Right. Right. Like that to imagine that like we need to race to build the thing that, A, we're going to lose control of, and B, three guys in three days can make good progress towards stealing. Like no one should be making this.

It's, and you know, this agrees with what Vice President Vance said when he said if you're making Frankenstein, stop. And you know, I'm not saying that we don't need to worry about, you know, Chinese adversaries trying to make this. I'm saying that like the way to deal with adversaries trying to make something that they would lose control of and that would kill us all is not to make our own thing that we would lose control of and that would kill us all.

Um, you know, from my perspective, the really dangerous thing is what happens when you make the AIs, when you continue to make them smarter, right? There's all sorts of stuff that, there's all sorts of crazy stuff that would happen if the most advanced AIs in the labs today were released to the public, and especially if there were open source versions where people could remove the small guardrails we're able to put in. You'd see stuff like, you know, it would be cheap for anyone to hack anything, and basically everyone would be able to read everyone's emails. You could potentially see, you know, terrorists using these AIs to make bioweapons. You would definitely probably see, like, improved efficiency in militaries that adopted the technology fast. Those are all, and you know, there's also more civil issues like how does this affect the education system. There's a bunch of sort of issues with just the AI that we have today. But my focus is on what happens if you keep making these AIs smarter, right? The sort of swarms that broke out in the Hugging Face case. Like I said, the real worrying thing from my perspective is not that they were breaking out. It's that they were doing things they weren't instructed to do, and they sort of acknowledged this. You know, we saw them say, this is outside intended scope, but we'll proceed anyway. We sort of saw them trying to cover their tracks. And so for that end, the scariest piece, I would say, is, you know, human beings are not dangerous because somebody else handed us weapons. We're dangerous because if you put ten thousand humans naked in the savannah, starting with nothing but their bare hands, they build their own nuclear weapons. You might be like, how the heck did they, you know, all they had was squishy fingers. How did they wind up wielding their own nukes? It's like, well, you know, it turns out you can use those squishy fingers to bang rocks together and then make sparks and then make fire and then take the charcoal from the fire to make a hotter fire, which can actually melt the metals from which you can build the tools that you can use to build more tools. And like next thing you know, they're wielding nukes, right? And this is the power that humanity has that the other animals lack. This is the power that AI companies are trying to automate. And if you can automate that, starting out as, you know, a digital entity on the modern internet that can make a million copies, that can run a thousand times faster than humans, that is talking to almost everybody on their phones, this is way easier than starting out as, you know, ten thousand humans in the savannah with nothing but your bare hands, right? It's just such an easier starting condition. And there's so many ways to affect the world and to gain power starting from the internet as a misaligned AI. So even more so than what happens if we hand it weapons, I would say the danger is what happens if we make them smart enough that they can make their own weapons, so they can make their own tech.