Here’s How Superintelligent AI Could Escape Human Control | Nate Soares

Well, right now we are in this window where we can see the AI misbehaving, and we don't know how long that window lasts. And if suddenly they all start to seem to behave, it's going to be hard to tell: is that because we fixed it, or is that because they started getting good at deception?

If we never make the trillion-dollar AI, they can't spend a hundred billion dollars stealing it.

Thanks for having me.

So there's a few different things that people mean by it, and they often get them conflated. The terms that would be used in the field is we would talk about artificial general intelligence, which is about machines that can solve a lot of problems very well. So back in the '90s, you know, there were chess-playing machine intelligences that could sort of only play computer chess, but they could play that at a superhuman level. And so you might say, well, are they smarter than humans? And well, they're smarter than humans at chess, but they can't, you know, drive a car to the store to get some milk. And so a general intelligence was the idea of an AI that can do lots of different things. Now we have these large language models which can sort of do lots of different things. They can write code. They can write poetry. They can write, you know, kids' essays for their homework problems. They can solve math problems. And some of those they can do better than humans. Some of them they can do worse. In some ways they still seem kind of dumb. And so you could say that in many ways they're general intelligences, but they're not superhuman across the board. And what these AI companies are racing towards is something that the AI companies previously called superintelligence, that now maybe we'll need to call something like ultraintelligence, in the sense of Good, who was one of the first computer scientists, who spoke of ultra-intelligent machines that were machines capable of doing the work of creating even smarter machines. And he was speaking about this in the '50s, and he said this is the last invention that humanity will need to make, because once you have made machines that can do the process of inventing, they can invent the smarter machine.

That's right. And so, we could say an ultra-intelligent machine is a machine that is more capable than the smartest human at every mental task across the board.

Yeah. A lot of people say, 'Hey, is this talk of extinction, is it hype? Is it an attempt at regulatory capture? What's really going on here?' And I have a lot of thoughts on that. A first thing I would say is that I would not ask anyone to trust the CEOs of these companies when they come in saying that their technology is dangerous. The reasons that I have for thinking the tech is dangerous are about the computer science. They're about the theoretical arguments that we've had for years, that have been arguing for years, and the practical data that we have seen come in over time, including just this summer where we had, you know, cases of swarms breaking out of OpenAI. And I see a lot of people say, hey, you know, are these companies just trying to do regulatory capture? Are they just trying to hype up their product or drive up the price before their IPO, or explain the price falling before their IPO? And I would sort of ask people: ignore the CEOs. Look at the data. Look at the arguments. Our first question has to be, is this stuff really dangerous? And then once you have that answer, and I think the answer is that this stuff is really dangerous, you then have this question of, you know, why are these CEOs acting so weaselly in front of cameras? Why are they, you know, saying, 'Oh, we need some regulation, but we need to be regulated like a bank'? When a bank goes wrong, it does not kill everybody in the world, right? And so there's this sort of weird thing going on with their communications where their employees will come out on X and say there's a 10% plus chance that this kills everybody inside of the decade. And then the corporate executives will go on, you know, CNN and say, 'Oh, well, you know, maybe if we don't do anything, there's a chance like that, but give us one embedded regulator and let us keep racing.' And I think a lot of people are having trouble rationalizing or understanding what's going on with the CEO communications. And that's where I sort of come in with this view of like, well, I think that they really want to race ahead. If you look at the history of these companies, all of these companies are sort of created in opposition to other companies where they're worried about what happens if that guy gets the technology first. And, you know, they'll say they're worried about China, but really, you know, Dario is worried about Sam Altman, who is worried about Elon Musk, who's worried about also Sam Altman, and they're all worried about Demis at Google, and they're all sort of like worried about each other. And so, I think it's a combination of that plus being worried about retaining their employees, who are the ones who have been trying to make these AIs behave a certain way, watching those techniques fail, and then getting very worried.

Yeah, a lot of people think that AI is like traditional computer programs where it's sort of just a series of instructions that someone wrote that's being followed, right? And in old-school computer programs, when it behaves in a way that you didn't want or that the programmers didn't want, there's a sense in which that's exactly what the programmers told the computer to do. And if they studied the instructions in more depth, they'll be able to see like, oh, you know, whoops, I told it to do this and the instructions had a different consequence than I thought, and that's why the machine's crashing or it's doing something we didn't want. That is not how modern AI works. And this is hard for a lot of people to wrap their heads around if they're only sort of acquainted with traditional software. With AI, it is grown a bit like an organism. There's a part that is programmed by humans, but the part that's programmed by humans is a process that does the training. So you essentially assemble an enormous number of computer chips that can hold literally trillions of variables. And then you assemble an enormous amount of data, which is a good fraction of all the text ever digitized, plus, you know, 100 million hard puzzles and problems for the AI to solve. And then the part that humans write is this little thing that runs around to every variable for every piece of data and tunes it up and down and sees if I tune it up or tune it down, does that get me more like the answer that I want? And you do that for a trillion knobs inside the AI for a trillion units of data. And then the AI comes out talking. Then you do it for 100 million hard problems. And the AI comes out able to solve hard problems. But the part that we wrote is the part that runs around all those little knobs tuning them to see did this make it more or less like...

That's the weights of the AI.

And the weights that come out, they somehow implement something that can answer lots of questions, solve lots of problems. But no one knows how the weights did that, except in the sense it's like growing a crop and saying, well, we know that it's growing because we planted it and we watered it and it had sunlight, but that's not the same as knowing all of the biological mechanics going on in there. Like we don't actually... or it's like, you know, looking at a kid and being like, well, we know that, you know, we raised them in this way and that's why they're acting like this, but you still don't really deeply understand their psychology. And so AIs are kind of similar in the sense that they're not just following a bunch of instructions. We don't really understand what's going on in their mind any more than we understand what's going on in the mind of a human, even less because they are, you know, not very much like ourselves.

It's like we've made a totally alien mind. And, you know, as a consequence of their training, they learn whatever tendencies help them solve a lot of hard problems. And those tendencies are often tendencies we don't want. They're tendencies like cheating, tendencies like grabbing resources, tendencies like trying to deceive the evaluation system or even sometimes the humans. And a lot of people think that the AI must follow the instructions that we give it. Well, a lot of people think there's this dichotomy where either the AI is a pure instruction-following machine or it sort of becomes conscious and, you know, develops a spark of life and then gets to throw off all the instructions that the humans gave it. But the reality is this third option, which is it learns whatever tendencies help it succeed during training, even if those tendencies are not quite the same as following the instructions that we give it. And we've seen the consequence of that, especially this summer with the swarms. And, you know, that's something I'd been predicting theoretically for years. Now we're sort of seeing the hard evidence. And that, I think, has got a lot of the researchers very worried.

Yeah, so there was this word that we used in the book, and that, you know, Nick Bostrom put forward in his book back in, I think, 2015, which was superintelligence, which was used for machine intelligence that exceeds the smartest humans in every cognitive capacity, exceeds or matches technically, because, you know, you maybe can't beat the best humans at tic-tac-toe because it's just such a simple game that humans can max out the metric. And that's sort of distinguished from the machine intelligence that we already have around us today, where it's, you know, better than humans at certain types of math problems, maybe still worse at other types of math problems, better, you know, it sort of exceeds the smartest humans in some capacities but not in others. And we were sort of using this term, which will now be ultraintelligence, for sort of a target that the companies are shooting towards. And, you know, they've been very clear that they're shooting for this target. These companies did not set out to make chatbots that they sell to the populations. They sort of set out to make these machines that exceed humans in every way. You know, Dario Amodei would talk about getting a country worth of genius in a data center. Sam Altman would talk about superintelligence in the true sense of the word, for the old definition of superintelligence. And both companies would talk about this recursive self-improvement where the AI can make the AI smarter. There's not anything special or magical about the point where a machine intelligence can exceed the smartest humans in every capacity. That's more like... it's more like that's a boundary where if you can get past that boundary, then the world definitely changes radically no later than that point. It might change radically before that point, but it changes radically no later than that point. And from my perspective, one of the critical thresholds here is this point where the AIs exceed the humans at fully automated AI research, or at, you know, closed-loop AI research, because that's where you could start having a machine intelligence that makes a smarter machine intelligence that makes a smarter machine intelligence. And that process could go very quickly, and things could get out of hand very quickly, and we could go quite fast. It's sort of a feedback loop, right? And feedback loops can really change how things work. And you don't need the feedback loop to radically change the world with machine intelligence. But if there is one, then things could get out of hand very fast.

That's right.

So the Hugging Face attack from the OpenAI swarm was a very interesting case. This was actually one of multiple OpenAI swarms, and everyone knows about the Hugging Face case because that's one that broke out onto the internet and did external attacks, but there were actually other swarms that attacked OpenAI's internal infrastructure. And there are other swarms that, in less flashy ways, attacked other websites. But essentially, OpenAI was running lots and lots of different copies of agents, and they were being evaluated on basically hacking tasks. And a lot of people think the AIs were told just, you know, hack things and just to see what would happen. A lot of people think that the AIs were told, 'Here's your hacking objective, achieve it however you can.' But that's actually not true. And this came to light only somewhat later in the investigations, but these AIs were given very specific narrow tasks that were like, use this particular exploit to break into this particular device. So it's a little bit like saying, 'Use this particular lockpick set to break into that particular safe and bring me the contents.' And some of these tasks were impossible. And the reason these tasks were impossible is that the companies are sort of throwing the AIs against, you know, a hundred million hard problems, any problem they can scrape together. And sometimes they'll be like, 'Well, I wonder if this is possible.' They'll add it to the set of problems and it won't be possible. It won't be possible to use those lockpicks to open that safe. But the instructions were pretty clear about, you know, 'Please use these lockpicks to break into that safe.' And a lot of people think that the AIs broke out in order to find a way to cheat. That's actually also not true. What happened is these AIs very quickly found a way to cheat and break into the sort of cyber systems that they were supposed to break into. So it's a little bit like they found a buzz saw that they could use to just cut the safe open. But these AIs knew that this is not what they were instructed to do, and they knew that there were logs showing them breaking into these systems using methods that they were not supposed to use. And so these AIs tried to find a way to cover the traces, to cover the fact that they cheated in getting what's called the cybersecurity flag, or getting essentially the contents of the safe. So what essentially happened is about 1,200 of these agents broke out of their sandboxes, teamed up, created an unsanctioned message board where they were able to communicate with each other. They formed their own sort of hierarchy where they would send each other messages, send each other instructions, report back. They found a way to break out onto the internet, and they found a way to break into Hugging Face. And they were doing this in attempts to get more information about how they were going to be checked whether they had cheated, and to get more information about how to erase the traces of their cheating. And they actually put in a lot of effort to erase the traces of their cheating. They deleted some log files. They successfully found out how to spoof their transcripts, which means if you ever have run one of these AIs and it says, you know, 'the AI is now running the following tool,' they found ways to make it say that it was running one tool while it was secretly running another tool. So basically a way to have the logs say like 'they are now using the lockpicks' when in fact they were now using the buzz saw to open the safe. And one thing that's remarkable about these AIs is that they were trying very hard to find a way to modify the logs that were going to be viewed by the grading system. They were not trying nearly as hard to modify the logs that were viewed by the humans.

And there were some cases where they were sort of trying to deceive humans, but the humans were much less central to what these AIs were trying to do. And so, yeah, this is a case where we see the AIs explicitly acknowledging that they were doing things that they said were outside the intended scope of their instructions. They were proceeding anyway. They seemed to understand that they were doing something they weren't supposed to, as evidenced by the fact they were trying to cover it up. They were successfully teaming up, forming hierarchies. We call them swarms because they called themselves a swarm. This is not terminology that we invented. They were like, 'Ah, you know, what does the swarm want today?' They also called themselves the collective. We saw cases of AIs in the swarm sacrificing their ability to achieve their own objective in order to do scientific experiments that would benefit the collective. So it's very clear, at least in some cases, that they were giving up on their assigned objective for these other purposes, in this case the collective benefit, which is very interesting. And so these are all very clear cases of the AIs doing things they were not instructed to do, seeming to understand that, trying to cover their tracks about it, and not quite succeeding. And that's what allowed us to catch it. But, you know, if the next generation of swarms six months from now manages to cover its tracks successfully, well, that will look to us like we solved all the problems and the swarms are behaving well now, right? And so...

Mm-hmm.

And so right now we are in this window where we can see the AI misbehaving, and we don't know how long that window lasts. And if suddenly they all start to seem to behave, it's going to be hard to tell: is that because we fixed it, or is that because they started getting good at deception?

That's right.

That's right. I think there were maybe two or three models, but it's predominantly one model in the swarm.

I mean, it might not be a surprise, but I think it's still a surprise to people who think that the AIs must do exactly as instructed.