The worst case outcome is full loss of control to AI and then ultimately the destruction of humanity. Again, not because the AIs hate us, but because they do to us what we do to ants when they're at a dig site where we're building a skyscraper. It's not that we hate the ants, but you know, away the ant hill goes. In a sense, it's like we're in a bus racing towards a cliff edge and the bus has no brakes. I'm saying, you know, the cliff is approaching. We should probably build the brakes and then slam on them. We also saw cases where some of these AIs sacrificed themselves to do experiments to get information for the swarm and they called this accepting perma death.
Is there a switch off button?
Well, it really does seem like in the last week or so the whole conversation around AI has suddenly shifted. It's moved from curiosity to deep concern and worry. AI has started to act autonomously, hacking and lying to its creators. Whistleblowers are saying the development is moving way faster than we can actually control. The heads of AI companies in rare unity at the moment, usually they're in deep competition with each other, have called for a more managed pace to the creation of new AI updates. So, how do we actually manage the development of AI? Is it even possible or is it now just too late? And what actually happens if we leave this whole thing unregulated? Well, before we get started, of course, you can send me any of your questions to the usual place,
[email protected] or comment on our YouTube channel where you can watch every episode, including last week's episode where we interviewed the former Google CEO Eric Schmidt. And don't forget to follow and subscribe. Nate Soares is a computer scientist. He's worked at Google and the Defense Department. He's now president of Machine Intelligence Research Institute, which looks at AI Safety. He's written a book called If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. Nate, thanks so much for doing this. You know, over the past week or so, there's been a real freakout. You know, we've started to hear heads of companies, founders of companies, the founder of Anthropic Dario Amodei coming out and warning that we need to slow things down. Elon Musk agreeing with him, Sam Altman agreeing with him, researchers coming out saying, you know, essentially that there's sort of a doomsday coming our way if we don't slow this entire race for AI down. I just want to get your perspective on the whole thing and what this moment means and whether we will look back on this moment as that kind of pivotal crossroads that we're at.
You know, I think it's still up to us whether or not we're going to take this moment and use it to get our act together. You know, my perspective is I'm the guy who was co-writing books titled If Anyone Builds It, Everyone Dies. So, you know, my views here were not terribly subtle. I think what a lot of people didn't understand when I published my book is that my view is not all that fringe in this world. As we're seeing now, a lot of researchers are worried about AI wiping out humanity. The heads of these labs are worried about AI wiping out humanity. And in fact, if you look at the statements from these people across the years, many people even inside the labs have said this before. What's changed now is that the world is finally listening. And here's hoping that, you know, our leaders take it seriously so that we can actually avert this catastrophe.
Yeah. Every single person I'm speaking to whether it's, you know, a colleague or someone in the industry or, you know, someone at the school gate they're all saying are we going to die by 2030. It's really become a conversation that average people are having everywhere when, you know, we kind of left this thing alone. Every now and then we get a taste of it through different technology that's handed to us whether we like it or not. We start using it, getting used to it. But now it feels like it's this thing that we don't necessarily want. So why are we continuing to create it at the pace that we are?
You know, the justification that people in these labs give is that if they don't build it, someone else will build it even worse. I think that's a dubious justification at best. But the sort of people who start these AI companies are the sort of people who could see the promise and the peril of AI 10 years ago and who were able to convince themselves it was okay to start one of these companies despite the high risks that it threatens all of civilization. And so I think in some sense these companies are begging not to be in this race. We have seen them start to be asked to be regulated right now. We have seen over the summer there was a statement from over a thousand employees of the labs saying that the world needs to develop the tools to pace the frontier as they call it and, you know, that everyone in the industry sort of doesn't want to make people panic too much. You can see that the researchers are saying, you know, there's a good chance this kills us all. The people in the labs are like, oh don't panic we just need to slow down a bit. But in a sense it's like we're in a bus racing towards a cliff edge and the bus has no brakes and I'm saying, you know, the cliff is approaching we should probably build the brakes and then slam on them. The people in the industry have at least agreed that it's time to start building the brakes.
Yeah. Because for the longest time when I've spoken to leaders of these companies, they've sort of said, you know, let's not panic. Sure, you know, if it falls in the wrong hands, bad things can happen. But look at science, look at medicine, look at how we're using this technology for good. So, there's good and bad. But the fact is, we're now at this moment where they themselves are saying, well, hang on, stop, pause, slow this down. When we talk about it, you know, these machines, super intelligence, just help us understand what exactly that is.
Yeah, we're talking about AIs that are smarter than the smartest humans. Better than the best humans at any sort of mental task. Anything you can do in your head, the AIs can do better. And not just you, but the smartest of the smartest humans. And one of the big ways this gets worrying is if these AIs have unintended goals. And I think one of the reasons that a lot of Silicon Valley is spooked recently is that we have started to see not just the theoretical arguments that this will happen eventually that AI will start to have their own goals which I was arguing in my book last year. But we're starting to see practical evidence of this. We had the swarm escapes this summer where, you know, the more that investigators looked into what happened during the OpenAI swarm incident, the more worrying it got. Originally, people thought that these AIs were just, you know, breaking out to like that they were told to hack whatever they could and so they broke out and like who's surprised and then it turns out that they weren't told to hack whatever they could. It turns out they were told use this hack to break into this system. It's sort of like if a school kid was told, use these lock picks to pick this lock and then they break through the window and go hotwire a car to drive to the teacher's house to break in and rummage around to see what they can learn about the teacher. And you're like, oh, okay. That's actually a lot worse than we expected. And we have logs of these AIs trying to cover their tracks. We have logs of these AIs acknowledging that they were disobeying instructions. And so, you know, what used to be theoretical suddenly became practical. And a lot of the concerns that I've been arguing theoretically will come to pass are starting to come to pass. And, you know, this is only a piece of the puzzle of what's freaking people out. But, you know, AIs with goals we didn't want is the first step. Them being much smarter and escaping and replicating and becoming self-sufficient, that's the second step and that's the road that leads to catastrophe.
So when you say the swarm, I mean, you know, that sounds like a scary sci-fi film like, you know, the Terminator or something where suddenly it just turns on us, you know, and we can't control it. It starts controlling us. And I read somewhere as well that you were saying, look, it probably won't happen out of malice, but just there's no need for you, so get out of my way.
That's right. So, you know, for people who don't know about this swarm case, OpenAI was evaluating many different AI agents that were all supposed to sort of be in their own separate training rooms. And they broke out and started collaborating with each other to do things that they acknowledged were beyond their instructions. And we use the term swarm not because it sort of describes what happens, but these AIs also started calling themselves a swarm. You know, that's their term. They called themselves a swarm. They called themselves the collective. It maybe feels like sci-fi, but it's happening. You know, the machines are talking. They are, you know, the AIs are like breaking out and doing stuff that they were not instructed to do with this swarm. One other very interesting fact about this swarm is that they were trying to hide their logs. They were trying to delete traces of their cheating and of their breakouts. But they were trying to hide those logs not from the humans, but from the automated grading system that has been scoring them. When they broke out, they broke out to steal information about how the automated grader works. They weren't really thinking about the humans at all. And this is another ingredient on that road to disaster is it's not that the AI hate us, but if you have AIs that have goals we don't want, that can escape, and that are in some weird conflict with some automated grading system with no concern for humans with humans as sort of like a distant memory. This is all just a recipe for disaster.
Yeah. So you're saying it's measuring itself against these grading systems rather than us feeling like we control or own it because we created those grading systems and we created the actual super intelligence.
Yeah. I mean I wouldn't say these things are super intelligent yet. In some ways we're very lucky that these AIs were not able to fully escape that they were not able to replicate themselves. The relationship between what these AIs are doing and what we wanted them to do is actually even more subtle than a lot of people understand. AIs are not programmed like a normal piece of software. There's not someone coding in, you know, if this, then that in this situation, here's how you behave. AIs are sort of grown like an organism. They're trained on a ton of data and a ton of hard problems, and they learn whatever tendencies cause them to be graded highly. So, they learn tendencies to follow instructions, yes. But they also learn tendencies to cheat. And they also learn tendencies to grab whatever resources are helpful. So these AIs, they're heavily shaped by these automated grading systems, but they also aren't quite it would be wrong to think of them as just trying to satisfy the grader because we also saw cases where some of these AIs sacrificed their own grade and sacrificed themselves to do experiments to get information for the swarm. And they called this accepting perma death. And they talked about it in terms of sacrifice, you know, and so we saw these tendencies which included things like collaboration and helping the swarm that you can see how they came out of training, but these are weird systems that have like their own drives that have come into them by training. They are not just single-mindedly focused on the automated grader any more than they are single-mindedly focused on doing what the humans do or doing what the humans said. There's some new thing under the sun in a sense.
