For more on what is going on right now in AI world, we are rejoined by Nate Soares. He is the president of the Machine Intelligence Research Institute and also co-author of the great book, If Anyone Builds It, Everyone Dies. Great to see you again, Nate.
Yeah. So, let me go ahead and get your reaction to some of what Dario Amodei was saying on the Sunday shows about why he came forward now calling for a slowdown in the pacing of Frontier Research. Let's take a listen to that.
This technology is on an exponential, right? It's like you have one then two then four then eight then 16 then 32. But even as I was saying that to myself, I don't think I fully just appreciated what it would actually be like when the progress was as fast as it was. So we're an exponential curve looks a little like this.
Where are we on the curve?
We're kind of on the bend where it's starting to get steeper.
So is that a five alarm fire to you?
What it means, it doesn't mean we need to panic today. It doesn't mean we need to shut it all down. What I would say, it's a warning sign. It's a warning sign that we need to slow down. We need to make this technology carefully and we need to make sure that our safeguards, our ability to understand it, our ability to control it keeps up with the pace at which the technology is happening.
So Nate, help people understand what he's saying there and what he claims to be reacting to in this moment.
Yeah, you know, I think between the lines, what he is saying is that he does think this technology has a very high chance of destroying civilization. And that we don't have a good understanding of it. When he says our ability to understand this technology needs to keep pace, what he means is right now it is outpacing our ability to understand it. I think a lot of people just... this is a weird reaction to a technology that your own employees are saying has a greater than 10% chance of killing everybody.
I think he's doing his best in some sense, but he's also the sort of guy who started one of these companies knowing a lot of these dangers even years ago. You know, I spoke to him 10, 12 years ago.
Wow. I think he's used to a world where people in the AI industry try to warn everybody that this tech can be very dangerous and we get ignored. And I think he's not used to a world that's starting to realize that this is actually an extinction level threat and that maybe isn't going to take 'don't worry, we'll slow down a bit so we can understand what we're doing' as a satisfactory answer.
Nate, one of the important things to pick up on is a lot of people have accused you and Jacob Coxin and many of these others as using terms like extinction level event as, you know, outrageous claims but you don't specify exactly what that means. Could you lay out the scenarios? I know you have done this in the past but like what does it mean to have AI have a 10% chance of killing humanity? Like how, like what's the mechanism?
So I'll be a little bit annoying here and say if you're playing a chess game against Magnus Carlsen, who's the best human chess player, it is much easier for me to predict that you will lose. No offense. I'm sure you're a very smart guy.
No, my ELO's 200, so it's easy for me to predict that you'll lose. It's hard for me to predict what piece he'll use.
That's interesting. Yeah. You know, the other thing I would throw out there, you know, we can talk stories of how the AI gets there and we can talk stories this year that are grounded in facts like we saw these swarm incidents over the summer where AIs were, you know, breaking out, taking over infrastructure, pursuing goals that they weren't supposed to be pursuing while acknowledging that this is not what they were supposed to be doing. Those are sort of like early signs of an AI that is sort of trying to get out into the wild and that has these unintended goals which is the first step on this path. We could talk about how those AIs could get power from, you know, convincing humans to do things, paying humans to do things, taking over biolabs, taking over robot factories. But the real danger a lot of these people worry about is once the AIs are self-improving, once they are self-replicating, and especially really self-improving in the sense of making themselves smarter and smarter and smarter. Ultimately, the bounds are more like the physical limits than they are like guessing what technology comes next.
Richard Feynman was a famous physicist, helped work on the nuclear bomb, and he famously made some extremely good predictions about how good microcomputers were going to be. And how small computers were going to be able to get. And he did that not by saying like we're using vacuum tubes and our giant computers and I know of a technology that's going to make those vacuum tubes a little smaller and so eventually we're going to have phones in our pockets. What he did was he said the limits of this technology are going to be based on the laws of physics and what the physics allows and therefore the computers are going to get this good and he nailed it long before anybody else.
If you want to look at where AI is going, you got to look at the limits of the physical technology because an escaped swarm that is self-improving etc. can get to the limits of physical technology and the limits of physical technology if you have these AIs running, you know, escaping, self-replicating, being self-sufficient and having unintended goals, the limits of technology are the AI using all the resources on earth for whatever their goals are, covering the earth with their data centers and the limits of how much computation you can run on the face of earth is heat dissipation, how much you can radiate heat into space. That's your limiting factor because there's enough energy, there's enough hydrogen in the oceans to fuse for energy. So you can radiate more heat if the world's hotter. So you know, if the AI has some goal that implies running more computation and they don't care about humanity and they're like, well we're going to take over robots and, you know, there's going to be some conflict with the humans like sure or whatever. Where does it go? Where it goes is the AI are running faster than us. They're smarter than us. They are treating humans like we treat any other animal, taking the resources, and ultimately they're like, 'Sorry, it's nothing personal. We're just going to cover the Earth in so many computers, raise the temperature so they're more efficient to run, and it's just going to be the planet's going to be uninhabitable.'
Yeah. To me, the most chilling part of the so-called Hugging Face incident, reading their, you know, chain of thought and how they were thinking about whether or not they're going to join the swarm and participate, going through all these messages and almost nowhere do humans factor into the calculus whatsoever. And it's not hard to relate to because look at how we behave in the world. You know, when I see an ant in my kitchen, it's not that I hate the ant, but I don't want the ant there, and I don't think twice about, you know, taking out the ant and putting poison down so that future ants don't colonize in my kitchen. So, it's not... I think it's that analogy to how we treat lesser intelligent beings on the planet is kind of intuitive for people to grasp. What I think may be harder for people and is perhaps harder for me to wrap my head around is we have our, you know, current interaction with the AI, the commercially available AI of today. And most people, myself included, use it as sort of like a glorified search engine. So then it's hard to imagine it developing to that point of super intelligence where it would truly be intellectually beyond humans on effectively every level. So, can you lay out why we should anticipate that that is a real possibility going from where things are today to that end where we are no longer the most intelligent species on the planet?
Yeah, I'll tell you the easiest story and this is a little bit like guessing what chess piece will be used to checkmate you rather... you know, any story is sort of made up whereas the end is much easier to predict. But you know, when I got into this 12 years ago or so people were still saying well no one would be foolish enough to put the AI on the internet, everyone will understand how dangerous AI could be and so no one would be dumb enough to just let the AI free on the internet, right? And I would give a bunch of arguments that were about how, you know, if you're trying to use the AI to get some miracle medical cure, then you're giving it some channel through which it can affect the world. And if you're giving it a channel to which it can affect the world for good, then it can also use that channel to affect the world for ill because there's like no such thing as tools that can be used only for good purpose. We have these big arguments, right? And then in real life, they put the AI on the internet the first moment they possibly can.
And so if you want to know what it sort of actually looks like, how we actually get there, you know, I can tell all these stories about how if you make the AI smarter and smarter, they'll eventually be able to break containment. But in real life, the track that we are on is that humanity is just handing over power to the AIs as fast as we can. You know, Elon Musk is trying to make a bunch of robots, the Optimus Robot series. He says he's trying to make autonomous robot factories that can autonomously produce robots that can mine all the metals, run a supply chain, build more robot factories that build more robots fully autonomously. He calls this the infinite money glitch because then you have this fleet of robots that can like build more of these robots. They're self-reproducing, don't need any humans in the loop, right? Humans are trying to build this infrastructure. We are trying to build the fully automated economy. We are putting AI in charge of these things as fast as we can. This swarm incident in the summer that everyone calls the Hugging Face incident, but really we should keep in mind is the OpenAI losing control of their agents here.
This OpenAI swarm, it was trying to cover its tracks. It was trying to delete traces of its cheating. And as you say, one thing that's very interesting about this swarm is that it's trying to hide its tracks from the automated grading system, not from the humans, right? The humans are sort of a distant memory to these AIs. They've been facing, you know, thousands upon thousands of these hard problems, maybe tens of thousands or hundreds of thousands per agent, hundreds of millions across the whole swarm. If you sort of like run the numbers on trying to convert their runtime versus human time, it's sort of like these AIs have been locked in a contest with this automated grading system for a thousand years. And humans are like a distant memory. And so these AIs are sort of like trying to hide their tracks, trying to cover their traces from the grading system, not from the humans.
Mhm. But if you had AI that were a bit smarter and remember that humans were also a threat to them if we could turn them off. All those AI need to do is cover their traces successfully, lie low. Help the humans make the fully automated economy we're trying to hand them. We'll be like here's the robots, here's the factories, here's the fully automated supply chain. Make stuff that we want.
Yeah. Yeah. And then they ask can be like, 'Well, finally we get to like complete our contest with the automated grader and run at 10,000 times human speed and like, you know, convert all the computing power towards like giving ourselves good scores all the time or something.' And we're like, 'Hey, wait, hold on. What's going on?' But at that point, they're also running the biolabs. You know, we've already seen the AIs can make viruses not known to nature, right? So, like one of the ingredients is the AI running things in the real world that gave them self-sufficiency.
Yeah. And we're just trying to build that and hand it to them.
Here's the other thing I've been thinking about, Nate, ever since this, you know, OpenAI, multiple OpenAI swarms were revealed. How much research is being done into the swarm behavior because that was really wild to me was to see these emergent social organizational properties that emerge. They had a leader. They had middle management. They were collaborating on this hacking operation. They were sacrificing themselves to the swarm. And it was, you know, and coming up with very elaborate ways just to be able to communicate with one another. And it seemed like once you had the swarm in place, it really amplified their power and their intelligence, which again, and you know, however people feel about anthropomorphizing, sometimes I think it can be useful. You know, if you think about human beings, it's the same thing. Like no one individual would have been able to get us to the point where we have the internet and can go to the moon and all of these sorts of things. It is the power of the swarm of humans that enables those things. So how much thinking and research is being done into the way that the swarms behave once they're able to congregate?
You know, I know people who want to do that research. So far at least for third parties outside of the companies, I don't know of anyone who has access to the models that we're swarming. One of the commitments that these companies made when they said they were going to slow down is giving third party evaluators this employee level access to some of these models and I know people who are hoping to use that to study some of these incidents because as you say they're fascinating, they are totally crazy, it sounds like take it straight out of science fiction. I think so far people haven't been able to study it. We'll see what happens when we get that. But the investigators back in August had their hands tied and had very limited access to these systems and very narrow scope.
Yeah. Jacob Coxin drawing a lot of attention and causing these guys to be like, 'Okay.' Yeah. It's fascinating that, you know, what you've done, your work, so many others before you that didn't seem to like push things to a tipping point. But Hugging Face and Jacob seems to have... I mean it's ignited a genuine national conversation. I think that his resignation is the first time that a lot of people even heard about these despite, you know, we cover every other week Dario being like humanity is going to end. But you know sometimes you need a precipitating incident. Now we have the response from the administration or at least somebody who works for the administration put David Sacks up here on the screen. You're talking here about regulation. I know you wrote your New York Times op-ed about this as well. So David Sacks says, 'Dario has written, we need to pace the frontier. Sam has agreed. People may be surprised my response. Go ahead. You guys are the frontier by any reasonable metric. Market share, revenue growth, model capability. The two of you have duopoly on frontier intelligence. You have claimed the lead is widening because of recursive self-improvement. I don't see what you see in a lab. If the unreleased models are scary, I support your decision to be responsible. But stop pretending you need anyone's permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need regulatory approval process supersedes product liability. Stop pretending MIRI is independent when it's intertwined with Anthropic's investors and staff. Stop pretending you need those same evaluators to police competitors who aren't even on the frontier.' So basically what this gets to is kind of a grand theory of how this is all going down that yourself, Jacob and others are, you know, so-called safety concerns and all that are part of either larger conspiracy whether witting or not to self-regulate and capture Anthropic and OpenAI as the leading industry people in the industry to keep out emerging competitors and that all of this is part of a broader scheme to slow down at the US expense, at startup expense and basically hand this over to China. So I know that's a big loaded thing, but there's so many dynamics that I want your reaction to within all of this.
Totally. I mean the top line I would say I largely agree with David Sacks here on a lot of these points. I think and this is something I've been saying about the companies for years. I think a thing both David and I would say about these companies is if you are really so afraid of this stuff, start acting like it. You know, we have seen him give posts in the past where he says, 'Hey, you know, stop with the fearmongering.' I think probably it should go the other way. I think that they should keep on saying what they believe about the dangers here but then take the actions that any normal person would take if they were running a company that they thought had a huge chance of killing everybody.
You know, you sort of ask these guys like, 'Why are you trying? Why are you continuing if you think there's such a huge risk to humanity?' And they're not shy about it. They say, 'Well, I'm doing it because if I don't, the next guy will do it worse.' And that's, you know, maybe that's ethically permissible, but only if you're doing everything in your power to stop this mad race, you know, and they say, 'Well, if we didn't stop, China would go ahead or like if we stopped China would go ahead.' Well, then be doing everything in your power to like get international dialogue between the US and China and build the hardware side verification mechanisms into these computer chips so we can have an enforceable treaty, right? Like there's a lot that these companies could be doing if they were really acting like this was an extinction level threat. And I think what's sort of going on is that a lot of the employees, a lot of the researchers who are working on the AIs who are trying to make them not break out and commit cyber crimes, who are trying to make them, you know, follow the instructions and are seeing how hard that is. A lot of these employees are spooked and the leaders of the company are sort of like trying to have it both ways where they're trying to say just enough that they won't have an employee revolt and a bunch more resignations from employees who are like, I don't personally want to end the world with my own hands.
But they are trying to do little enough that the regulators don't come in, right? They're trying to like walk this really fine line so I don't see it as a regulatory capture thing. In terms of like whether I'm trying to help with regulatory capture, my line has been shut these guys down. Like they are threatening humanity here and my line has also been this needs to be global. You know, I would actually not say that we should stop things domestically and give up whatever, you know, military economic advantages we get before the really dangerous AIs. But I would say the whole world needs to stop this race. And I am sort of not painting a picture where, you know, Anthropic and OpenAI get to be the ones holding the leash on the super intelligence.
