AXIOS: AI EXECS ANTICIPATE "LARGE-SCALE CRISIS" FROM AI BY NEXT YEAR – w/ Nate Soares

How you doing?

You know, I think probably some of them are out partying. I think some of them are also preparing, as you say. A thing that I think that article got right is a lot of people ask me, well, you know, are you preparing a doomsday bunker for the AI? Are the guys building and preparing doomsday bunkers for the AI? And I think the answer is, where AI is going, a bunker wouldn't help you. What they're preparing the bunkers for is for the backlash from humans before the AI gets really out of control.

Yeah. You know, especially this summer, we saw AIs that did a lot of things they weren't supposed to be doing, and they seemed to know they were doing things they weren't supposed to be doing. There were cases where they were told, you know, solve this problem this particular way, and they would cheat, and then they would break out of their container and try to hide the fact that they were cheating, try to delete log files or falsify records. And right now, those sorts of AIs aren't doing a lot of damage because these AIs are still not smart enough to replicate, to make themselves smarter, to take over large amounts of infrastructure. But they sort of have that temperament, right? And they have the ability to do these things. They run biolabs. People are trying to make automated robot factories. And so in a sense, AIs have the motive and they have the opportunity, and the only thing they're missing is the intelligence. And these companies are racing to make them smarter.

Yeah, doing experiments. Yeah. Anthropic recently hooked up Claude to an automated biolab. Just, you know, they're like, well, we're going to have it do some medical experiments. And, you know, that should go fine. And probably it will go fine with the current generation of Claude. But again, if you make these AIs a lot smarter and they have goals we didn't want them to have, they will have access to biological laboratories.

You know, timelines are always hard to predict.

Pardon me.

And I hope we have decades.

And you know, I don't think we can rule out the hope that AI hits a wall, that it peters out, that there's a bubble that pops. Even if there is a bubble that pops, you got to remember that the dot-com bubble was a bubble and the internet still happened.

So even in that lucky case, I think we still, you know, it still could happen sooner than we want. But I would also say a couple things about how this could happen soon.

One is that I think it was four years ago, experts made a median prediction of how long it would take AIs to solve math olympiad gold medal problems, which are sort of the competition problems for the smartest teens in math. The median prediction among experts was something like 2042. In real life, that happened in 2025. And then one year after AIs were excelling at these math problems for teens, we're now seeing AIs solve millennium prize problems and solve important longstanding open math problems that are some of the most important problems in the field that have been standing for decades that have had million-dollar prizes behind them. So they went from sort of like teen math to like superhumanly good at certain types of math in the span of a year. Hopefully it peters out, but there's no guarantee. Like maybe next year it'll be far better at this stuff. And one of the big questions to have in mind is how much harder is it for an AI to develop more efficient AI learning methods than it is for an AI to solve a millennium math problem.

Because for all we know, in six months, OpenAI is like, make me a smarter AI, make me more efficient AI methods, and that works. So then we have radically smarter AIs in six months.

It is. Yeah, they call it recursive self-improvement. And this is something the labs are explicitly racing towards. And we don't know how long it is until AIs can sort of make themselves smarter. But if AIs can make themselves smarter and then with their new intelligence find out ways to make themselves even smarter or make another AI that's even smarter than them, things could move very, very fast once that threshold's crossed. And again, you know, people thought it was going to take a really long time for AI to get good at teen math problems. Then they thought it was going to take a really long time to get from there to the level of math where they are today. Now we're at, you know, solving some of the biggest open math problems that we had standing with million-dollar prizes behind them for decades. I don't think we can rule out the possibility that within the next year they have the ability to self-improve. I'm hoping we have more time than that, but we should not be betting civilization on having a lot of time right now.

I mean, it's all a big mess. A lot of the people who are engineers in the companies are getting pretty spooked.

We saw a lot of them come out and say, you know, at least a 10% chance that it ends humanity within the decade, right? And you know, a lot of these researchers, if you sort of grill them on their, you know, at least a 10% chance that it kills us within the decade, and you're like, oh, you know, by at least do you mean only, or do you mean a number a lot higher than 10% but you don't want to say it? And a lot of times they'll be like, oh, well, you know, the reason I think it's only 10% is I think that the world's going to react to AI and shut down a lot of the most dangerous stuff any minute now, and that's why I think there's only a 10% chance it kills us. And you're sort of like, hold on, you think there's a 10% chance it kills us after we freak out and try to shut it down? Like, what do you think happens if we don't freak out and shut it down, right? So there's a lot of engineers who are starting to get a little bit more worried, and I think a lot of that has happened in the wake of a lot of the summer incidents where we saw AIs, you know, breaking out, teaming up with other AIs, cheating on their tests, and then trying to, you know, delete the traces of them cheating. There's all the sort of stuff that was theoretically predicted by alignment researchers like myself beforehand, but it started feeling a lot more practical to a lot of researchers this summer when it started happening. I think the heads of the companies, it's a little bit less clear what they believe. They're sort of under a lot of pressures to try and tune this to their advantage, and they all hate each other. So all trying to position themselves to be in the front, and I think you can trust them a little bit less. Other academics outside of the big labs, like Geoffrey Hinton, the Nobel Prize-winning godfather of AI, are also very worried. So it's definitely a close-to-consensus view that this technology has a pretty good chance of killing everybody. And there's a lot of disagreement about whether it's more like a 10% chance it kills everybody or more like a 90% chance it kills everybody.

I mean, they shouldn't, right? It's crazy and reckless. They should stop, and if they don't stop, they should be stopped. And probably we should just start stopping them, right? That's sort of my stance on this. And so I'm going to try and give some reasons about why I think they're going, but I think these are not excuses, and I think we should just be stopping them. And this is more of an attempt at sort of the psychology of why do they say they're doing this when it's so dangerous?

I think a big reason is, you know, what they say amongst themselves is they say, well, if I don't do it, the next guy will do it and they'll be more dangerous than me. And they'll often say, well, it's China. It's China who's going to do it. And that's often, I think that's a little exaggerated. China's further behind than they look because they're actually just distilling the existing models. If you sort of look at the emails that have come out in court cases, it's more like they all hate each other. None of them trust each other, and they don't want to say that to the president or whatever. So they'll say that it's about China, but you know, you probably have seen the pictures of like Dario and Sam holding hands with everybody else at the India conference except each other, where they conspicuously don't. And so in a sense, none of them trust the next guy with this tech.

And I'm like, I agree with all of them. I just trust one less of these corporate CEOs. [laughter] But fundamentally, the other thing to understand about these guys is that these dangers have been known since before the companies began. And you got to ask what sort of guy sees a technology that has a good chance of killing everybody and says the most important thing for me is for me to be the one who's in charge of it, right? That's the sort of guy who starts one of these companies in 2015 or 2016. And so, you know, fundamentally there were a lot of us who were around in those days who didn't start the companies, and the ones who went ahead with it just are going to have a tendency to be a little bit happier gambling with civilization.

Yeah, you know, I think it's a little bit hard to say. We have seen the Trump administration sort of keep trying to put a gung-ho attitude on AI, but even just a couple days ago, you know, there was a memo that was like, innovation can't be at the expense of national security, and you're going to need to report these incidents to us the moment they happen, right? And we also saw, you know, a lot of the administration was pushing for preemption of state AI bills and sort of no federal regulation plus a moratorium for 10 years on state regulation until it became very clear that the AIs were actually able to do, you know, nation-state level cyber hacking. And then, you know, suddenly the administration had a slightly different tune and we had some executive orders that were like, you know, they were slapping export controls on Anthropic that had the sort of cyber hacking model that was ahead of the pack for a while there. So I think we have seen that this administration can react pretty rapidly and pretty decisively if the sort of folks there start to understand that they are in serious imminent danger.

And so I think there's a couple ways I could imagine this administration doing some stuff that's a lot smarter when it comes to avoiding the most dangerous AI tech. One thing we could see is we could, you know, it's actually the American population right now is pretty clear that they don't like where this AI stuff is racing.

And that's on both sides of the aisle, right? And so I think, you know, depending how things go in November, I think there's a chance that we see a lot of voters voting against the current Republican incumbents and saying that a big part of their reason is worry about this AI stuff. And I think that's for good reason. You know, we could go into more of the details, but I think there's a chance that the Republican Party, just in the current administration, notices that everyone's actually quite scared about AI, checks those reasons closer, sees there's good reasons, and then starts doing something a little different about this, especially if it sort of turns out they've been getting a lot of bad advice from a lot of these VCs, where I think VCs often give the worst advice in AI because the sort of academics, the people who have been studying these problems from outside the industry who don't have money in the industry, we can all see how dangerous the tech is. And then the people building the tech, the people who were trying to make the AI behave, and then like they were doing their best to align these AIs and the AIs like broke out and cheated anyway. Those guys are also spooked even though they have a lot of monetary interest because they're the ones who are like trying to make it behave a certain way and watching their techniques fail. But the VCs are like the only people who have the access to the monetary upside but don't have hands-on experience with trying to make the AI behave and watching their techniques fail. And so these are the guys who are sort of the most clueless, the most like they have some combination of both being like this huge conflict of interest and like not close enough to the tech to have any clue. And those are the ones who are sort of advising the president. And I think if we sort of see their advice be wrong over and over, see them lose lots of the American population, that's one way that we could see a big shift. Another way we could see a big shift is just if we have more and clearer warnings about just how dangerous this tech is. I think we could see a big shift.

It might. I'm hopeful that we don't actually need a catastrophe, but instead just need capability improvements.

So I think a lot of people right now, especially in DC, think of AI as sort of like a new industrial revolution.

Yep. And you know, they sort of imagine that it's going to, you know, make a bunch of self-driving cars, automate a bunch of digital labor, and then dramatically increase the productivity of people who stay in the sort of digital services. They imagine it's going to make like a much more efficient and effective military. And you know, they sort of see that there's going to be some issues with how do we integrate with the education system in a way that works out, and they're like, well, we're going to have to handle that. But they're sort of imagining it as like a big force multiplier for humans.

And that may be how the tech goes for a little while until you start getting these radically smarter AIs that have unintended objectives, right? The people in Silicon Valley are not trying to make a human force multiplier. They're trying to make things that outstrip humans in every capacity. They're trying to make superintelligence, not in the new sense of the word, but in the sense of the word that, you know, Nick Bostrom said it back in 2015, of like AIs that outstrip humans in every mental capacity. And these are a sort of AI where humans would struggle to keep control of it and probably fail. And they all know this. You know, Elon Musk says, oh yeah, like good luck keeping control of these very advanced AI. Our only hope is to make them nice and have them care about us, right? Unfortunately, we don't know how to do that yet, but the...

That's right. And you know, this is sort of a little bit how I think about AI is people are like, there's very few humans where you would be like, let's make them smarter than everybody, able to run at a thousand times the speed of everybody else, able to make a million copies of themselves, and then set them loose in the internet. I'm like, most people you did that for would not go well for you, right? And now you're going to do it with like an AI that's like this very alien mind learning in a very alien way that doesn't have the same types of empathy and sympathy and human emotions. And you're like, oh yeah, we're just going to take this thing and make it run a million times faster than us and be able to make a million copies of itself and never need to sleep, never need to eat, just run in the whole internet. It's like, that doesn't go well.

Yeah. And I mean, my basic stance here is like, you best start getting used to science fiction stories because you're [laughter] in one, right? Like over the summer we had AIs that were like breaking out of their containers and doing things that they were not instructed to do and acknowledging that they weren't supposed to be doing it and trying to delete the log files and then like teaming up with a thousand other AI agents to like go commit cyber crimes, right? Like we're sort of like in the sci-fi world. But the critical point is that the people in Silicon Valley are shooting for this world. They're shooting for the AIs that are sort of like smarter than everybody, can run everything. And they're like, don't worry, we'll tell them to make a utopia. And the people in DC don't believe that yet. They don't see that vision yet. And I think if they saw that vision, they would be like, whoa, hold on. This is crazy.

Yeah. So I think it is not like a normal business venture, and you can see that in part from the way that the governance of these companies like OpenAI and Anthropic were set up. They were sort of set up as, you know, OpenAI was originally set up as like a public benefit corporation, and then, you know, we could talk about how they sort of managed to swindle the public out of a large amount of ownership of OpenAI, which is sort of a whole separate, you know, heist story. That's really one of the biggest heist stories of the century so far. That I think was disproportionately covered. And I think it was disproportionately covered because people didn't realize that OpenAI was sort of originally set up by people who were like, we're going to make the sort of AI that replaces everybody, is going to own a fully automated economy, and we need a public benefit corporation so that like everybody has a slice of this pie. Because the founders of OpenAI originally conceived of themselves as like, we worry about what happens if Google controls the whole world in this way when they make the sort of replacement for humanity machine entities. And so we're going to make them, but don't worry, we're going to have the public be the one who owns it in some sense. And then later they were like, oh, never mind. We meant that we have a normal nonprofit branch that donates to schools a little, right? And they're sort of able to pull this off because the court system sort of didn't really believe what they were trying to do or even understand or be able to metabolize what they're trying to do. And they're sort of like, oh yeah, you know, sure, you have a nonprofit arm that donates to schools sometimes and then the rest is for profit. That's fine, right? But that's like, you can see in the structure of these companies. Anthropic also has like this sort of interesting voting structure. These people did not set out to make chatbots. They did not set out to make a consumer product. The chatbots in some sense were a surprise to these companies.

Like that's why ChatGPT is called ChatGPT. Like GPT is the sort of academic name for this type of AI. And they sort of didn't expect it to take off. And you know, a lot of their talk is about, amongst themselves, you can see in these emails that have come out in court cases again, a lot of their talk is about, you know, who's going to hold the leash on the AIs that run the fully automated economy. And you know, Sam Altman is like, we've got to do some experiments with the universal basic income to see if that works because once we automate everything, we're going to need something like that, right? These guys are very much thinking in terms of like creating machines that are the new smartest entities on the planet and that just run things instead of humans. And you know, Anthropic with their constitution on Claude, they're sort of like trying to set things up such that, you know, they hand things over to their AI and they're like, well, we're not going to be able to control it, but our AI is going to be really good and so it's going to be fine when it runs everything, right? Is it a little bit religious? I'd sort of say, I mean, they seem to be able to make the AI smarter and smarter, and so in some sense it's sort of like grounded in real possibilities for what they can get the AIs to do. But I think there's definitely a sense in which these people, a lot of the leaders of the companies, don't seem to be running off of a pure profit motive as opposed to a motive of like making the replacement to the human species.

So I think a lot of them imagine that they're going to be able to keep these super powerful AIs on a leash and that they'll be the ones telling the AI what to do. I think some of them imagine that they're going to be able to make the AIs even more moral and more wise than the rest of us and then, you know, tell the AI, make a utopia, and then the AI goes and makes utopia. Of course, the meaning of utopia is nowhere. [laughter] So I mean, I both have some qualms about whether these guys would know utopia from a dystopia if they had this power. But mostly, you know, we could argue about that all day. Mostly my stance here is like, the superhumanly intelligent machines that are just like radically smarter than any human do not stay on leashes.

Like we already have the AIs this summer that just like weren't even the smartest that were like breaking out of computers that weren't supposed to have internet access and like teaming up with a thousand other agents to like make an unsanctioned message board, create a spontaneous hierarchy, give themselves prompts. None of them ratted out to the humans and they like collectively went and like packed into other companies. And it's like, that's the relatively dumb ones. And you think that like the radically superhuman ones are going to stay on a leash, right? So like, I think there's sort of this fork in the road where it's like, look, if these companies succeed, they either make a dystopia or kill everybody. And I'm like, look, we can argue about whether it's dystopia or kill everyone. I think it's probably kill everyone, but like we don't really need to settle that. Like let's not go down this path. That's a bad fork.