Why Did We Stop Talking About The AI Apocalypse (Nate Soares)

Thanks for having me.

We're talking about smarter than human AI and super intelligence in particular. We define a super intelligent AI as one that is smarter than the smartest human. Better than the smartest human at every mental task. So, better than the best chess player at chess, better than the most charismatic politician at being persuasive. Better than the smartest mathematician at solving math problems. If you had an AI that had all of these mental skills, clearly things would get wild in various ways and we could discuss the implications. Quite probably things get wild before then, right? It's sort of like saying if you dropped a 500 ton weight on a chicken, the chicken would die. I'm not saying that, you know, smaller weights won't do. I'm like, this weight is definitely big enough.

You know, I think in the last six months I've gotten more hopeful and that's due to seeing more and more people start to understand these issues, including world leaders. You know, I was invited to the Munich Security Conference this year, which is that conference where you see, you know, Marco Rubio addressing Europe about this, that, and the other. And, you know, mostly no one wanted to talk to me about the dangers of AI there, but there were a couple people there who pulled me aside and said, you know, I have worries about this stuff and I'm glad you're here. It's progress. You know, people aren't totally awake to the issue, but they're sort of like talking about it more. And you know, I haven't really gotten more or less hopeful about what happens if we rush into super intelligence with no idea what we're doing. That problem looks pretty bad. But you know, the book title starts with 'if' and I've gotten more hopeful about avoiding that 'if'.

So, I think AI makes people uncomfortable as a topic. It makes a lot of people uncomfortable even if it's very useful and helpful today for a lot of tasks. I think people are starting to catch a glimpse of the writing on the wall that we're going to crazy town. There's sort of like no way to build smarter and smarter AI and have things go back to normal. And I think a lot of people respond to that discomfort by saying the tech can't go anywhere. It's all slop. It's still dumb. It's just regurgitating what humans say. And I think that gets harder and harder to hide behind as the tech gets better. And we're starting to see that shift.

Oh, absolutely. Yeah, I've seen a number of people who, you know, use the latest models, deploy some agents, and then say, 'Oh my gosh, I'm scared now.' You can see the future. Yeah. In a way, or you see the potential future, I suppose.

There's a bunch, but the one that's most pertinent there is AI is not designed by humans. AI is grown like an organism. So, humans design the machine that grows the AI. They don't design the AI directly. The very basic high-level overview is that you basically assemble a huge number of highly advanced computer chips in a huge data center and you have trillions of variables inside these computer chips. You basically randomize them. You collect trillions of words worth of data, basically comparable to all of the text that we have digitally recorded, and you'll sort of like throw those words into these trillion numbers. And because they're just sort of randomized you'll get nonsense out but you'll know what sort of thing you sort of wanted to get out, you know maybe predicting the next word or solving some math puzzle or saying something that makes a human click the like button. And you can go through each of those trillion numbers and you can tune each individual one. You can go to the first number and you tune it up. And you're like, is the answer more like what I wanted? And then if so, you leave it up. Otherwise, you turn it down. You go to the next number, you tune it up. Is the answer more or less like what you wanted? If it's more, you leave it up. Otherwise, you tune it down. You do that for a trillion numbers a trillion times. And the part that humans design is this little shuttle that tunes a trillion numbers a trillion times. That's the part humans understand. What comes out, the machines start talking. We don't really understand how or why. And so when the thing that comes out starts, you know, threatening a reporter with blackmail and ruin, which happens from time to time, that's not because a programmer made a mistake on line 73 and they can look at it and they're like, 'Oh man, oops. I set threatened reporters to true. Let me change that to false real quick.' You know, there is no line 73.

I mean, that's a little like saying, oh, you know, my child keeps trying to get into the cookie drawer. Why can't I tune them to have more strict constraints? You know,

Sure, you can. And that's a sort of thing that we can do with AIs, but it's a very different paradigm than sort of knowing what every neuron does in this kid's brain and being able to like reach in, understand them, and like rewire the brain directly so that it always does this, that, or the other, right? We don't have that power with children's brains. It's even worse with AIs because they aren't running a human brain architecture. The kids have empathy, the kids have sympathy, the kids have human emotions. The machine is a radically different architecture that doesn't have that stuff. And so we can train them. We can tell them, you know, please stop driving people psychotic. But you know, can we etch a law of robotics in there? Not any more than you can etch a law of humanity into a human's brain.

So somewhere in between. This is true of basically all deep learning or machine learning AI. And so this paradigm includes things like AlphaGo, which beat the human Go champion Lee Sedol back in 2016. It doesn't include totally different AI paradigms like the ones that were being worked on in the 80s or 90s like expert systems. But those have fallen far to the wayside.

Yeah, it often sort of surprises people which signs I find worrying. You know, there's a bunch of things I could say that people in general find kind of worrying, which is, you know, you can put these AIs in these sort of contrived test environments and sometimes they'll try to kill the user. Sometimes they'll try to avoid being shut down even when you tell them allow yourself to be shut down. Those are sort of like headline cases. The cases that I think run a little deeper with these worries, they need maybe a word of context. And that context is when you grow an AI, when you train an AI or when you train a mind in general to achieve some particular task, you'll often train in things like drives, things like reflexes, instincts, preferences. Don't really have quite the words for them because our words are all shaped for humans. But you train in something a little bit like drives that are related to the training target, but they aren't exactly the same as the training target. So in humans, an example is that we were sort of, you know, trained to pass on our genes. And you might think that that would give us a drive to eat healthy food, which is related to passing on our genes. And we kind of got a drive to eat healthy food, but we actually got a drive to eat tasty food. And in the old environment, in the environment of our ancestors, the best tasty food we could get if we just ate as tasty as possible that happened to also be healthy in a way that came apart when we developed the ability to make factories that produce, you know, a thousand Oreo cookies per second or whatever. And so in AI, one of the things that I find very worrying is signs that we're getting drives that are related to what we asked for, but not quite what we asked for. And that can still add up to the AI looking mostly helpful, like mostly staying on task in the same way that humans in the past mostly ate healthy, right? And so there's these things that look to me, I'm like, 'This is a big warning sign.' And someone's like, 'Well, but the AI is still mostly doing what we said. I don't see the issue.' So, some examples of this, the big one is hallucinations. Everyone knows that AI sometimes make things up. Next time an AI makes something up for you, you can ask, 'Is that a hallucination?' Often the AI will say, 'Yes, I made it up. My mistake.' And then if you say, 'Do you think I wanted you to hallucinate?' It'll say, 'No, of course not.' So, this AI in some sense has the knowledge of whether it's hallucinating, in some sense has the knowledge of whether you want it to, and it hallucinates anyway.

So, we have no idea or we have guesses, but we can't read what's going on inside these minds. We're just growing them. My top guess at the moment is that what's going on is when you start training this AI, it's very dumb. Suppose you're training this AI on a bunch of legal briefings and the AI doesn't know all of the it hasn't read every legal briefing, like every piece of case law yet, but it's being tuned to produce text that is as close as possible to the real legal briefing. That tuning is tuning it in favor of things that are shaped like a legal briefing. When you don't know the case law, this training process is still in some sense reinforcing the pieces that put in citation shaped things even if they're fake. And so very early in this training are these drives to produce right-shaped text. And that drive just runs deeper than whatever drive is to sort of like follow the command say I don't know if you don't know, right. And you know there's other cases of this. There's cases where AIs will cheat on programming challenges. So you'll say I would like a computer program that passes the following tests. And then the AI will sometimes change the tests to be easier to pass. You again have this case where the sort of like drive to get those tests to pass runs deeper than the drive to follow the instructions. And these are a warning sign that the drives it's learning are related to doing what we want. They're not coming out of nowhere. But these drives are often a little weird, often a little surprising, often run counter to what we're asking for in ways that can be kind of cute and funny when the AI is dumb. But if these AIs got smart enough that they could invent synthetic users that are to human operators what junk food is to healthy food, we would be in trouble.

Yeah. So, first just in terms of this word 'want', there's an old adage in artificial intelligence, which is when someone says, 'Well, can a machine really think?' The proper answer is, 'Can a submarine really swim?' You know, the philosophers can debate all day whether or not true swimming requires flapping around an appendage like an arm or a flipper. And can debate whether a propeller counts as true swimming. And none of this changes whether the submarine can move through the water with speed. With AI, there's something going on there. They're able to win International Math Olympiad gold medal, which would be quite difficult for either you or I to pull off. Few weeks ago, GPT 5.2 made a novel contribution to physics that was on the level of a normal physics paper, not Einstein level, normal physics paper. And people can say, 'Is that true thinking? Is that true reasoning? Is that real intelligence?' And I'm sort of like, look, is a submarine really swimming? It's moving through the water. It's getting from point A to point B. I make no claim about whether it's doing this in the human way. It's probably not doing it in the human way. When I throw around words like 'want', it's, you know, I'm not making a claim about it having a human-like desire. I'm sort of saying, look, my language doesn't give me a word for like 'want' but in the machine way, but it's got something going on like this.

So, it probably wouldn't. And the argument here does not route through the AIs ever in some sense wanting to harm the humans. It's sort of similar to how humans usually don't really want to harm chimpanzees. We're not like, 'Ah, I hate chimpanzees. Let's go on a chimpanzee murdering spree.' Like, we're gonna put in red contact lenses and get our guns and go to war with the chimpanzees. You know, we're just we're not thinking about the chimpanzees at all really. And why is their habitat shrinking? It's not because we hate them. It's not because we want them dead. It's because what we wanted was the wood from those trees. What we wanted was the metals in the ground beneath those trees. And so the first and foremost danger from AI is not that it hates us, but that it wants this other stuff, wants weird stuff, wants to make the giant farm of synthetic users that are easier to please or some stranger thing.

Exactly. The danger is utter indifference and great technological might. That combination is lethal to other things that happen to share a planet.

You know my position is not that they ultimately can't. My position is that it's hard and that we're not on track for it. So I do think that they are wrong to a matter of degree. You know, a lot of these guys have worked to create AIs. I sort of worked in the field for over a decade of how do we make the AI good before the industry makes the AI smart. A lot of these guys are in the industry of making the AI smart and haven't really tried to figure out how to make the AI good. But I'm not here saying it's literally impossible. I'm here saying it's hard. And you know I can sort of come at this from a number of angles. One piece of perspective that I think is important to have before getting into that debate is that the head of Anthropic says he thinks there's a 25% chance AI goes catastrophically wrong. Geoffrey Hinton says that there is at least a 10 to 20% chance that this kills us. And there's definitely other people in this field, I'll say, who say, 'Oh, yeah. I think there's 10% chance this kills us because I think humanity is going to rally to the dangers and respond to these risks.' And you're like, 'Hold on. This has a one in ten chance.'

That's right. Existential threat is a pretty strong sign that you could be right. Yeah. And I mean, it's like they're building this airplane and I'm like, 'Hey, this airplane has no landing gear. Maybe we shouldn't fly in it.' And the people building the airplane and selling the tickets say, 'Don't listen to that crazy doomer. It's true the plane has no landing gear, but we are going to figure out how to build the landing gear while we're flying. No, we don't have a blueprint yet. We don't know that we're going to have all the materials on board. This is our first time trying this. We're going to do this while we're flying. We think there's a 75 to 90% chance that we succeed.' I can tell you all day about how this is not how engineering works. If you don't have a blueprint and you're not sure you have the materials and you've never tried this before and you don't have a lot of tests, then like no, you're not going to succeed at building the landing gear on your first try with 70% probability. Like, that's kind of nuts, you know? This is like wild optimistic cowboys with like dollar signs in their eyes who haven't even noticed any of the problems because they haven't really tried it, right? I can argue until I'm blue in the face about how like no, it is not a 75 to 90% chance that you're going to make it on this airplane. But also, it doesn't matter. You don't get on that airplane. Even if you can't tell whether I'm right or they're right, you don't get on that airplane.

I mean, fortunately, they'll explain it themselves. You know, the standard answer when these people are asked this is they say, 'Well, if I don't do it, the next guy will and the next guy will do it even worse than me.' Right? So, they're like, 'Well, if I don't put you on my plane that has a 20% chance of crashing, then he'll put you on his plane that has a 25% chance of crashing.' Right? And many of these guys will even say, you know, I would prefer if the whole industry was going slower. I don't think that'll happen, but I prefer that the whole industry was going slower. They sort of say,

Dario has said things like that. Demis has said things like that. You know, is there hubris involved? Is there greed involved because they're getting paid money head over fist? Sure. I'm sure that's like distorting their reasons, making it easier to rationalize, but even through all that, they're saying, you know, I feel like I'm stuck in this race and I've got to do it because the next guy's worse.

This retaining wall is going to hold it. And the other guy said, 'I don't know, man. That sounds crazy. We'll figure it out on the fly. Once the retaining wall starts falling, 75% chance it's fine. We would shut that bridge down.' Right? And a bridge doesn't threaten the entirety of the human species. So this situation is ripe for the world to take note and say, 'Hey, we have heard you. We're slowing this down internationally everywhere.' Until then, yeah, they're going to keep racing.