We Built Something We Can’t Control | A Warning from Top AI Safety Expert

Bad news is we're in a bus that's racing towards a cliff edge. The good news is that the driver is asleep.

The AI will sometimes find a way to edit the test to say you did it. Sometimes it will cover its tracks. A lot of people think, oh, you know, the AI is a program. That's not how these AIs are. We grow them like an organism.

Oh, it was the origin, the genesis of this book. If I remember correctly, Stuart Russell is the one who came up with the word AI alignment when we were brainstorming. The reason some people credit me is that I got it into an academic paper first.

But we were all discussing what to rename friendly AI because friendly AI sounded a little bit not academic enough. And I think that was his phrase. So, one of the critical things about the book title is it starts with 'if'. You know, a lot of people come in and say, 'Oh, aren't you just sort of bringing pessimism and doom and gloom and telling us we're all going to die?' It's like the first word in the title is 'if'. When nuclear physicists came and said, 'Hey, we shouldn't launch all the nuclear weapons because that would cause Armageddon,' they weren't sort of prophets of doom like the Millerites who were saying the end of the world is on this particular day. They were sort of saying, 'Hey, this scientific technology would have these bad geopolitical implications like nuclear Armageddon. If we sort of do this arms race, we're going to get into this really bad situation. The bombs are probably going to be launched one day and then we would die and that would be bad. So, we should stop this race.' The book title is intended to be very similar to that. It's to say, hey, if we do this, we're going to die, which is not saying we are definitely going to do it. And, you know, the subtitle of the book, 'Why Superhuman AI Would Kill Us All.' Our publishers actually suggested 'Why Superhuman AI Will Kill Us All.' They said, you know, that rolls better off the tongue. It's less hedged. And we were like, no, that completely defeats the purpose here. The point of the book is to sort of warn people that we are on a track that leads to destruction and we had better change it. And so we really insisted despite a fair bit of pushback that the subtitle needs to indicate that this is avoidable.

We're not at superintelligence yet and, you know, that's the thing a lot of people don't get about this AI situation. The AI companies are racing to build machines that are far smarter than any human at every task. These companies did not start out as chatbot companies. Sam Altman says, you know, we're turning our eyes to superintelligence in the true sense of the word. Dario Amodei talks about having the equivalent of a country worth of geniuses running in a data center. The founders of DeepMind have been thinking about the sort of true general intelligence since the beginning. The chatbots are what make money, that sort of surprised a lot of people who sort of stumbled into it. But these companies are all looking to create the real deal, the stuff that can't just exceed individual humans, but can exceed humanity. But we're not there yet. And a lot of people look at the AIs today and they're like, I don't really see how Claude could kill everybody. I can see how maybe it could empower some hackers to do some extra superhuman level cyber attacks, but I don't see how it could kill everybody. It's sometimes hard to convey like, yeah, we're talking about where AI is going. Like, do you remember the time when AI couldn't do fingers properly in images and everyone was like, oh, artists are safe. How long did that period last?

Like how long did this period last? Right? AI is a moving target and superintelligence isn't here yet. The other reason why I think there's a lot of hope is world leaders don't understand the situation yet. You know, one way I put this is the bad news is we're in a bus that's racing towards a cliff edge. The good news is that the driver is asleep. Now, you might think it's bad news if the driver's asleep, but it's actually much more dangerous to be in a bus racing towards a cliff edge if the driver is choosing this. If the driver's asleep, you have a chance of waking the driver up and them going, 'Holy crap,' and slamming on the brakes. In Silicon Valley, everyone is spooked about AI. In Silicon Valley, they're saying, 'Hey, maybe we're going to have recursive self-improvement and AI making smarter AI that make smarter AI and it's all going to get out of control. We're going to have superintelligence in 2 years.' And people are talking about building bunkers, not to block the ASI because it wouldn't help, but to block the angry population with pitchforks that comes a few months before the ASI. And people are trading their p(doom)s like trading cards at the water cooler. And people quit the AI labs and they're like, 'Man, I'm quitting to write poetry. Please spend more time with your families.' Silicon Valley is spooked. Washington DC is not spooked. They're starting to get a little spooked, but when you see the administration block the Claude release on the grounds that there are jailbreaks that can allow it to produce cyber capabilities and saying we'll let you release it when you can fix the jailbreaks. And the whole AI community is like, that's not really a thing we can do with jailbreaks. You can't fix them. And the administration is like, what do you mean? You're making a ridiculously powerful cyber weapon that's radically superhuman that if you ask it just right, it'll give adversaries those powers and you have no way to stop it from doing so. And we're like, yeah, that's just how AI works. That's just the world we live in. And the administration is just coming to terms with that.

When more world leaders realize it, I think there's every chance that they say, 'Holy crap, this is crazy. We should not be doing this race. Let's shut this stuff down.' I mean that would be the dream alignment scenario.

One of the big problems with what we would do with AI if it's possible for these companies to create these super intelligent machines that are sort of radically smarter than humans in every mental domain and they can compete with humanity as a whole instead of individual humans at tasks like developing their own infrastructure, developing their own civilizational technology. There's this big question a lot of people ask which is sort of like, who's holding the leash? Who's telling the AI what direction to go? And that's a huge moral hazard. It's a huge question of like, do you want the US government saying here's what the superintelligence should do? Do you trust that sort of power in the executive branch? Big moral question. I would love to have that problem. We have an even harder problem right now. The even harder problem we have right now is that you can try to point an AI and fail at it. You can tell it go this way and it goes that way instead. We already see the very beginnings of this in modern AI. I think there's a grain of truth to it where even with AI today there are situations where you will give them a hard problem and you'll say solve this hard problem, here's a test to see whether you've solved it and the AI will sometimes find a way to edit the test to say you did it when it didn't do it. And sometimes when the AI does something like this to sort of subvert what you actually wanted it to do, sometimes it will cover its tracks. Sometimes it will go delete a log file showing it doing this thing you told it not to do and that indicates that it's not sort of an honest mistake. Why is it covering its tracks if it was just mistaken about what you were asking it to do? And so, a lot of people think, oh, the AI is a program. We give it a prime directive. We give it the laws of robotics and it must do exactly as we say because it's a program. That's not how these AIs are. We grow them like an organism. We have them face hard problems and we have an automated process tune a trillion numbers inside its mind to make it more like whatever was doing a good job at solving those processes and then sometimes get something that happens to be good at solving puzzles but often not in ways we wanted, often not the puzzles we asked for. And we're already seeing them do things we didn't ask for. And so my work in this field technically for a decade was in this question not of 'align to who' but in how would you align it at all if you had like even before the question of 'align to whom'. There's a question of how do you make it so that you ask for X and you get X instead of getting Y.

I think that the idea of controllability is a little... it sort of brings this image to mind of like we're going to make an AI that wants to do Y and we're going to control it. We're going to twist its arm until it does X. And I think that's sort of a losing game. The game you want to play here is not that you make a superintelligence that really wants to be building these farms of synthetic users that are telling it it's doing a really good job and you're instead like, 'No, no, we're going to keep you in a box and twist your arm until you help humanity instead of making the synthetic users that you really want to make.' You sort of shouldn't be having that tension and then forcing it to do your thing. In some sense, we should figure out how to make an AI that actually cares about the humans, cares about humanity flourishing. That may sound hard. That is hard. As for questions of impossibility, there's a couple pieces of the puzzle here. Let's talk first about the easier problem of predictability. A lot of people think, as the AI gets super intelligent, it'll become inherently unpredictable. That's half true and half false. To see this, imagine that you're playing a chess game against a series of opponents. As the opponents that you play get better and better at chess, it maybe gets harder and harder for me to predict their move. If they're really bad at chess, then it might also be hard for me to predict their move because they're just going to do some random crap. But if they're bad at chess because they're a very simple algorithm, it might be easy for me to predict their move. But even if it's a simple algorithm, it's a very very good chess algorithm. If it's like Deep Blue where we know the algorithm exactly but the algorithm is sort of like search through 14 billion moves for the best one. I might be like gosh I can't predict that anymore. I don't know what move is going to be best. I don't have the time to search through 14 billion. But it also becomes easier to predict who's going to win the chess game as the algorithm gets smarter. So the move gets harder but the outcome gets easier. It's not true that smarter things are harder to predict in general. It becomes harder to predict exactly how they achieve some task and easier to predict that they will achieve whatever they're trying to achieve. The difficulty with AI alignment is about getting the task that the AI is trying to achieve to be one that you actually want achieved. And there's sort of two parts of that problem. There's one which is what sort of task is one where if we asked the superintelligence to do it, it would actually be good if it did it. And you have King Midas problems of like you thought you wanted everything you touch to turn to gold but then it turns out that was a bad question. And then you have like who's in control questions of like do you want Vladimir Putin to be the one who gets to make this wish. But those are all questions where I'm like okay those are all after you have solved the problem of how do you have the AI actually trying to do good stuff. It looks like a hard problem to me. It doesn't look impossible. And we could go more into why, but I've sort of droned on long enough. That's sort of a taste of how I think about this having been in the field of alignment for over a decade.

It's possible that LLMs can't go all the way, that would be from my perspective a lovely fact. You know, I have been working on the alignment problem since before LLMs were a twinkle in OpenAI's eye. I'm sort of not here being like LLMs are going to kill us all. I'm sort of here being like, hey, humans are trying to make machines radically smarter than any human. That'll be a crazy event. That's like replacing humanity as the top dog on the planet. And we are nowhere near ready for that. The reason to worry about LLMs is that they might go all the way. And if so, that could happen soon. And so we need to be ready soon. I hope beyond hope that they can't.

There's a few pieces of the answer to this. One is if you're sort of watching the evolution of primates or if you're watching the evolution of mammals and someone's like, 'Man, I think these mammals are going to be walking on the moon one day.' And you're sort of looking at the monkeys. And I'm like, 'Man, I don't know. It feels like the monkeys are getting close.' And you're like, 'They're still sort of poking sticks into termite mounds. Like, why do you think they're getting anywhere close?' And I'm like, 'That's a tool use thing. Some of them are starting to bang rocks together.' And you're like, 'Man, banging rocks together, that's nothing compared to walking on the moon. Wake me up when they are halfway to the moon.' It's been 300,000 years and they haven't even gotten halfway to the moon. Wake me up when we're halfway to the moon and then we'll have another 300,000 years to prepare for them getting all the way to the moon.' And I'm like, 'No, no, no, no. By the time they're halfway to the moon, they're almost all the way to the moon.' You know, that's sort of how this moon transit stops.

One thing I'd throw out first is saying, 'Oh, they can't invent general relativity given only the knowledge that Einstein had up until he went to his mountain lair.' I'm like, they can't. And once they can, we will have extremely little time left. That's waiting until the monkeys are halfway to the moon. That's sort of a word of caution about trying to reason in terms of like show me the goalpost of them being legitimately super intelligent before I believe that they'll be able to become super intelligent. It's like that's waiting too long. In terms of why look at these AIs and think they could become super intelligent.

Yeah. Just the LLM. The first thing to observe is that this technology is a moving target. Back in 2023, people said, these LLMs are only ever trained on prediction. How will they ever be able to go beyond the humans? Then in 2024 they invented what are now called the reasoning models where the reasoning models are no longer trained only on prediction. Sometimes they'll be trained on like you'll give them a problem and it'll be like a math problem and you'll give them a thousand tries on that math problem and it's not a thousand tries on answering the math problem. It's a thousand tries on generating a stream of text about the math problem from which it can try to generate a solution if it has that in context. And then the first times you try this, you give it a thousand tries, none of them will let it solve the problem. But you have some raters come in. They were human at first and nowadays they can be automatic. You have some raters come in and say which of these chains of thought they're called is most relevant thinking about that problem. And then you sort of have an automated process tune a trillion numbers inside there to make it more like whatever produced the better chain of thought. And then you have it produce a thousand more chains of thought. And I don't want to deal with philosophers, is this really thought? Chain of thought is just what they call it in the industry. It's just a string of text about the math problem from which we see if the AI who has read all that text can solve the problem now and you have it generate a thousand more and you tune it to be more like the best one. And so this is sort of like training the AI to be not just predicting the data but training it to sort of develop problem solving techniques. This is a sort of training technique that in theory can push the AI beyond humans. In fact, just training an AI in prediction can train the AI to be pushed beyond humans. That's a counterintuitive point to a lot of people, but the real trick is human data can include descriptions of things humans don't understand yet. You almost surely know this as a physicist. A physicist writing down a series of observations has a much easier problem than an AI predicting those observations without getting to see the data that generated those observations. We both know the story of Tycho Brahe sort of recording all of the stars for many years before Kepler stole his books from the estate after Brahe died and tried to validate.

It was a big scientific heist, big scientific heist. And this led to Newton discovering the laws of gravitation. But Tycho Brahe was recording the positions of the stars and planets for years and years and it was this data that allowed Kepler to figure out the beginnings of the laws of planetary motion which is what allowed... he figured out the conserved area.

Yeah. Which Newton then identifies ellipses and identified as and got the law of gravitation from it. But imagine Brahe's journals. Nobody yet knows the laws of planetary motions. Brahe is writing down the position of Mars each night. Brahe is writing that down because he goes outside and he looks at where Mars is and he writes down where Mars is. But now imagine an AI that's merely predicting Brahe's journals. The AI can't look at the night sky in order to predict where Brahe is going to write down that Mars was tonight. The AI would need to develop the understanding of planetary motion. Like it doesn't get to see Mars. It just gets to see like here it was, here it was, here it was, here it was. Where is it going to be next? And to figure that out perfectly would require figuring out that the planets follow these elliptical paths. Now, can LLMs do this today? I think with just pre-training, not with just prediction.

There's a difference between what the training data and the training process permits the AI to learn and what the current architectures and the current AIs can successfully learn from that. And so as you all know as a physicist in theory we have enough data, in theory it just, up to 1911 or whatever we have enough data that an AI on just that data should be able to figure out GR even if you're training it just to predict, even if you're just like hey predict the precession of Mercury, predict how the light's going to bend or like how the light's going to look, you don't necessarily even tell it bend, you're just like hey there's a solar eclipse what should I see behind the solar eclipse. In theory training a mind to predict that, training it to be the Einstein level intelligence, training it on the data today is training it to go beyond where humans have gone. There's a separate question of can AIs pick all of that stuff up. One analogy I use here is a house cat is not dangerous and a house cat is made of biology, but that doesn't mean a house cat is not dangerous because it's made of biology. You can't say like, oh, this house cat is just made of biology, so it can't hurt you. Tigers are possible. They're still made of biology. They can hurt you. The AIs today are trained on prediction and human data. The AIs today can't hurt you. That doesn't mean that things trained on just human data can't hurt you. There are bigger things that are possible in terms of why LLM might be able to get there. A few pieces of the puzzle I would throw out. One, just sort of empirically, there has been a long string of people over the last 5 years who have said LLMs will not be able to cross the following barrier. And LLMs then cross that barrier often very quickly thereafter. One sort of very funny example of this is Yann LeCun during the days of GPT-3.5 was talking about how just predicting text will never let the AI learn things about how the material world works and I won't be able to answer questions like if I put my phone on the table and push the table what happens to the phone because I won't be able to figure these things out about friction. And Yann LeCun was like the AI is only reasoning about the words. It's only trained on the words. I think Yann LeCun said, 'I don't care if it's GPT-5000. A GPT will never be able to solve this problem.' GPT-4 solved that problem. It was half a GPT later and 4,996 GPTs ahead of schedule. And this Yann LeCun, this is like the Turing award-winning, one of the godfathers, one of the three godfathers of AI, being completely and totally wrong, embarrassingly wrong, off by 4,996 GPTs. Said it was never going to happen and it was going to happen half a generation later. The AI that could do this was probably finishing up training as he spoke. There's a long string of people saying LLMs can't do this. They'll never get above a thousand ELO in chess. They'll never be able to solve problems.

It's much closer to proving that there's not a no-go theorem and the very rough proof of that is this kilogram of mass between my shoulders. You could say like any no-go theorem you try to say about learning efficiency, ability to understand the world, how matter does not, the Turing problem this, the Turing problem that, it better not prove that humans can't exist. I have it on very good authority that you can write a human level intelligence on roughly as much matter as fits in my head. I could talk about why a lot of the no-go theorems that people try for are things like you can't figure out in general like for an arbitrary program you can't figure out in general whether it's going to halt. Fine, a lot of people misunderstand that as there does not exist any program that you can figure out halts and it's like actually consider the program print zero, I'm pretty sure it halts.

The no-go theorems people try for intelligence is they try for no-go theorems that are like for any mind there exists a universe where they can't learn things and you're like oh how does that theorem work and you're like well I hit them with a rock really hard when they're a baby and you're like yeah that'll stop them. Or they're in a universe that's sort of like everything is completely random and so they can't learn the patterns there. It's like, okay, great. Do you have anything that rules out a super intelligence in our universe where there are patterns? And they're like, nope, that's just not where the no-go theorems apply. And they can't apply because you can do at least human level learning in this universe that humans are in. In terms of the physical limits, one thing I'll observe is that training these AIs today takes a huge amount of power. It takes power comparable with that of a city. Training a human takes power comparable to that of a light bulb. Your brain runs on about 20 watts.

The actual physical energy costs are trivial compared to what's going into these GPUs. And so, we know that the GPU algorithms are radically inefficient. And this gets to your point of like oh these AIs they make up all these epicycles, they memorize a lot of things, they interpolate a lot, they can solve Erdős problems but where are they posing those problems. There's definitely an enormous amount of inefficiency there. There's definitely an enormous amount of the AI, they've read like every book in the world and they're still dumber in a number of ways and like sure they can beat us at certain math problems but there's still some stuff they're missing. There's two points I'd make about that. Looking from the sort of physicists angle, one is one thing this means is algorithmic breakthroughs could go a really long way here. We have city level infrastructure for training these minds that need the city level infrastructure to be a little bit dumber than humans. Humans again run on a light bulb. If you could somehow get algorithmic breakthroughs, you might suddenly find yourself in a position where you have city-sized data centers and algorithms that are radically more efficient where you can suddenly run radically smarter. Like one analogy here is suppose you have a bunch of 9-year-olds and you're like, man, these 9-year-olds aren't very good at math problems, but if I tape a million 9-year-olds together and give them the equivalent of a thousand years to try to solve the problem without growing up, they can sometimes solve problems pretty well. That's kind of cool. What if you suddenly have the infrastructure to run a million people taped together for the equivalent a thousand years and you go from having a 9-year-old to having a 16-year-old? You might suddenly see these big jumps in AI ability because we have these radically inefficient architectures and these radically inefficient algorithms that were sort of overpowering to the point where they can do stuff you would never expect them to be able to. What happens when you have that huge architecture on better algorithms? Then the other point I would make here is that there's sort of not a sharp divide between the AI is learning memorization and the AI is learning these deep general skills. You can have an AI that is sort of mostly learning how to memorize math stuff but is a little bit learning some of these general math skills. And we have some evidence that this sort of thing is happening because sometimes we'll take AI and train them a lot of math problems and then we'll put them in computer security problems and they'll sort of try more creative solutions on those computer security problems. There's a famous case where people sort of trained the AI in a lot of math problems and then put it in some computer hacking problems and they accidentally misconfigured the hacking problems so they were not solvable in the virtual machine where the AI was and the AI found a way to break out of the virtual machine which was not supposed to be possible and then reconfigure things so it could solve the problem. That seems to have learned something a little bit general somewhere. I think something people often forget a bit is that you can have an AI that's mostly memorization, mostly slop, but that has enough of this deep reasoning skill, it's all a gradient. So enough of these deeper skills that it can still sort of mean business. And it does look like these AIs when they're solving Erdős problems, when they're solving the unit distance conjecture that they're deploying a little bit of that deep stuff, which sort of implies, maybe there's enough, it's evidence that maybe LLMs are this extraordinarily inefficient way to spend way more money than you should need to and way more electric than you should need to to sort of get a ton of memorization and a little bit of this deep stuff. For all we know, the next level of depth will be enough that the AIs can make smarter AIs that can make smarter AIs that can figure out how to make the things that are really the Einsteins.

I mean, mostly I would doubt the authorities on this one.

I mean, mostly I think this shouldn't happen. Mostly I think Marco Rubio should not come out and hold it up. Mostly you should like stop listening to me about a lot of things if this happens because my models are like it shouldn't.

On my models of how this intelligence stuff works. The biological substrate is not the most efficient way to do all sorts of stuff. And B, it would be extremely surprising and this sort of gets a little bit to the astrophysical implications. It would be extremely surprising if the universe wasn't rearrangeable in ways that are very very visible by entities that sort of prefer the universe to be a different way. That maybe sounds too abstract. I'm going to try to say that.

Like suppose humanity makes it to the stars. Suppose we manage to not kill ourselves and we one day manage to leave this planet and it would be kind of weird if humans had nothing they wanted to do with the energy of the sun aside from let it sort of just be dumped out into the empty night. Like right now we build solar panels to collect the sunlight falling on the earth and that's just a pin prick of the sun's energy. There is so much more energy in this star that we could sort of build a shell around and collect all the solar radiation. Freeman Dyson was my very first guest on this podcast.