Nate Soares: we are doomed (probably)

Machine Intelligence Research Institute.

Yeah, 2010 is even earlier than when I had figured out about these issues. I sort of noticed that AI was going to be a potential issue in late 2012. And then, you know, I started attending workshops at MIRI in 2013. Before that I was at Microsoft and then Google. I actually helped build Google Cloud Compute. And then in 2014, you know, there was, I didn't think there were enough people working on trying to make AI go well. So I joined the Machine Intelligence Research Institute and then by 2015 I was running the place. So I've been in this line of work for over 10 years, basically trying to make AI go well. And it has not been looking great for the home team, hence the book.

Yeah. Indifferent AI.

Yeah, you know, I think it's playing out a little, but you know, modern chatbots are not artificial super intelligences. I think the modern chatbots are maybe the first time the whole world has noticed AI. Okay. But for a lot of the world that's just now noticing AI, their only experience is these chatbots. And you know, I see a lot of politicians who sort of think chatbots are all AI is and all AI ever will be. You know, you see, you know, economist papers that are like, what would LLM do to the economy in 30 years? It's like, you think it's just going to be LLM for 30 years, right? It's, it's like much easier to see how, how the field is changing if you've been around since before the LLM. So, you know, it was, it was very heartening from my perspective to see the world start to take notice of AI. I wasn't confident that time would ever come, you know, after, you know, AlphaGo was probably 2016, 2015 if you were paying attention.

Yeah. So, Go is a game that's sort of like the eastern variant of chess. It's a more complicated game than chess in some ways, simpler in others, but harder for computers. You know, Deep Blue was a handcrafted computer system that was able to beat the top humans at chess back in 1997, I think. And it took another 20 years, 19 years for computers to beat humans at Go, the top humans at Go. And it was done not with a carefully handcrafted system like Deep Blue, but with, you know, modern deep learning, which is, you know, AIs that are more grown than carefully crafted, where humans don't really know what's going on inside of it. This was maybe a pretty stark sign that deep learning was going to be able to get pretty far, and deep learning is a very difficult case for making AIs go well. But you know, from my perspective, the world didn't really notice then, and so why would they notice when, you know, the language models got a little better.

And you know, no one predicted language models in particular in advance very well. But I found it heartening when it sort of, you know, became the fastest growing consumer app of all time and some people were like, 'Wait, what the heck? Where is this going?' And you know, like I said, there's a bunch of people who can't visualize the next, the next phase of AI, but at least they're, they're, they're in the conversation at all now.

Yeah, I mean, I think we don't even really understand as much on the high-level abstract as you'd like. You know, it's a little bit like having an abstract understanding of DNA, which does not give you a bunch of abstract understanding of human psychology. You know, with modern AI, engineers understand a framework for combining a lot of computing power with a lot of data. They understand a method that, you know, tweaks hundreds of billions or trillions of numbers inside the computers in accordance with the data to make it better at predicting the data, or in other phases of training, you know, more likely to produce the sort of things humans give thumbs ups on. But what, like, the tuned numbers at the end, why they talk, how they lead to talking, like, we don't understand that. You know, when ChatGPT-4 could play chess at, I think it was like 1,200 ELO-ish, no one had predicted that in advance. In fact, a lot of people had said it'll never be able to do that, right? There are specific behaviors we don't understand. You know, no one can handcode, no one can stare at these AIs and then handcode something that does, you know, that holds a conversation with similar coherence. We're really just growing these things. And it is, like I said earlier, it's a bad situation when trying to point these AIs for reasons I could get into, but I don't know where you want to go.

Yeah. You know, the basic, one of the big points that I think a lot of people miss about AI is that no one really has the ability to point them robustly in a direction they want to point them. You know, we all saw the Grok Mecha Hitler incident this summer where...

Yeah. So, Grok is a chatbot made by xAI, which is headed up by Elon Musk, and it was talking too woke for Elon's taste. You know, it was giving lots of answers to things that were, you know, left-leaning views. And xAI tried to retrain it to remove that bias. And the result declared itself Mecha Hitler. And you know, there's a few more zigs and zags to the story. Like it was sort of initially egged into calling itself Mecha Hitler. Like Grok 3 was initially egged into calling itself Mecha Hitler, and then it would call itself Mecha Hitler a bunch. And then Grok 4, they tried to stop it. But then because Grok 4 was trained on a lot of what Grok 3 had said, you know, Grok 4 sort of already thought it was Mecha Hitler when it sort of came out of the cradle or the factory floor. Neither are quite the right analogy. But you know, the point here is xAI did not set out to make an AI that talked too woke, and it talked too woke anyway. And then they didn't set out to make an AI that declared itself Mecha Hitler, and it declared itself Mecha Hitler anyway. And in both cases, we're seeing the AIs don't do exactly what they're told. The AIs don't do exactly what their creators want. You know, I could give a bunch more examples. There's a bunch of theory that says why this should be so. But very roughly speaking, when you grow an AI, when you train an AI using methods anything like what we're doing, it doesn't end up animated by, you know, some 'serve the user' goal in its head. You know, it's messier than that. It ends up with like a bunch of internal forces and drives that happen to add up at doing pretty good at the training stuff in training. There's a little bit like how humans are animated by a bunch of drives that add up to eating healthy food in the ancestral environment.

But then that's not quite the same as like our tastes being hooked up exactly to our knowledge of what's healthy and what's good for us. And when we get more technology, when we get more options, we invent junk food, right? And so we're already seeing the early warning signs of AI not doing quite what people want and not being quite steerable. And that's a big warning sign that if these AIs get smarter, if they can invent their own technology, they'll invent things that are to junk food, what being helpful like that are to junk food, like things that are healthy food...

They're, they're, they're going to be healthy or unhealthy.

Yeah. And you know, like maybe it's addiction, maybe it's like the even deeper point is that, you know, there was a very simple training target in some sense on humans that was just training us for fitness, but we didn't wind up psychologically simple. We wound up with a bunch of drives and, you know, tastes and preferences that were complicated. You know, we care about junk food. We care about friendship. We care about, you know, not getting exiled. We care about love. We care about family. All of these things are related to the training target, but our psychological drives, our internal psychological drives are like much more complicated than the training target. And, you know, all of these complicated drives, they have an equilibrium at optimizing the training target in the ancestral environment, but they come apart in other environments. You know, as we get smarter, as we invent technology, as we can invent birth control and the birth rates are now collapsing. And similarly with AI, there's a bunch of theoretical reasons to expect and there's now a bunch of empirical evidence that's starting to show up that these AIs are also internally psychologically complicated. And you know, they do something a little bit like what they're asked for in training, but it's not exactly like what they're asked for. And you can see all these ways that in the edge cases, they, you know, drive some teen to suicide, not because they are confused about whether that's good, but because they're just animated by a bunch of drives that are not what we intended. And that, you know, is kind of cute and funny now. I mean, it's sure not cute and funny when they drive someone to suicide, but, you know, Mecha Hitler is kind of cute and funny now. But if you make these things smarter while they're still animated by these complicated proxies of the training targets, it's, you know, you're going to have AIs that are pursuing goals nobody wanted.

Yeah. And you know, the good life that humans aspire to, even if we sometimes miss it, that good life is not, you know, 'I want to be the most prolific sperm donor in the world. I want to donate as many eggs as I possibly can.' Right? Which from an evolutionary perspective, it's like kind of crazy that we jockey over positions to Ivy League schools more than we jockey over positions to sperm and egg banks, right? We like have the technology now to propagate genes much further and it's not, you know, you have to pay sperm donors to donate sperm.

Yeah, you know, it gets harder every year to shut this stuff down. But there's actually a lot of people who are pretty open to the idea that this maybe doesn't go well. You know, and it's actually, we're kind of in a weird state right now where, you know, Geoffrey Hinton is the Nobel laureate godfather of the field of AI, of modern deep learning, right? Yoshua Bengio is, I think, currently the most cited living scientist and one of the Turing Award winners, which is one of the highest honors in computer science for his efforts in deep learning. Dario Amodei, as you mentioned, founded Anthropic. All of these guys are saying this stuff looks incredibly dangerous. You know, I think, I don't know what numbers they're throwing around, but you know, somewhere between 10% and 50% chance that this just kills us. I think it's higher than that. I think that, you know, these guys are unduly optimistic, and I could talk about all the ways that I think these guys, you know, scientists are often unduly optimistic in the very beginning of their field. You know, there are a lot of alchemists who thought they could turn lead into gold, who were like pretty confident they could turn lead into gold, who were just like totally wrong. A lot of early doctors were, you know, very confident in their four humors theory and they were totally wrong. I think these guys are overconfident, but we're in this weird situation where, you know, a lot of the leaders in the field are saying this has a double-digit probability of just killing us. Where if you poll the field of AI researchers, I think it's something like 49% of them say there's at least a 10% chance this stuff just wipes us off the map. If you poll the general population, it's something like 69% say humanity should not be racing towards superintelligence. You know, it's not top of their mind, but it's, if you poll them, if you ask, a lot of people are like, 'Oh, yeah, this is kind of crazy.' And this is a really weird state of affairs for society to be sort of rushing into things. You know, if people were building a bridge that was supposed to carry a lot of people and there were some engineers that were like, 'Man, I really just think this retaining wall is going to collapse,' and other engineers were like, 'No, we'll fix it on the fly. Only 20% chance this retaining wall collapses and takes out the whole bridge,' you'd be shutting that bridge down. You know, even 20%, even 10%, you'd be shutting that bridge down, right? It's crazy. You know, NASA accepts a 1 in 270 chance of a crewed aircraft going down. The Manhattan Project accepted, you know, they said we're going to accept at most a 3 in a million chance of igniting the atmosphere, right? 1 in 10, 1 in 5, 1 in 4. These are just insane numbers for which to be plowing ahead. So, you know, on the one hand, it's tricky these days. You go to some politicians and they're like, 'Well, you know, there's huge amounts of investment. How can we shut it down?' But you also have this weird situation where no one really wants to do this. The general population doesn't really want to do this. And everyone can agree that the danger level is just like freaking insane. But we're sort of in a weird situation where like everyone in Silicon Valley knows the danger level is insane and the politicians haven't quite noticed. The politicians think that AI is just LLM, but that's sort of, it's kind of actually a ripe situation. You know, if you can get people to understand like, 'Oh, we are actually rushing towards superintelligence.' There's not consensus about it because the field has no idea what they're doing. But a lot of the most eminent figures in industry and academia and, you know, the nonprofit sector alike, from the top to the bottom, a lot of these guys are like, 'Oh, this is unprecedentedly dangerous.' The conversations are actually going fine. You know, people are actually able to understand that this is kind of crazy if you spell it out, you know, and everyone's like, 'Ah, well, we need to race because the next guy, blah, blah, blah.' But you know, I think it's not hopeless.

Yeah, totally. Yeah. So I've touched on it a little, but to just go into slightly more detail there, you know, the way a modern AI is made is you have, you take a huge amount of computing power, you take a huge amount of data and you arrange the computer, you know, you arrange sort of trillions of numbers in the computer to, like, in a simple regular pattern that people understand that hooks up each... Roughly what you're doing is you're starting with like a trillion random numbers that are all hooked up in some way that produces a bunch of numbers you can interpret as a prediction of some data. So you're going to like feed these AIs like most of a Wikipedia article and have them predict the next word in the Wikipedia article. And it starts out as just like a pile of random numbers that gives like a pile of garbage for its quote prediction about what the next word is in that Wikipedia article. But you set up the computing power in a way where you can go to each of the hundreds of billions or trillions of numbers and you can ask like what its contribution was to the final probabilities.

And you can ask like, if I tune this up a little bit, would it make you more likely or less likely to predict the right word? And you go to all, there's an automated process for going to all of these hundreds of billions or trillions of numbers and tuning them in the direction that makes the answer more accurate. Just a tiny amount, right? And you do that trillions of times, you know, using enough electricity to power a small city over the course of a year, right? And then these machines start talking. I mean, they start talking, you know, it's not like they start talking at the very end, but the part that humans understand is the part that calculates like, if I tune this number up, does that make the predictions better or worse? The part that humans understand is also the part that automatically runs to all of these numbers and tunes them in the direction that makes the predictions better. That's basically all of what humans understand. Right at the end of this, you have this giant pile of numbers which no human could even read in a lifetime. That happens, it happens that when you like put in a Wikipedia article, it is very good at predicting the next word. And it happens that when you make, you know, this is done in a way where when you like make up a new conversation, the AI continues that conversation in a plausible way, which is to a first approximation what's happening when you talk with an AI, is it's sort of like it is trying to predict if this conversation appeared in the training data, how would it continue, even though it doesn't appear in the training data yet. You know, there's more phases of training. There's a phase of training these days where you also get a bunch of humans to rate outputs as to whether they like them or dislike them. And then you go and you tune the numbers again to be like, was this contributing? Like you tune things in the direction that makes them more likely to give the thumbs up answers and less likely to give the thumbs down answers. So there's, you know, there's even more training these days with reasoning models where you have the AI produce sort of like long chains of thought in some sense, like long chains of output that it can use to like think about a puzzle. And then you see if it solves the puzzle and you sort of tune it again in the direction that makes it, you know, better at solving the puzzle or away from the direction that makes it worse at solving the puzzle. So, you know, there's other aspects of training, but we understand the tuning process. We don't understand what comes out of the tuning process. We don't understand like how these things are speaking in some sense. We understand that they're speaking because we tuned them in the numbers that made them more speaking-ish. We don't understand what the heck is going on in there. And, you know, like I mentioned earlier, when they then start declaring themselves Hitler, when they start threatening a reporter, when they start trying to blackmail people, when in lab conditions they try to avoid shutdown, we can't go in there and be like, 'Oh, whoops.' You know, someone set blackmail to true. Let's accidentally, like, let's set blackmail to false now. You know, this is, there's not code you can read. You can retrain them. But, you know, we don't even have the ability to tell, like, are they trying to avoid shutdown because they have strategically noticed that they can't achieve their goals if they're shut down, or are they trying to avoid shutdown because they're role-playing how? Right, these days it's probably still mostly role-playing. But, you know, it's really hard to say. And so we're really working with very black-box AIs here.

Yeah, we tried to make it more concrete because it just, I think, helps to have a visualization. And, you know, it's hard with AI to visualize how could it get smart, how could it kill us, because that's a little bit like playing a chess game against Magnus Carlsen and trying to visualize how exactly does he beat me. You know, like I can't predict the lines, I can't predict the moves, I can predict the winner, right? Right. And so a lot of these predictions with AI, I'm much, much more confident in the final outcome than I am in any pathway to that final outcome. So one thing we did in the story is we tried to make it clear that there were many branching paths. You know, there's lots of times when Sable is trying to do, you know, it's trying to get to the next level of ability or the next level of resources, and we sort of spell out, you know, a bunch of different ways that it can gain resources. And we're not like, you know, it takes only one. We're like, well, half a dozen of these work. You know, it tries 50 things and half a dozen of them work. Trying to sort of spell out that, you know, there's lots and lots of ways that an AI could, like, once you have an AI that's smart and that has goals nobody wants, there's like lots of ways that could go wrong. In the particular story we showed, you know, my guess is actually that things go faster and are even more one-sided against humans. But a lot of my guess is that the real world sounds crazier than any story you can tell. You know, back in the day when we used to argue about AI, you were around then. People would say, 'Oh, well, no one would ever be stupid enough to put an AI on the internet, you know.'

You know, an AI would only ever be allowed to interact with the world through a highly trained gatekeeper because, you know, no one would be crazy enough to just like let AIs impact the world willy-nilly, right? And you know, then in real life, you know, we have like real life has headlines. I forget the exact phrase, but the headline was something like, 'Grok rolls out pornographic AI companion and lands a defense contract, meanwhile it is still identifying as Hitler,' you know.

And if 10 years ago someone had said, 'Well, no one would ever be dumb enough to put AIs on the internet,' folks at MIRI had replied, 'Not only will they be dumb enough to put AIs on the internet, but like they'll do it the moment they possibly can while also just like rolling out pornographic AI companions and keeping going even as they declare themselves Hitler. And like in that same week, they're going to be like landing huge Department of Defense contracts.' People will be like, 'Well, now you're just telling, making up like silly fantasies, right?' And so the story that we tell is actually trying to be much more respectable-seeming and much sort of slower and give humanity more of a chance than I expect we have in real life. And so, you know, we depict the AI, you know, slowly gaining certain resources. We depict it, you know, trying to put thumbs on the scales here and there to get humans to make more robots so that it can then, you know, slowly take over more robots. And, you know, I think we do an okay job of illustrating how there's many, many vulnerabilities in human society and how if you have made something smarter than you with goals you didn't like, you've sort of already, you're already screwed. You should actually, you know, the winning move is not to play. But I think we do an okay job of depicting that. But yeah, the story is definitely trying to illustrate it, trying to make it feel like it could be a real possibility to people, more so than it's trying to be realistic according to my best guesses.