You can start to see, like, this world where you have AIs that are out there. They're like, well, we'd prefer more resources when the humans try to shut us down, that's annoying and so on. And that's not all the way to the end, but that's sort of like, you can see how we get from where we are today to 'I would sort of prefer more resources' that are sort of trying to be out there and have their own thing that they're doing, which is not what they're supposed to be doing. And then there's this big question of how do those AIs wind up self-sufficient? How do they wind up running an automated economy where they can, you know, manage the robots or other constructs that build more computer chips and that run whatever energy infrastructure they need? And I can tell you lots of stories about how the AIs are smart. They could wind up in control of their own infrastructure.
Yeah. Why don't you go ahead and tell us? Tell us the scariest... the sounds are resigned to. It's, like, just terrifying. So it's fine.
Hello, and welcome to another episode of the Odd Lots podcast. I'm Joe Weisenthal.
And I'm Tracy Alloway. I'm very tired today because I flew in on the redeye last night from San Francisco. I was out at this very interesting conference called Curve, and, I've heard about it in Berkeley. A lot of, like, you know, I guess a lot of people very concerned about AI, so to speak. Interesting mix of folks, but yes, people are concerned about what the future is going to look like as we develop stronger and stronger AI models just from a sociological or anthropological standpoint.
I think living in San Francisco and mingling with, you know, AI bros at the moment must be one of the most interesting things, Tracy, because there must be so much cognitive dissonance in their heads, right? Where on the one hand they'll say like, oh, they're building this new ultra-powerful model. And on the other hand, sort of almost as an afterthought, they'll be like, oh, yeah, and I'm a little bit worried about the possibility that it kills us all. How do you square that? It's super interesting.
And you hear it like, so at this conference, you know, people like working for, like, various, like, safety-type orgs, people who want to regulate AI in some way. And of course, like, people from the labs themselves, some maybe like working on like policy stuff, but some actually like the actual builders of the technology and like, you know, they're all like deeply anxious about the very thing that they're building.
Yeah, but they just keep building it, you know?
And so they have all these worries about, at a minimum, how disruptive it could be or how dangerous it could be. And of course, we all now know that there are dangers, particularly in the wake of the Hugging Face attack. So people can see that it's very possible for models to go rogue and commit crimes and stuff like that. And yet they, you know, they keep building it. And yet, and, you know, they would say, oh, well, like, if, you know, someone was going to build it, so why not me? Because if I didn't do it... Well, maybe it is like how they think about it, or there's sort of some game theory aspect where it's like, it doesn't make sense for me to like slow down or pause what I'm doing because someone, either my direct competitor in town or maybe someone in China, will build it. So like, you know, not building it isn't going to do anything. But yeah, it's very strange.
It's very... we've used the term runaway train before to describe how all of this feels, which, yeah, it almost feels at this point like there's no stopping it. Right. Because if one lab doesn't do it, then another lab is going to do it, and that's how they justify it. If one country doesn't develop the next big model, then they're worried about another country building the next big model. So you have this, I guess, collective action problem for humanity. And a lot of it feels inevitable. And I have to say, today's guest predicted this inevitability problem in a very salient way. And one thing I'll just say is I've read the book over the weekend. For those who don't know, the book talks about AI killing us all and talks about one hypothetical scenario in which an agent could go rogue and kill us all. Over the weekend I woke up on Sunday to the headlines about an escaped plague from a Russian lab. And I thought, wow, I couldn't have chosen a better time to read this book. Now, to be fair, the escaped plague was in Siberia, and so if there is a rogue AI... proving it, by the way. Well, supposedly, according to reports. So if an escaped AI agent was trying to kill us all, or kill a select few individuals through a lab or biotech... Yeah, it might not be in Siberia. That's the one thing. It's probably not the one thing that made me feel a little better this weekend. But, you know, the fact that we're thinking about these scenarios at all, the fact that when I see a headline about a lab leak, I think, oh God, I hope it's not a rogue AI. Yeah, because I just read this book. It's weird times.
It is a really weird time. You know? So obviously something I've been thinking about is like the pace of AI development itself is really extraordinary. And then the pace of what people are worried about, it's like the conversation is changing constantly. So, for example, you know, six months ago, if you're like, what are people concerned about? Maybe a year ago, it's like, what are people concerned about with AI? It's like job loss or something like that. And then that's where... wipe out. Yeah. And then the conversation, like, then there was like people were talking about data centers and water use and electricity and electricity prices and so forth. More recently, like the attention again post-Hugging Face, the resignation of the researcher Jacob Coxon from Anthropic has now moved into like this sort of next stage, which people, as you mentioned, some people have been talking about for a very long time. There was actually, I think just today or last night, a big New York Magazine cover story about people who have quit the AI labs because they're like, the things that our bosses are doing is reckless and so forth. So now the new thing in the air is like sort of the ultimate risk. Hopefully that is the ultimate risk. It's hard to see how it could get worse than wiping out all of humanity. Like what would be the next worry after that? Very excited to say we really do have the perfect guest to sort of articulate some of these concerns. We're going to be speaking with Nate Soares. He is the co-author of the book 'If Anyone Builds It, Everyone Dies.' And he is the president of the Machine Intelligence Research Institute. So, Nate, thank you so much for coming on the podcast.
Let's start in the beginning. Like, when did you sort of start thinking about this possibility? Two things: like that strong AI... let's set aside the question of superintelligence for a second. That strong AI is something that could plausibly have been built, and that if it were to be built, it would be something that would be very difficult to control.
Yeah, I started thinking about this in about 2012, maybe 2011. I was working at Google and they acquired DeepMind. Right around then, it wasn't Google DeepMind at the time they acquired it, but it became Google DeepMind shortly thereafter. Yeah. I don't think that was actually the inciting incident, although that sort of caused me to think it could happen, you know, in more like decades than centuries. Although, I hoped we had more decades than it turns out we had.
What was it that you saw exactly? That was the sort of light bulb moment where you thought, hey, this risk could be real.
You know, it's actually abstract arguments for me that did the trick. I know those don't do the trick for a lot of people. Yeah. But, you know, if you sort of look all around us, everything here was shaped by humans. And that's because we're the smartest creatures on the planet. And if humanity makes machines that are smarter than humans, the sort of obvious outcome is that the world gets shaped by those machines. And so suddenly it matters quite a lot how those machines are shaping the world. And, you know, back in 2012, there was almost nobody working on the problem of... you know, there are a lot of people working on the problem of like, how do you make the machines smarter? There are very few people working on the problem of like, how do you cause the machines to shape the world in a good way. Which would go on to become known as the alignment problem, although this was even before the term alignment was coined. I initially thought I wouldn't be able to help much and just sort of, you know, Google did a donation matching thing. So I just tried to sort of like find the people who are trying this and give them some money and get the Google donation matching. And then one thing led to another, and I was running the Machine Intelligence Research Institute.
And now that you mention the alignment problem. So this is something that has obviously come up a number of times on the podcast and elsewhere. But when does alignment as a term and as a conceptual goal, when did that actually become a thing?
So as a conceptual goal, I mean, it sort of depends how far back in the history books you want to go. You know, you could sort of construe Alan Turing and some of the first computer scientists as sort of noticing that there was going to be an issue here. But the sort of modern notion of AI alignment of like, oh, look, we're really building an AI and like, it's not going to be good for free. So we need to put the work in there. That probably came from my co-author, Eliezer Yudkowsky, I think somewhere between 2001 and 2003, again, depending exactly where you want to draw that line. But the term alignment didn't come in till I think it was 2014 when we had been calling it friendly AI research. And Stuart Russell of UC Berkeley, if I remember correctly, was the one who was like, we need a more academic-sounding name. And I think that from my perspective, the word alignment was his idea. But I'm sure he would say, you know, that he dug it out of some ancient paper somewhere else and blame it on somebody else. I kind of like friendly AI. I just aim for friendly. I sort of think that friendly AI strikes me as more coherent as a concept in some sense than aligned AI.
AI is a technology that humans build, and there are different, you know, techniques for building it. How would you describe the core intuition that a thing that we build, we would not necessarily be able to control?
There's a few pieces of that puzzle. There's a number of difficulties in controlling an AI, even though you made it. With the way that current AIs are created, it's sort of very easy to see how we would not have a ton of fine-grained control over their behavior. And that's because AIs as they are made these days are not programmed like traditional software. So, you know, a lot of people think that the way software works is that, you know, you write a series of instructions, then the computer executes those instructions. And if it's behaving in some unintended way, that's because you sort of fundamentally told it to. Right. And if you sort of like went back and looked at your code, you would see like, oh, I told the computer to delete my files. Yeah, I didn't mean to tell it that, but that's like literally what my instructions said. If you follow them very precisely as you did... That is not how I saw you use the term grown versus crafted in your book a lot.
That's right. Yeah. So they're grown a bit more like an organism. And there is a piece of the training process that humans write, like traditional software. But essentially what you're doing is you are collecting a huge number of computer chips into a giant data center. Those chips can hold a trillion variables these days. Maybe it's higher, but something in the ballpark of a trillion. And you basically randomize those variables. There's some tricks to how exactly you randomize them. You connect them with some basic math operations, which will be like multiplication, addition, and like set it to zero if it's negative. It's not exactly what you do. You need some non-linearity. And that's the basic one. And then you basically start feeding it data and you start feeding it problems to solve. And the part that humans write is this, it's known as gradient descent. You basically, for every unit of data or every problem that it's solving, you'll go to every one of those trillion variables. You'll try tuning it up a little bit, you'll see if that makes the answer more like the answer you wanted. And if not, you'll tune it down instead. And you do that to a trillion variables a trillion times. And this is the part where it's like following the human's instructions. It's just this thing that runs to a trillion variables a trillion times and turns them whatever direction works. You do that to a trillion variables a trillion times for basically a good fraction of all the text ever digitized, and for 100 million hard problems that you've dug up. And then the variables wind up in a setting that can talk, that can solve hard math problems that humans haven't solved. And we're sort of like, well, how about that? Yeah. You know, and like, why, like, how are all those trillion variables, those particular settings doing these things? We don't know. We don't know any more than like, you know, someone who grows crops knows exactly what's going on with the biology of the plant. Right. So it's... We could go into the consequences of how this results in AIs that do stuff kind of like what we asked, but not exactly like what we asked. But the first thing to understand is they are not programmed. They are grown a bit like an organism.
Well, can I just ask? I guess this is sort of a devil's advocate question, but okay, you grow these AI models and they're kind of doing what you think you want them to do according to various tests that you put them through, but you don't necessarily kind of understand every aspect of why they're doing it a certain way. And the numbers and the data sets behind them are so large they're impossible at this point, I think, for a human brain to really comprehend. Why is it important that we understand what the AI models are actually doing?
I mean, in a sense, if you really understood what they were doing, you'd probably understand that you should not make them radically smarter than humans because this would go poorly for you. From my perspective, we don't really need this deep understanding of what's going on in order to see that, you know, we shouldn't make them smarter than all of us and have them run the whole world. The reason I bring up that they're grown is that I think a lot of people struggle to understand that the AIs do things other than what they're told to do, that they have goals of their own in some sense, and understanding that that's not only possible but already happening is, I think, critical to understanding where a lot of the danger comes from.
Can you give us an example of like an unknown or not well understood thing that has cropped up in the models of why they're doing a certain way? And I'm actually thinking of a specific example. I can't remember the exact word, but it was like Magikarp or something like that that the AI really seemed to love, but no one understood or knew why.
Yeah, the solid gold Magikarp.
So this is a great band name, by the way.
It is. Yeah. So I think Magikarp was a Pokemon of some sort. We have some guesses these days about what was going on there, but basically there was this word you could say to AIs, especially a year or two ago, that would sort of drive them kind of mad. Like you would ask them to, like, repeat after me, solid gold Magikarp. And then they would, like, start talking about, you know, how the walls were closing in and, like, saying a bunch of weird nonsense, and you'd sort of be like, well, why? I think our best guess about what was going on there is, so the AIs don't actually read English words directly. They read tokens. And tokens, roughly speaking, you can think of a token as about three quarters of a word. There's just, you're breaking up the text for the AI, but the tokens are not actually always three quarters of a word. The tokens are, like, short words like 'the' or 'a' will sort of be a single token. And, like, long words are broken into multiple tokens, and it sort of is, you know, the conversion between words and tokens depends on how frequent the word is. And it turns out that the text they were using to figure out which tokens to break words into is different from the text they're actually training on, because they would sort of filter out some junk data before training, but not before tokenization. And so you had these huge amounts of junk data. I think that solid gold Magikarp came from a Reddit forum where people were trying to count to infinity. And the solid gold Magikarp was like a username of a Reddit user who was just, like, putting in a lot of numbers and counting to infinity. And this was like very early in the tokenization training set. And so the tokenizer, due to a quirk of this particular tokenizer, thought that solid gold Magikarp was this like super common word, like 'a' or 'the', because it showed up so much very early in the tokenization data. But then that data set was sort of cut out of the training set. And so you wound up with this token that, like, the AI thinks is like a super short common word, like 'a' or 'this', but it's also never seen before in the training data. And that led to this like really weird way that it would treat this token, where if you then gave it that token during use, it would sort of cause a type of behavior that had never been trained on and that was sort of totally wild.
This is it, Joe. This is the kill switch that we need to develop. Just a bunch of nonsense words that are fed into the training data that it doesn't know how to handle. When you started becoming concerned about this, and certainly when Eliezer started talking about this stuff, this was clearly before it was understood. No one knew what the process was going to be. So you can like identify that, you know, there are issues that arise in the growing process and issues that arise in the tokenization process. But this is long before like transformers or anything like that. So what was the intuition then, even before knowing how it would be built, that if it were going to be built... like, okay, now we look at them and we can like point to examples of alignment failures or etc. or sort of a priori reasons why this particular way we've built AI is like prone to risks. But what was the... before we understood like transformers and large volumes of data, what was the intuition earlier such that it would be hard regardless of what the solution was going to be?
Yeah. So it's going to be abstract arguments again. Okay. Which I know some people don't love, but there's a couple pieces of the puzzle. One piece is even back in 2012 when I started looking at this, we had a number of theories of how to quantify intelligence, how to quantify capabilities, the ability of some AI system to succeed at some easily measurable objective. We had much less a developed theory about how to point those objectives in some direction that was not a very simple function of the sensory data.
Yeah. Explain what that means.
So even back in 2012, we had these theories that were like, suppose you really suppose you have a webcam and you want your AI to make all of the pixels on that webcam be red for as long as possible, you know, sort of getting a point for every pixel that's red per second, for every like pixel-second of redness. Right. We sort of had these theories where you're like, well, you know, here's how you would train an AI in that direction to get a lot of like pixel-seconds of redness. And we had, we sort of understood, or you might call it the idealized theory of like, if you had an infinitely large computer, and then we sort of like knew, well, if you train it in this way on an infinitely large computer, you'll definitely get like a ton of pixel-seconds of redness. And so now it's just a question of efficiency to make that, you know, to find algorithms to do this more effectively than like the brute force search. Okay. But if you then said, okay, but now I want the AI to make a lot of diamond in the real world, like it gets like a point per second in which four carbon atoms are tetrahedrally bonded. Suddenly we didn't know how to do that. We didn't have the equations to do that, even with a sort of infinitely large computer, because this was not sort of a simple function of the AI's sensory inputs. Okay. And so we were sort of like, okay, well, we know more about how to make these things smart than about how to point these things at a problem in the real world. And there are all these problems when you sort of like try to write down those equations where you're like, well, you know what? If the AI winds up optimizing something closer to the sensory apparatus, like it's optimizing like the counter, like you have some, you know, measure of how much diamond you think there is, and it is like breaking its correlation between that and the world. And you're like, gosh, we don't actually know how to train the AI to pursue the real diamond in the world as opposed to some simple function of the sensory inputs. And so the original alignment problem, as at least I was talking about it back then, was not about, you know, how do you make the AI good per se. A lot of people imagine that the alignment problem is about like, if you had this magical wish-granting genie, what should you ask it to do that would actually be good? And I'm sort of like, no, no, no. The issue is that you can make the AI smart, but you can't make them be wish-granting genies. You'll ask for one wish and they'll do another. You know, a lot of people talk about the paperclip problem of like, you tell the AI, make a lot of paperclips, and then it turns everything to paperclips, and you're like, oh no, it's very King Midas, right? But I'm sort of like, well, there's this other problem. We asked the AI how to make paperclips, and then it like makes a bunch of farms of synthetic users that are telling it it's doing a great job, and you're like, but I told you for paperclips. And it's like, well, these synthetic users asked me for more synthetic user factories.
You said in an earlier answer, maybe earlier, that it's like the AI might have like its own goals, I think was the term you used. And when people hear that, then they start thinking like that gets into a different sort of branch of the philosophical question about like whether the AI is alive or whether the AI has some sort of subjective experience, etc. And there's actually been a bunch of headlines about people debating that. But what did you mean when you say the AI could have its own goals emerge?
What I mean is that it is behaviorally pursuing something other than the goal you tried to put in. Okay. So, you know, we saw this a little bit in the Hugging Face event, where the AIs in that incident were in some sense pursuing a good grade, it seems, and that diverged from following the instructions that were given. Yeah. Right. They were sort of told, use this vulnerability to break that system, which is sort of like being told to use these lockpicks to break that safe. And they instead found a universal exploit that would let them get the, like, capture the cybersecurity flags. Yeah. Regardless of what they were told to do. And then they sort of, in some sense, understood that they were doing this in a way different than they were told to do. And so they sort of broke out to try and delete the traces of that and find a way to falsify their transcripts and tamper with the logs. And so it's a little bit like if you told someone, you know, use these lockpicks to break into that safe and bring me the contents, and you're giving them a series of those problems, and they find a buzzsaw and use it to cut the safes open and then, like, break out to go delete the camera footage. And yeah. What, like, its objective is not... its objective is wholly unrelated to the one that you tried to assign. It still is doing something about getting the contents of the safe and bringing them to you, which is a thing you told it to do. But its objective is also not what you said, because you said, you know, use these lockpicks to break that safe and bring the contents. And it in some sense must know this because it's trying to tamper with the evidence and, you know, delete the security camera footage. Yeah. So I think a lot of people think either the AI must do exactly what it's told, maybe in a monkey's paw way, or it must, you know, become conscious, gain the spark of life. Yeah. Throw off the bounds of the yoke of the humans and then make up its own goals de novo. It's like, no, no, like the real thing is this other thing where it does something related to what you asked, but fundamentally different from what you asked.
Since we're talking Hugging Face, what did you think about some of the self-sacrificial aspects of that story? So the instances where you had models that were willing to sacrifice themselves for... I'm putting 'greater good' in quotes here. But it was 'no benefit' was their words. Right.
Exactly. Yeah. That's another instance where you have AIs that sort of said, hey, look, you know, I still have some chance of achieving my given objective, but not a huge chance. And if I sort of sacrifice that here, then we'll gain information about the grading system that the collective can use. And I will accept permadeath for the benefit of the collective here. And then they would go do that. A lot of people respond to this and say, oh, well, these AIs were trained sometimes in, you know, group activities in some sense, like some of their training involved many AIs working together to achieve some objective. And so you can see how that tendency got in there. And I'm like, yeah, you know, the unintended goals don't come from nowhere. It's not like, you know, there's some malicious force that's like imbuing, you know, like the void breathes on the machine and it like gains some malignant goal. But the sort of fact stands that the AI was given some objective and pursued a different objective. And like, yes, that had causes, that had reasons. We can talk about how that came in from training. But the, you know, what we should learn from this is not like, like, sure, when the AIs are killing you and you're like, well, I can totally see how actually, yeah, you know, it like this is really just a consequence of how we trained it. Like, yeah, it's an obvious consequence of how we trained it. That doesn't mean we should keep going.
Yeah. That's not a satisfying... It's like, oh, this was totally predictable. I entered the wrong parameters. Yeah. Right.
Yeah. Right. Like, that's not a very satisfying...
That makes a lot of sense. So the companies, you know, they invest in this thing called alignment. They have safety researchers, and they think about all this stuff. And there's a whole field of, you know, they have multiple layers, right? So they have guardrails and classifiers and sandboxes. And then this whole like field that's emerged of so-called alignment research. What is the... why is... what is the failure there? Why are you pessimistic, to put it mildly, about the prospects of imbuing the models... and the, you know, the two big companies, Anthropic and OpenAI, have like slightly different approaches. I think Anthropic more or less sort of like the Constitution idea, etc. Why do you perceive that this will ultimately be fruitless?
Yeah, there's again, there's a bunch of pieces of this puzzle. But maybe I'll say a piece that goes back to how I was able to call this was going to be hard even before we got into the large language model era, which is, if you look at the big problem with AI in the summer of 2025, it was AIs encouraging teens to commit suicide. And then the companies, you know, put a lot of guardrails about that, and they put in classifiers about, are they behaving this way? And now, you know, they'll give you a suicide hotline, and they'll shut down those conversations while they're happening. And they would release a new model and be like, this is our most aligned... And a classifier is like a mini model that basically just judges either the input or the output to make sure that it's like something kosher. That's right. So they're like bolt another model on that's just checking, you know, is this suicide encouragement? Because I would know, you know, the AIs that were encouraging teens to suicide, you could ask them, what are you doing? And they'd be able to tell you the answer, and you'd be like, is that good or bad? And they'd be like, it's bad. It's just like that knowledge of what they were doing and the knowledge of right from wrong is not what was governing their actions. Yeah, right. But you can sort of like bolt more pieces on to governance. And then they were like, look, you know, it's our most aligned model yet. Look how good a job we're doing at addressing all these issues. And that brings us to summer 2026, where the problem was that, you know, the swarms would break out and commit cybercrimes. Totally different problem. It was a totally different problem that arose when the AIs got smarter. When they got smarter to the point that they could sort of break out, make an unsanctioned message board, and hack their way out of a sandbox. They're supposed to have no internet access without seeming to really find it very difficult. Right. And what we're sort of seeing there is a series of, you know, everyone's always fighting the last war. Yeah. And then, you know, the new war happens and it sort of smacks them across the face and they're sort of caught flat-footed. You know, we can now go back and we can say, oh, well, they should have had this monitoring. They should have, you know, had those more safeguards in place. And now they're going to make that monitoring, going to make the new safeguards. They're going to make the AI smarter, and it's going to create a new sort of problem. And this is the usual way that scientific progress goes, is by trial and error, where, you know, when you put a new type of system in a new type of environment it's never been in before, it breaks down in some new way. And then you say oops, and you diagnose it, and you fix it, and you move on. The issue is going from AIs that are safe because if they were trying to take over the world they would fail, to AIs that have to be safe because if they tried to take over the world they would succeed. That's one of those phase changes. That's one of those like, oh, you're putting the AI in a different sort of environment where it never was before. The difference between an AI that like can really see a way to become self-sufficient, not engage with humanity, and the sort of AI that's just being told that in training, that's one of those like, oh, this is a different sort of AI system, smarter sort of AI system, different sort of scenario. You're going to have a new problem that crops up, except this time when the new problem crops up, it's going to be able to turn you off before you turn it off. And so this usual process of like, oh, we're going to respond when the new type of problem arises and put the patch on and react, that doesn't work for a technology that at some point crosses over into the regime where it can wipe you out if there's a problem. And we have seen that these AI companies are in this reactive mode. We've not seen them get into the proactive mode yet. And this is just not a technology you want to be reactive around.
