Hello and welcome to the American Compass podcast. My name is Orin Cass. I'm the chief economist at American Compass. And we are going a bit off the beaten path today for a discussion about doomsday artificial intelligence scenarios with Eliezer Yudkowsky and Nate Soares, both of the Machine Intelligence Research Institute and authors of the fascinating and much-talked-about new book, If Anyone Builds It, Everyone Dies. As I told them at the start, I love the title. You don't have to go reading through to figure out what the thesis is. It's if anyone builds superintelligence, everyone dies. And as you will see, I am somewhat skeptical of this premise in both respects, but was very interested to talk to them about it. They are researchers and very prominent voices making the case for much higher AI safety, regulation, ultimately some sort of international governance. And I think it's really important to understand that perspective. It is certainly playing a role in our policy discussions and you'll see I interrogate it fairly thoroughly. So certainly my own view will come through. They do a great job articulating their side of it and hopefully it will be a useful conversation for folks in understanding what the parameters of the debate are and let you all make up your mind about whether we need to be unplugging all the computers and building our bunkers. Although I gather the bunkers will not help you if the superintelligence comes for you. So enjoy that conversation.
But first I have to of course tell you about everything else going on at Commonplace. Had some really interesting stuff this week. Senator Jim Banks on the need to bring rare earth processing back to the US. Had a lot of great stuff on rare earths over the last week and I think this piece does a really nice job framing up the legislative side of it. Our own editor Hasten Willis on why traditionally you would expect the left to support tariffs and the ways in which they are undermining their own agenda by allowing their dislike for Trump to lead them into a position that whatever allows us to get the cheapest stuff from China must be the best for Americans. And then finally brought back a great piece that John Carney had actually written a few months ago for us. Zoran's Park Slope Populists, looking at the Mamdani coalition and the ways I think some folks maybe misunderstand it. So be sure to take a look at that and enjoy this conversation with Eliezer and Nate.
Nate and Eliezer, thank you both for joining the podcast.
Thank you, Brent. And thank you especially for writing a book whose thesis is so clearly stated right in the title. I think often you've got to read through the whole thing, then you're trying to figure out, okay, but what are they really saying? Not here. We've got it perfectly laid out in a two-state, I guess what a two-clause statement is. I never did formal logic. Is one a predicate of the other or something like that?
Yeah. If anyone builds it, everyone dies. I really just, I kind of want to dive into those elements and work through it, which I guess begins with the 'it'. When you talk about if anyone builds it, what is the 'it' that you have in mind?
Yeah, it's artificial superintelligence, which, you know, despite Mark Zuckerberg recently saying that he'll build superintelligence that helps you get better Instagram reels or whatever, when we talk about superintelligence, we're talking about AIs that are better than humans at every mental task, better than the best human at every mental task. You know, that's not where ChatGPT is today, but it is where these companies are racing. And maybe a bit more precisely, we're talking about artificial superintelligence built using anything remotely like the current methods where AI is largely just, you know, grown more like an organism rather than carefully crafted like traditional software.
Yeah. And why is that distinction so important? I mean, obviously what is going on with these LLMs is very different. Is it, you are less worried about something, I take it, that is sort of written with code than something that grows out of a training process?
I mean, it'd still be tricky to solve a lot of the problems of making AI go well if it was carefully crafted. You know, I got into the AI alignment line of work back in 2013 or 2014, depending how you count it. And back then it looked like the AI might wind up being, you know, carefully crafted by people who knew what they were doing. And I was still like, gosh, it seems to me like there's going to be some difficulties to work through. But when we're just growing them, it's way harder. That's a big part of what's caused me to switch from like, you know, we'll try our best to figure out how to do the job right before the industry figures out how to do the job at all, to like, oh gosh, we're just not ready here. You know, there's other aspects too of like things are going faster than it maybe looked like they would go from the perspective of 2014, but that's definitely a big one.
I got into this in 2001 or 2003, depending on how you count it. I used to be noticeably more optimistic. But as you know, back in the old days, there were a lot more legible techniques in the standard toolbox of artificial intelligence. The black-box techniques have now taken over most things. As that has become true, I have gotten a lot more pessimistic.
And I guess the other clarifying question I have, when, you know, I've heard superintelligence defined in these terms as sort of better at all mental tasks, and something I've seen already as we talk about kind of some of the existing models is people will say, you know, look at how well it does math problems or look at, you know, look at how well it can synthesize this molecule or whatever else. In terms of kind of a range of very discrete tasks, in your mind, a superintelligence can kind of tackle discrete tasks, or also has to have the same sort of more integrated cognitive capabilities to manage among tasks, address problems when they arise, change course and so forth. Is that sort of part of what would be superintelligence?
Yeah, I would classify those as mental tasks. You know, in part because we're just growing these AIs and we still don't really know what intelligence is or how intelligence works technically, either in the case of the human brain or in the case of these sort of fledgling AI brains. You know, it's not like I have a super rigorous definition of what's a mental task here, but yeah, you know, from my perspective, things start to get really worrying once you make the sort of AIs that can do their own scientific and technological development, much like how human civilization can do scientific and technological development. And you know, that's, I mean, having an AI that can do automated AI research might be enough to get you quite a bit more. You know, there might be some relatively discrete mental tasks that can unlock a lot more of these mental tasks down the line. I'm not saying that like you're going to need to grind on like literally all of these things to get the whole package. But yeah, I think of superintelligence as sort of being the package deal.
Okay. And to what extent are we on the path to that, or do you think that's emergent? I mean, we have some people who are sort of saying, or were very recently saying, 'Oh, we're at AGI. We'll be at AGI any day now,' realizing AGI might not be the same as superintelligence. In your mind, is the kind of LLM you're seeing trained today plausibly on the pathway to the kind of superintelligence you're talking about? Because I have to be honest, to me, it doesn't look like it.
I think it's plausible. It's not my personal best guess. Eliezer and I might have a minor disagreement here. But you know, from my perspective, it doesn't matter a ton. A lot of people who are talking about AI these days are talking mostly just about these large language models, about these chatbots, because they sort of only started looking at AI when these chatbots came to the fore in, you know, 2021 or 2022, depending how you're counting it. If you've sort of been in the field of AI for a bit longer, you know, you could imagine someone in 2016 looking at AlphaGo, which was one of the leading AIs at the time that beat the human champions at the game of Go. And you could imagine them saying, 'I don't see how these gameplay AIs are going to get dangerous. I don't see them sort of going all the way. I know they're more general than the previous AIs like Deep Blue that can only play one game, whereas here you have one architecture that can play many games. I just really don't see how they're going to do qualitatively more stuff, qualitatively better.' And you know, you'd be right. But then a year later, the transformer paper came out which unlocked large language models which did a whole lot more stuff qualitatively better. Even in the domain of these large language models, you know, a year ago now, there was the release of the reasoning models which sort of were maybe a new subparadigm where you're not just having the AI generate text directly to solve problems. You're having them, you know, generate reasoning about, or like words about how they would solve the problem and then with those words in context try to generate a solution to the problem. That sort of unlocked more abilities for these things. It gets around theoretical obstacles that some people said LLMs had towards ever being really quite smart. And you know, will another step like that be enough? I don't know. Will you need another step as big as the transformer step that unlocked large language models in the first place? I don't know. My basic take is that these companies are rushing towards superintelligence. It's their stated goal to build superintelligence. Most of these companies existed before large language models and see them as a stepping stone towards superintelligence. And I think it's pretty likely that they'll get there at some point. Calling exactly when, exactly how many steps, that's much harder.
Yeah. And besides the sort of point about it helping to have been around for long enough that you've seen people say you'll never do X, you'll never do Y, you'll never do Z because our current methods don't do it, and then somebody invents a new method, as happens every few years, the other part of it that I would worry about is that the current companies are saying they're going to build AI research AIs, and that arguably is on a slightly more direct path from what they've got now in the sense that we've got current AIs that are writing code, tweaking code, observing the results of the code that they have written and then tweaking based on that. And I do worry that there may be a path to the AI that builds the superintelligence out of the current technology even if the current technology wouldn't do it. So the two main reasons or concerns are: one, there's always a new method every few years, and two, possibly the AI that builds superintelligence is on a straight line even if the current AIs are not.
Right. I guess the reason I'm curious about this question, and this will sort of come up a few times as we're talking, is sort of, you know, where are the red lines? When should we be really concerned? It seems to me if what we think of the current approach to LLMs and their capabilities and what throwing ever more GPUs at them and so forth can do does not qualify as the 'it' in the title, and it maybe doesn't, you know, it doesn't have a clear pathway to the 'it' of the title, then it seems like there's a lot less cause for concern. Wouldn't you want to have the type of technology that plausibly is 'it' to have something to point at and say this is now the thing that's dangerous?
So I think, you know, as Eliezer said, it's possible that these LLMs will get to the point where they can do automated AI research and, you know, build better AI paradigms. That's one way things could start going fast. You know, another way things could start going fast here is it would be very difficult to look at the different breeds of monkeys before humans started civilization and tell which ones were the ones that were really going to be able to walk on the moon. And even to tell whether, you know, if you had looked only at some types of monkeys and then you sort of looked at these hominids, it might be hard to tell. You know, there's not really one thing that early hominids were significantly better at than the other animals around. They're sort of a little bit better at a lot of things. But being a little better at a lot of things was sort of enough for some, you know, loop to close.
Yeah. Is that really true? I'm not an anthropologist, admittedly, but isn't like spoken language, like a brain capable of spoken language and then the emergence of spoken language actually a quite clear break point that is easily discernible and absolutely necessary to everything that follows? It's, you know, there's, you know, if you also wait until they're sort of smelting iron, that's another thing where, you know, in retrospect you can be like, these ones are maybe going to the moon.
Yeah. But I want to stick with your analogy because I guess that it seems to me that language capability isn't a technology, like it's the actual biological capability, right, that distinguishes at the sort of brain, you know, is much more analogous to what we're talking about on the AI side. So even that came pretty fast on evolutionary time scales, and even that we see a lot of precursors in a lot of other great apes, you know, and there's many people who write papers about how they, you know, like, oh, what do humans have that no other animal has? And there's, you know, a whole cottage industry of papers saying, well, actually, other monkeys have limited versions of all of these things, you know, like if you sort of try to break down what it takes for a brain to do language in the way humans do, it's sort of made of a lot of parts, a lot of which have smaller versions or like earlier versions in these monkeys. And so humans sort of got here by being like, I think language is an example of humans didn't, you know, we didn't grow a whole new language module in our brain. We sort of got a little better at a lot of pieces of that puzzle in a way that eventually was able to come together.
So Eliezer, I want to ask you about the point about sort of thinking about this all the way back in 2001, 2003. You know, from one perspective, I take your point that, you know, you could talk about it back then and obviously there have been all of these leaps since that no one knew about or could necessarily have predicted, certainly in the specifics. Seems to me the other way of looking at it would be to say, well, we've now had 25 years of people saying, 'Yes, artificial intelligence could hypothetically become it, but it isn't yet.' And so we're sort of still there. Why is where we are now different than where we were in 2001 or 2003?
I mean, from my perspective, and in some sense, there's only a, you know, gradient rather than a sharp difference. You know, back in 2001, 2000, 2003, if you ask people like, where do you, where are you going to draw the red line for where if AI gets this far then we should start worrying, I think a lot of people back then would have said something like, 'Well, AI that talks to you like a person.' And I would have said, 'Well, it talks like a person. That's very advanced. Maybe, you know, like I would have expected that an AI that could write code wouldn't necessarily be an AI that could talk to you like a person.' And I would have been wrong about that because the specifics are pretty hard to call. But you know, from another perspective, you know, back then you're in a very similar situation of not quite knowing what the path is to superintelligence. But you could maybe at least look at current progress and say that it was slow and that the AIs couldn't talk to you like a person and they didn't show very much of an ability to write code and draw these other sort of attempted boundaries by which to say the issue is probably maybe not urgent. And you wouldn't have been saying that out of the kind of very solid science knowledge that it takes to really say something and be sure of it. Somebody could have, it could have always been the case that there is some, you know, like secret technique somebody cooks up in a laboratory somewhere, but things would have not looked like they were advancing rapidly and it would have been very easy to point to things like talk like a person or write code. The sort of precursors of generality would have been visibly absent in some sense. I would still say it's a gradient rather than a sharp divide. But it does seem to me that if you were to ask people to sort of naively name the red lines, most of them would have named red lines that we are now past. And the only one that people might have named that we're not now past is something like guys that hang out on, you know, that have rented their own server hardware, are independently earning their own money and like visibly plotting against humanity. We're not there yet. Now, we have some AIs do have what is arguably their own money at this point due to trading cryptocurrency in a sort of independent fashion or like soliciting donations of cryptocurrency and touting cryptocurrency. There is one guy called Truth Terminal that got up to $50 million on paper at one point. So we're not completely not past that red line, but most of the red lines, AI claims to be sentient, AI that can talk to you, AI that can talk to you and that can write code, we are past most of what people would have named as red lines in 2003.
Right, but in hindsight then doesn't that just mean those were bad red lines given that as where we started we agree that this thing that does all of that is nowhere close to being 'it'? Like, can we draw better red lines? I take your point, like yes, at the point where the superhuman intelligence is already trying to destroy us, we're clearly past the red line, but it seems like there should be some useful red lines between here and there. How would we, and this goes to my point also that I'm very uncomfortable with some of the, or frankly lack of measures we have of what AGI might mean. What would we be looking for? What would it mean to say, yes, this is a technological approach, this is the kind of functioning that is 'it'?
If we get to the point in 2027 or 2028 according to this company's own projections, which are maybe just completely made up numbers and maybe not, who knows.
I'll take the maybe, but go on.
And we're starting to get like major AI breakthroughs that they say was done by their AI or like their AI was one of the primary movers in it, then you might be getting pretty close to the about to cut your throat if it hasn't already.
Yeah. There isn't a solid field of science that says here are the milestones you pass in this order before you get to the thing that self-improves or the thing that kills everyone or the thing that has ensconced itself on the internet and is very, very difficult to shut off without shutting off the entire computing infrastructure. If we had lived through this one time, we'd be able to look back in retrospect and things would look inevitable and we would know what the milestones were. The thing that makes this dangerous is that we're doing it for the first time.
Yeah. So, yeah, I think the recursive AI point is a great one because you can almost sort of, you can understand how once the AI can improve the AI, there is no limit to how good the AI can get. In a world without recursive AI though, are there particular capabilities? Are there things you look for as you're reading the research that would signal this, yes, this actually qualifies as superintelligence?
I don't think superintelligence, you know, I mostly expect by the time I'm looking at something and being like, 'Yep, that's a superintelligence,' that we're probably already too late. Looking at things and being like, 'That seems like it's really on the pathway.' You know, one lesson I would sort of take from the fact that all of yesterday's red lines turned out to be kind of murky is that probably today's red lines will be murky as well. You know, you already have these AI companies saying we're going to build automated researchers. You already have people at Anthropic saying the AI is writing a lot of the AI code, right? But those numbers are like kind of squirrely today because it sort of turns out that like yes, the AI is writing a large fraction of code, but that's because the AIs can produce a huge amount of sort of sloppier code, right? And so, you know, you could have said, well, once the AIs are writing half the AI code, that'll be a bar, but then that turns out to be a messy red line. You know, part of why this situation looks pretty worrying to me is it seems like we might need to act before we have high confidence because it might be the case that we don't get the confidence until we're already past some point of no return.
Yeah. But I mean, you know, every year or so, well, probably more often than that, there's lots of papers saying we found a way to do what LLMs do with far less computing power, right? And we know that in theory that should be possible because, you know, training a frontier AI takes as much electricity as a city, training a human in a comparable sense takes as much electricity as a light bulb. So we know that there's like a lot more efficient algorithms around or like available to be invented someday. And there's lots of these papers that are like, we found a way to do this a lot more efficiently. Maybe one of those will actually work one day. You know, maybe you'll switch to a new paradigm that's much more efficient. You might not have a lot of time after that happens, but that, you know, there's sort of one thing which is like, can the LLM ever do AI research? There's another thing which is like keeping an eye out for a next breakthrough and seeing how far that breakthrough goes, where I could imagine seeing some breakthrough and that sort of is hard to guess in advance what it looks like, but seeing the AI do qualitatively better after that and being like, oh gosh, this is looking like it's now fairly directly on the path to superintelligence.
Take your current mental situation at face value right now. You don't know what the definite super red light clear warning marker where you get that before you're past the point of no return, either in terms of AI technology itself or in terms of, you know, the world having gotten too far, you know, put too many, you know, built too many data centers in too many different places. You don't know currently where's a great place to draw a red line. Let that be your baseline. Maybe you can come up with a really brilliant red line and sell the whole scientific community on it. I bet this will never happen and I bet it will never happen for good reasons. It feels like something we just don't know at this point. But your current mental situation is of not actually knowing where that sharp red line is and take that at face value that that's the correct mental situation to be in.
