If we're right now about to give birth to superintelligence, it's like what are the chances that you would just happen to be born right when that happened and then be having interviews about that, and maybe somebody would have to be there. So it does seem like there are these kind of larger dynamics at play somehow. Like I think this could be like a really monumental thing. So you could view humanity as kind of being heavy with child at the moment, and the child is some kind of superintelligence form. That trying to somehow manage that well seems to be the big challenge of our time. It's like we are first-time parents here. We don't really know what's going on. Maybe it's a world where there are no manuals for how to do obstetrics and all the rest of it. So we are sort of trying to improvise our way through it, which is a bit disconcerting.
Hanging in there. Still at large.
Okay. Well, that's hanging in there is victory by my standards. Let me introduce this. I'm Robert Wright, publisher of the Nonzero newsletter. I'm also relevantly to this conversation author of a recently published book on artificial intelligence called The God Test, excerpts of which are available at thegodtest.net. You are Nick Bostrom, a philosopher and a very well-known figure in AI circles. You are the author of the famous, now famous, paperclip-maximizing thought experiment in which an AI told to maximize the production of paper clips follows instructions, winds up destroying all of humankind. You are also, well, you know, recently you're author of a book called Deep Utopia that you and I discussed on this podcast a couple years ago, but also the author importantly of a 2014, I think, book called Superintelligence, which was read by a lot of people who are now known as leading lights in the field of AI, like Sam Altman, Elon Musk, had a big impact on them, on the conversation. It's probably the reason, I would think, that the word superintelligence is suddenly on so many people's lips.
Now you are also the originator of a concept that I want to bring into this conversation called the singleton. You didn't think of the word, but you came up with a meaning of it that you use in the word superintelligence, and I made use of the concept in my book, The God Test. In fact, there's a chapter called The Singularity and the Singleton. And here's what's interesting to me, among other things. I deployed the concept of the singleton in the course of arguing that we should slow AI down. You are not on the slow down bandwagon, as I understand it. And so it'll be interesting to kind of compare and contrast our views on the wisdom of developing fast as opposed to slow, and how the whole idea of the singleton figures into that. We should, of course, pretty soon define a singleton. First, I just want to pause and give you a chance to correct anything I've said that you think is wrong, or provide context for anything you've said that I think is right.
Um, yeah, no, I think that's broadly, I think, I think this paperclip idea, I think had been floating around. I think Eliezer had some version of this as well, earlier on, slightly different, but sort of also related to...
Yeah, there's an interesting, we won't get into this, but when I hear him elaborate on his version, it seems like he's not actually making the same point that you were making, but maybe I'm confused. In any event, yeah, you are commonly cited as the author, but I'm willing to give you two co-authorship, so fine. Go ahead with anything else you want to say.
Okay. So yeah, so first of all, let's talk about your view on the wisdom of rapid advance right now. And, you know, because you come from a philosophical background that not everyone shares, and there may be some people who believe in rapid advance who might hear your particular argument and, I don't know, not agree with it. For people who want to see a full version of it, by the way, there's a paper, I think you call it a working paper on your site, your website, that lays out some of the logic behind, you know, full speed ahead.
Yeah. Yeah. I mean, I'm not an accelerationist either. I think...
I would say up until this point, fast progress in AI has been good. We have quite useful AIs. Now I think moving deeper into the machine intelligence transition, there could easily be scenarios where it would be very valuable to have the option to slow down, to pace the frontier, to use the terminology that has recently become fashionable.
Okay. For example, if capability advances run ahead of safety and alignment capabilities, right? If you then could maybe moderate the pace of capability advances to allow safety and alignment to catch up.
Okay. Right now we have a very competitive situation between the frontier labs in the US. You know, the leader might have, you know, a month or two on the closest follower. So if they felt that they can make some breakthrough that they need a little extra time to sort of double check everything and align it, and like, you know, they could slow down for a month or two, any more than that and they basically just hand over the initiative to maybe some less scrupulous competitor. So the race goes to whoever is most capable but also most willing to accept risks. That seems suboptimal.
Okay. Although we should say that the concept of the singleton is all about solving coordination problems, right? So if you imagine a world in which it wasn't going to be the case that a less scrupulous competitor caught up, would you then be fine with a longer pause?
I think the length of the pause depends on a number of factors. One is how much would the risk decline with extending the pause. I think possibly there will be very high utility to the first bit of pause. They're like the low-hanging fruits you can pick if you had a little extra time, and then there will be sort of diminishing returns. So the per month benefit of pausing will probably be largest for month one and month two, and then like after you've paused for like 10 years already, like how much is the risk level really going to decline in year 11? So I think the benefits start plausibly very high and then diminish. And then there is a sort of, from a personal point of view, if you care about currently existing people, a somewhat constant cost to each month of pausing in that many people who would have been saved if we successfully develop aligned AI who could then, you know, participate in medical breakthroughs and so forth will suffer and die in the meantime if we delay it. And then on top of that, there is an extra layer if you care about future generations. If you have a more utilitarian perspective, you also have to take into account how existential risk goes up or down over time. You know, AI itself is one source of existential risk, but aligned, safe superintelligence could help protect against other existential risks. So it depends there a little bit on how high you think the background level of existential risk is from other sources.
Right. And let me quote a sentence from this working paper of yours, which is at nickbostrom.com. The paper's called Optimal Timing for Superintelligence. It says, this is the first sentence of the abstract, actually: developing superintelligence is not like playing Russian roulette, it is more like undergoing risky surgery for a condition that will otherwise prove fatal. In what sense will the condition otherwise prove fatal? And this is a paper, by the way, in which you lay out some of the logic behind, you know, I would say, relatively rapid advance, some of the virtues of getting to superintelligence, a term that you acknowledge is hard to, you know, precisely define. But what did you mean by that sentence?
Well, I should say first, by way of framing it, so this particular paper looks at one of the relevant questions, which is if we are thinking about what would maximize life expectancy for currently existing people, then what would be the optimal timeline for developing superintelligence. So I think that's an interesting question. I think it's not the only relevant question. I think helping existing people is not the only thing that should factor into our decision calculus. I think we should also put some weight on, you know, ensuring that the farther future goes well, whether or not we are parts of it. So in an all things considered evaluation, you would need to add in other analysis as well that takes into account how it affects levels of existential risk. But if we zoom in and focus on the impact on currently existing people, then what I meant by the condition being terminal is that we look very likely to die soon in the absence of some radical change, a breakthrough in AI that could sort of unlock some almost magical technological powers. And I mean, in fact, most people who have ever lived are already dead, and the rest of us, you know, depending on age and health conditions, might hope for another few years or few decades. And we are all kind of rotting from the inside from aging processes, and there is just like a huge amount of misery and death and disability that is expected to occur every year that we don't have some major change in that. So, you know, I think what prompted it partly was this, Eliezer Yudkowsky and Soares have this book called If Anyone Builds It, Everyone Dies. And it seemed to me that one could make an equally or maybe stronger case for the claim that if nobody builds it, everyone dies. That seems very likely given all the empirical data about people dying all the time, dies a lot sooner than they would otherwise.
And well, I mean, let's talk magnitude. Maybe I think it might help people understand your argument. You're imagining that superintelligence could help us add hundreds of years to our lives, maybe even thousands. Right?
That's part of it. It's quite open-ended. I mean, once you solve aging and like have this technological maturity, I really think, I mean, I sort of for the sake of having a concrete number in the paper, look at what happens if we weren't aging. If we had the death rate of somebody's like in their 20s, so then maybe life expectancy would be about 1,400 years or thereabouts. That'd still be deaths from like murder and suicide and stuff, but like way lower. Now, I think likely that is a big underestimate because there are other things that you could do at technological maturity that could, you know, push down some of those remaining risks further. And, you know, eventually we could maybe move to a digital substrate if that's your thing. And then you could have backup copies and all kinds of things.
I'd kind of just like to stop at, you know, after you extend my life by a thousand years. I'm just fine there. You don't need to plug any chips into my brain. But let me ask you a question about your framework. You know, there's this concept called expected return, various other things, where, you know, like if it's a stock portfolio, you know, you say, well, the chances are, you know, 10% that this will make me a million dollars and 90% that it'll wind up with nothing. And so the expected return is $100,000 if I'm doing the math right. Now, if you apply that kind of logic in a utilitarian context, you know, you could wind up saying things like, well, the chances are, you know, 99% that AI will wipe out all of us, but if we don't stop it, if we let it go, there's a 1% chance that it will increase, you know, the total amount, you know, sentient, rich, sentient life kind of like ours by a factor of a million. Now, if you're going with expected return, then you say, well, then, okay, even though the risk that it will wipe us all out is 99%, you let's full speed ahead. And I'm just using that to ask, are you an expected return guy or not?
Well, I mean, there are a number of premises needed in the reasoning you outlined to get to that counterintuitive conclusion. And so you can reject some of those premises, which in fact is what I would do. The first, in your example, it seemed that there was a utilitarian premise, which in the classical formulation counts everybody equal, including possible future beings who might come into existence if we make certain choices and live happy lives. And one reason why I'm not a utilitarian is that it does seem to have these very counterintuitive implications if one really thinks through the logic. And there are some other problems with what I call aggregative consequentialism. This is when you sort of aggregate over the entire world, right? And then choose the action that has the best consequences.
But am I wrong in thinking that as kind of a lay person who took three philosophy courses in college, thinks about these things and thinks like virtue ethics, utilitarianism, a couple of other things, you would be closer to the, there's a utilitarian spirit to your analysis, right?
Well, maybe to some of my analyses, because often, and this is like one common source of misunderstanding about my thinking, is that for any particular paper, often I try to zoom in on one specific sub-question. So I analyze like, if you're a utilitarian, then what follows from that given these assumptions, and then in another paper, maybe what if you accept these starting points, you end up here. And then some people read one of these papers and assume that I adopt the conclusion and therefore the premises of that particular paper. But it's a way to make intellectual progress, the divide and conquer, like rather than trying in every paper to answer what all things considered is my bottom-line view about where we should be going as a human civilization or what I should do with my life, that is kind of intractable often to answer in a small contribution. And so you sort of chop out a little part of the problem, analyze that, then move on. So if you're asking about my all things considered view, I mean, I'm closer to something like a moral pluralist. I have this conception of a moral parliament, is this idea for how to make a decision under moral uncertainty where you would assign, you know, maybe probability to different moral theories and frameworks. A utilitarian, the utilitarian party would have some representation there. They would send some delegates to this imaginary parliament, right? The deontologists would send some, the virtue ethicists, ethics of care people. You can also throw in self-interest, like, you know, to some extent I'm not a perfectly moral being. I care more about, you know, myself and my family than a randomly selected person. And so you sort of allow these different components of your overall value function to send delegates in proportion to the strength of that faction. And then you imagine this parliament kind of deliberating and trading off. So maybe in some situations where other parties don't care much, maybe the utilitarians would get their way. They really care strongly about this issue, and then they would give way to these other parties on issues that seem more or less indifferent to the utilitarians. And I wouldn't necessarily, I don't know, maybe compared to the average person on the street, I would have slightly larger influence from the utilitarian faction, but much less than many of my friends who are sort of more wholeheartedly committed to some aggregative consequentialist ethics, particularly the effective altruism community. I think that's a fairly common view.
Okay. So, let's talk about the singleton now.
So, in other words, I'm sort of less crazy than some people believe in that way. Maybe more crazy in other ways that people don't know about.
Okay. I'll leave it for the crazy people you're referring to to decide whether they agree with that categorization and leave it to everyone else. Okay, so let's talk about the singleton and let me say a little bit about it and then explain why I became so interested in it and then give you a chance to correct me. So it is a, you know, this is, it appears in your book Superintelligence and it appeared earlier in a paper called What Is a Singleton? back in 2005. That's actually where I came across it and I must say I was very gratified to scroll to the bottom and see that you had cited my book Nonzero, which made perfect sense to me because, so singleton, and again, you're going to get a chance to step in and say that everything I just said is wrong, but a singleton, it is a global coordinating mechanism. And you write in the book, a singleton, so that could be if we're talking about humans doing the coordinating, especially, could be a fairly conventional kind of conception of global governance or something. You write, quote, a singleton could be a democracy, a tyranny, a single dominant AI, a strong set of global norms that include effective provisions for their own enforcement, or even an alien overlord. You know, so the main thing is it would be something that's governing the planet. In your paper, you had written a democratic world republic could be a kind of singleton. Now, I had argued in Nonzero that technological evolution was pushing us toward global governance because it was increasing the kind of number and magnitude of non-zero-sum dynamics among nations. There were more opportunities for win-win games, but also a lot of opportunities for loss-loss, you know, nuclear war, climate change, various things. And in my current book, I depict AI as just kind of a logical extension of that, except I would say an extreme one. It greatly, greatly strengthens the logic behind international governance. There are a lot of possible downsides of AI that cross national borders. That's the argument. So I made heavy use of your concept of the singleton in my analysis. And, you know, part of the idea is that, and it's striking to me that over only the last few weeks this has become a subject of common conversation, but part of the argument is that if you want a slowdown, which I think would be a good idea, it has to be coordinated ultimately on a global level. Not right now. I mean, there's only two superpowers in AI, kind of the US and China. And in fact, the true frontier companies are all in America. So you can imagine a slowdown that wasn't globally coordinated. But ultimately, I think a lot of things are moving in the direction of global governance. And one thing I liked about your book, Superintelligence, was by my reading, and again, this is maybe a little impressionistic, and you should feel free to correct me about this if it's wrong, but by my reading, it did kind of do a good job of arguing that, you know, ultimately some kind of singleton is probably in the cards. My own view is like you either get to some form of international governance, setting aside the question of whether humans or machines are in charge, you either get to some sort of coordination or chaos and death and destruction will reign. That's my own view of where technology is pushing us. But I want to now let you have your say about what exactly you meant by this singleton and where you see the logic behind the concept applying to our current situation.
Yeah. So the concept, I mean, the word comes from set theory where a singleton is simply a set with only one member. And I introduced this concept of the singleton in governance because it seemed useful to have an abstract concept for a certain kind of property that the world order could have, basically the property that the substantial global coordination problems are solved. And then that could be instantiated in many different ways. Some of which could be horribly bad, some of which might be really good. But when trying to organize one's thinking about possible futures for humanity, there are certain concepts that are useful to have because they make it sort of easier to think of similarities between certain scenarios. There are certain dynamics that will occur if you have a multipolar future where there's competitive dynamics, whether evolutionary or others, that kind of constrains the range of possible outcomes. If you have a singleton, in principle, a wider range of different possible outcomes are conceivable because you sort of have removed one of the constraining factors that operate when you have a competitive situation. And now I would also say that it looks to me that the root cause of many of the biggest risks that we face of having either no future or like a really bad one arise from conflict in different ways. The fact that the world is sort of often working at cross purposes and not just with AI risks, but also with, I mean, obviously with nuclear war, something is obvious, but I mean, even some global commons things, global warming, like it can be sort of in each individual country's interest to pollute at a high level and reap the economic rewards, and that might result in a suboptimally high level of pollution. And so, and you have the whole arms industry is driven by this and all kinds of things. So, so that I very much agree with, and of course your non-zero, like, so my concept is very abstract, like you paint a much richer sort of story of how this could emerge from sort of lower level interactions kind of growing up and eventually consolidating. Now it is also at the same time true that singletons could also be bad, really bad, in that it's kind of putting potentially all the eggs in one basket. And if something happens to that basket, if some pernicious ideology seizes control over this coordinating mechanism, or it suffers some kind of, I don't know, internal bureaucratic sclerosis, there is nothing outside it that could eventually sort of replace it or overthrow it, or that even one could look at to realize how badly things have gone wrong. Like I think if we look at nations today, one of the things that sometimes leads to reform is that you could look out and see others are doing things differently and it seems to work for them. You know, over longer time scales, maybe you get bought out or influenced or invaded. Throughout history, sufficiently dysfunctional societies have often then been replaced by, you know, societies that organized in a different way that allow them to exert influence. And so it then becomes a more complicated question how you sort of trade these different risks off against one another. And there I think the answer might not exist at the fully general level. One has to then zoom in and look at what are the actual current major risks, like what are the types of singleton, you know, that they might have different levels of protection against sort of going off the rails. So that's roughly, I guess, at a very general level, my thinking about that.
Mhm. Yeah. On the point that singleton could be good or bad, I rarely begin a sentence with a phrase in defense of Peter Thiel. But I will say in defense of Peter Thiel, when he says things that make him sound to some people like a maniac, like, you know, climate change activists are the antichrist or some of them are kind of compares them to the antichrist, the underlying concern that you can imagine an authoritarian repressive form of global government eventually emerging is not crazy. And in fact, I would say it's much less crazy in a context of artificial intelligence precisely because it's a technology that can be used to repress. And I think that's one reason I want to start the conversation about global governance early, because I think the more gradually and intentionally and consciously we approach it, the less likely we are to get kind of, you know, pushed into a bad version of it. I mean, you have, you sketch out in your, in, I think it's a whole chapter on the singularity in your book Superintelligence, these various scenarios by which you could wind up getting to a singleton, and some of them do involve just, you know, flat-out takeover, right? Like a group with, for example, access to the most powerful AI just succeeds in taking over. And by the way, that could happen at the national level, which is another reason I worry about overly rapid development of AI. I think if it's too socially destabilizing, you may be opening the doors for an authoritarian takeover at the national level. But to get back to the singleton, I think you, you know, you do a good job of showing, I mean, at the end of my book, the AI book, I kind of say like, it's as if technology is saying to us, we can do this the easy way or we can do it the hard way, right? Like one way or another, unless you want a world full of chaos, death and suffering, you're going to wind up, or extinction, that's always a possibility, you're going to wind up with some degree of global coordination. And, you know, if you don't start thinking about doing it carefully, it could be done in a less careful way that winds up being bad for most people. Does that logic make sense to you? Is it, I see it as being consistent with the argument you make in that chapter of Superintelligence, but maybe I'm wrong.
Yeah, I think it would be positive if, you know, countries could get along better with one another for a start. And I think there is often an underestimation of the value of certain norms that we've managed to develop for governing the interactions between states and basic human rights norms and non-aggression norms and stuff like that, where in any one situation, if you're a country and you have a strong view and desire, it seems kind of inconvenient to sort of honor this norm that is anyway very loosely enforced and often violated by others. But I think it's a sort of endowment of humanity that we've slowly accumulated from different people doing responsible things over hundreds of years, and it's still very weak. And, you know, but you sort of draw down on that endowment at your peril because, like, even though these norms are very weak, paying respect to them might still be worth a lot in sort of taking, like, blunting the edge of some of these international competitions. So I think things that could strengthen that would probably be good. I think a lot depends then, like, on exactly which norms and rules or institutions, because, like, you can't have sort of a local progress, like we strengthen some form of coordination that's partial, and then it's very inefficiently or poorly implemented, then people look at that and they say, do we want more of that? Like some, I don't know, like some dysfunctional UN committee that just seems to waste resources, and then the result of that is people recoil away from the idea of moving further in that direction. And so it gets very complicated. Sort of, I think I tend to maybe think more on the margin with these things. So there's like various possible ideal states that one could imagine, but given the way the world is currently, like what currently on the margin would be a small little notch that wouldn't cost anyone too much that would sort of make outcomes a little bit more cooperative.
Mhm. And do you have a very specific idea of the kinds of coordination that you think may ultimately be necessary? I think, you know, we've established that slowing down AI would require coordination among companies, nations. Ultimately, there's a collective action problem. And so, you know, it needs to be some sort of implicit or explicit agreement. But, well, go ahead. You wanted to say something about that?
