Oxford Genius: AI Will Become Earth’s Dominant Mind | Nick Bostrom

Maybe God can do anything that is logically consistent, but simulators would be subject to the physical constraints operating at their level of reality. If they didn't want us to notice any glitches, I think they would be able to prevent us from doing so. In the long run, I think AIs will be completely cognitively dominant.

If we are in a simulation, I think, for example, an afterlife is a much likelier prospect. What happens after if one simulation terminates? We need to make more progress on AI alignment. We need better mechanistic interpretability tools to be able to see what's going on inside these AI minds and understand it. Why do we find ourselves alive right now in 2026? Is it possible that we're just...

Yeah, I think one of the most obvious gains from machine superintelligence would be the ability to massively accelerate progress in medical science. So diseases that maybe we could eventually get around to curing, you know, in 50 years, 100 years, I think there would be a good chance of those becoming cured very shortly after the development of AIs that can sort of do all the medical thinking and maybe run the experiments and so forth. So I think there is some urgency. I mean, if you look around at the condition of the world as it is today, it's hard to feel very complacent and think, 'Ah, well, let's just take another few centuries to sort this out.' There's kind of a lot of desperate need out there, so there's certainly a cost to delaying.

Well, I think eventually, even if there remains jaggedness, the part where the jaggedness goes inward will still exceed the disk of what the human mind is capable of. I think there are fundamental reasons to think that information processing in artificial substrate can just vastly outmatch what's possible in a biological substrate like a human brain. So in the long run, I think AIs will be completely cognitively dominant.

Well, or a good scenario. It sounds like a scenario in which there will be a fairly profound transformation in the world from the current condition where we humans are responsible for everything to a condition where we might increasingly rely on AI acting on our behalf to take care of most of the instrumental tasks that need solving.

Dead? Well, hopefully not. I mean, we humans have superior cognitive capabilities and technologies relative to, say, many animals, and they are still around, not always treated as well as they should be. But we have one key advantage with respect to superintelligence, which is that we get to build it and therefore potentially get to design it and shape it in accordance with our values. And so the idea is to develop these AIs such that they are actually harmless but also helpful to humans and can kind of empower us to achieve more of what it is that we are trying to accomplish in the world. And that would presumably include human flourishing. So far from killing us, the ideal would be that they would be saving us from all kinds of other sources of mortality to which we are subject in the status quo.

Well, there's the safety science is key in terms of ensuring that the future goes well, that we need to make more progress on AI alignment. In terms of just getting capabilities up, I think, I mean, we're already seeing extremely rapid progress. I think continuous learning would be a significant unlock if that happens, perhaps later this year or next year. I think the sample efficiency seems to still be lagging behind humans. That if you think about a human child who goes through school and maybe university, if you look at the number of words that they have read by the time they become a sort of, say, a fully competent adult, it's a small fraction of the number of words that an LLM has read during its pre-training phase. So we seem to have this ability from a relatively small corpus of data to develop the full panoply of normal human adult capabilities, whereas current AIs seem to need more data to achieve comparable levels of performance. So there might be some unlock there as well if one could figure out sort of how to make these AIs learn more efficiently from a given set of data points. Then of course we have sort of physical manipulation where humans are quite dextrous. Like I think we have a long biological heritage that has endowed us with kind of maybe specialized neural circuitry for motor control. Out-of-distribution generalization, this is kind of when you're able to think reasonably and creatively outside a domain that you have been taught, where maybe humans still have some advantage, whereas the AIs excel in situations where you have a lot of training data doing the same kind of task, but then transferring that to a completely different context, there is some question of how reliable current models are. That's probably related to sample efficiency.

So yeah, there are still some areas where current systems are lacking, but if you look at the trajectory, the pace of progress, it's really quite astounding. And now we're beginning to see AI is feeding back into AI development, actually helping accelerate AI research with the coding assistance being heavily deployed in the leading AI labs. So that creates this feedback loop, right? So now each generation of AI systems that get smarter can also contribute more to the development of the next generation. This is one of the dynamics that can potentially lead to an intelligence explosion.

I don't really see how a stock market crash would solve the alignment problem. It might temporarily reduce investment flows into the buildout of AI data centers by some modest amount, but then presumably the stock market would recover and the flows would resume. So I think there is no substitute to actually solving these technical problems, to get scalable methods for AI alignment. We need better mechanistic interpretability tools to be able to see what's going on inside these AI minds and understand it. Increasingly we will need to, I think, leverage the capabilities of these AI systems to assist in the alignment of AI systems. So we could have sort of weaker AI systems that are more trusted supervising and monitoring more capable AI systems. Already the AI coding tools are of course used by human AI alignment researchers.

I don't agree with that. And I actually haven't heard anybody arguing for that. I think the question is how difficult it is. In principle, there can be very smart minds that are nice and care about humans. I mean, within the human distribution, for what it's worth, we see, you know, some human geniuses are nasty and evil, and some are really nice and pro-social and friendly, and there seems to be no necessary connection between some level of cognitive capability and your motivations, whether you care about other people. You could have any combination, right, of stupid, dumb, smart, nasty. And the same, I think, holds in the space of digital minds that can be created. And I think in fact there would be some superintelligence if we could create it would be a lot nicer than humans. You know, we are kind of the side product of an evolutionary process that didn't even try to make nice humans. It was purely optimizing for the ability to survive and have a lot of children. Often that involves kind of dominating tribal contests and, you know, stabbing your enemies in the back and all kinds of stuff, being jealous. In the context of AI, we have the opportunity to actually deliberately try to select and steer the process that generates these AIs with a view of trying to make them ethical or friendly to humans. So the chances in principle should be excellent that we could do that, but there are a number of ways in which things can go wrong and that one needs to be careful about.

The simulation argument. This was this paper I published back in 2003. Shows that at least one of the following three propositions is true. It doesn't tell us which one, but the three alternatives are: one, that almost all civilizations at our current stage of technological development go extinct before they reach technological maturity. So they sort of fail to technologically mature. That's one possibility. Another is that there is a strong convergence among all technologically mature civilizations that they all lose interest in creating detailed computer simulations of people like their historical predecessors or variations thereof. So they at technological maturity would have these enormously powerful computers. So you could sort of do basic modeling of this. You could have planetary-sized computers built with nanotechnologies, like enormous computational resources, and they might spread through the galaxy building these structures, and yet they wouldn't even use a tiny fraction of all of these resources for that purpose of running ancestry simulations. So that's the second possibility. And then the third is that we are living in a computer simulation built by some advanced civilization. So the simulation argument shows that at least one of these is true. It's an important constraint, I think, about what you can coherently believe about the structure of the world and our place within it.

In the third possibility, our world would have a creator in the sense of the people having built the computer in which we are running would in a literal sense have created us and the world. Now that is consistent with there being a kind of more fundamental creator, like so, like God might have created everything including this advanced civilization that is then building the computer on which we run, like just as, you know, you can, you know, breed a bunch of ants in a glass jar, like in some sense maybe you created those ants by breeding them, but still, you know, your parents created you and maybe the sort of divine creator created the whole thing. So it's logically consistent with that either being the case or not being the case.

No. So, it's not meant to be a metaphor, but in a literal sense, if the third hypothesis is true, then the world around you, the microphone that you're looking at there, you know, your sweater, all of this stuff, exists as information being processed inside a computer that is built by some advanced civilization somewhere. And it has some power supply and it's, you know, generates heat that has to be reflected. And it's different from the idea with which it is sometimes confused that, like, you know, there's like a kind of perspective where you could look at fundamental physics in information processing terms. So rather than thinking about stuff like billiard balls bouncing in, maybe we should think of the basic constituents of the universe as being bits. Now that seems to me more a question of which theoretical framework is most elegant for organizing our basic physical postulates. But whichever way that goes, it's a separate question whether our world was literally brought into existence through a physical process that involved an advanced civilization creating a computer and then using it to implement us.

So if you're a Bayesian and you think in terms of probabilities, right? And they've got to sum to one. So anything that is, if there is something that could be evidence for the simulation hypothesis, then the absence of that is evidence against the simulation hypothesis. Right? So now we can think, are there possible observations we could make that would increase the probability of the simulation hypothesis? Now we have this simulation argument that at least one of these three is true. One thing that would increase the probability of the simulation hypothesis is evidence against one of the two alternatives. So for example, if we get close ourselves to the level of technological maturity that would allow us to create these ancestor simulations, then that would be strong evidence against the first possibility. Like if we get through to technological maturity, then it's almost certainly not the case that virtually every civilization at our current stage of development fails to get there. Similarly with respect to the second one, if at that point where we reach technological maturity, we still remain interested in creating ancestor simulations, that would be strong evidence against the second alternative. And so imagine the situation where we have built, like we've reached technological maturity. We have, you know, created all these massive computers. We are just about to reach in and sort of press the on button that will start generating ancestry simulations. At that point, as we press that button, we would have very strong evidence that we are in a simulation. Now there are other ways, like for example, if a window popped up in front of you saying you are in a simulation, click here for more information, that would also be pretty conclusive evidence. So clearly there are possible observations we could make that would be evidence for the simulation hypothesis. Therefore, the absence of those, anything that makes it more likely that we will go extinct before reaching technological maturity or anything we perceive where we sort of converge towards thinking we should never run ancestor simulations at technological maturity, the absence of any windows popping up, all of that would be probabilistic evidence against the simulation hypothesis.

People criticize religions for all kinds of reasons. No, I think the fact that we are not seeing glitches is some evidence against the simulation hypothesis. It's just very weak evidence because it would be likely that these superintelligences that would be responsible for the vast majority of simulations that would ultimately be run would also have the capability of preventing glitches of a sort that would be detectable by the creatures, the observers inside the simulations. So if they didn't want us to notice any glitches, I think they would be able to prevent us from doing so, or they could even edit out our memories afterwards if necessary.

Simulators would presumably be subject to a lot more constraints than the laws of logic. So maybe God can do anything that is logically consistent. But simulators would be subject to the physical constraints operating at their level of reality. They would have to build this computer. They would have to ensure that it has sufficient memory capacity. You know, they would need to repair it if it breaks down. There would be a limit to the number of these computations they could perform. They would be more limited and they would also be, yeah, subject to all kinds of limitations operating at their level of reality. You know, maybe they have issues with their neighbors, you know, that they need to defend their structures. Maybe they face various natural catastrophes. Like from our point of view inside the simulation, of course, we have a little inkling as to this deeper level of reality. We can, however, recognize that it does seem to increase the space of possible future scenarios for us that there are certain things that at least if you're a sort of have a naturalistic outlook, like suppose somebody who's not religious, they might think, well, after you die, there's really not much hope of an afterlife because we can see the brain literally rots or if it's cremated, all the molecules scatter, so all your memories are erased. If we are in a simulation, I think, for example, an afterlife is a much likelier prospect.

I guess first of all, theology is not just one thing, but there are sort of many different theologies. I guess it's less, I mean, it does allow you to ask new questions, but primarily it allows you to answer certain questions. I think the main reason for being interested in the simulation argument is that it's actually supported by this specific line of reasoning. I mean, we didn't go into it, but that's kind of what makes it interesting to me, rather than it just being like, oh, imagine if, you know, what if this whole thing is a dream? How can we be sure we're not dreaming? Or maybe there's like a demon that is deceiving us. And so these kind of skeptical speculations are old. They go back to, you know, ancient Chinese philosophy and like Descartes was like this long history in philosophy, right, of using this as a sort of in epistemology, the study of knowledge, as a sort of challenge, like prove to me that the external world exists, prove to me that you're not dreaming. And then if you're interested in epistemology, it's a kind of an interesting thing you can philosophize about. But the simulation argument starts from the assumption that everything is as it appears to be. There is this physical world, you know, it has tractors and air ventilation systems and computers and there is technological progress and then thinks through what the consequences of that is, which is that you would eventually reach a stage of technological development where you have the ability to run these vast numbers of ancestor simulations. And then it thinks further what it would mean if the world is such that there are all these ancestor simulations with brains like ours in them being simulated at a sufficient level of detail to be generating conscious experiences. Then in that world, most experiences of the kind that we are now having would exist in these simulations, right? Rather than being implemented by basement-level biological reality. And so then conditional on that being the case, we should think we are probably one of these majority simulated ones rather than the rare exceptional non-simulated ones, given that from the inside you can't tell the difference. And so then you're led to this interesting constraint that at least one of those three alternatives is true, which kind of narrows down the ways the world could be in a surprising way. And that to me is the most interesting thing about the simulation argument. Now, of course, then it does open up a whole bunch of further questions as to the motives of the simulators. What happens after if one simulation terminates? When will it terminate? Are there conditions that would result in it terminating? What more broadly is the distribution of these different kinds of simulations that might include people like us in them? Why do we find ourselves alive right now in 2026? This seems to be a kind of peculiar time given that we are sitting just at the threshold to the development of superintelligence. Most people lived a long time ago. Maybe they were hunter-gatherer farmers in Egypt or something like that, right? And right now we are sitting here talking about the prospect of this like most important thing in human history and we are really close to it.

Let's first address the quantum. I think it wouldn't be necessary to use quantum computers in order to perform these kinds of simulations, although maybe it would help. Now, what do you mean by puppets?

I mean, it might be that our behavior is controlled by our brains. The brains were created perhaps by the simulators, or maybe they didn't even directly create the brains, but they just created the world in which brains arise through natural simulated processes. So there could be different levels of indirection that could be used to achieve a specific outcome.