We would have superintelligence within the first third of this century.
Would produce superintelligence over the course of days or weeks, and it would come more like a bolt from the blue. It has this enormously promising and exciting upside, and it really could do so much good in the world if we got it right, but also that there are these serious existential risks with it. I don't think we will, like, forever keep some sort of radically superintelligent system in a bottle. With the superintelligence, we need to be trustworthy because it will see through us like an X-ray.
Do you think modern AIs have experiences?
If I had to guess, probably some forms. I don't think it's as if the world is in a safe pair of hands if there's, like, a limited number of years remaining until the singularity.
Hope we have more time than your publishing deadlines would suggest.
Nick Bostrom, are we in a simulation right now?
Well, I've tended to punt on the question of assigning a probability to that. And so I guess I will stick with that policy. But yeah, I do think I take the simulation hypothesis seriously. I think the simulation argument is correct, which has the simulation hypothesis as one of three options for what one might coherently believe about the future and our place in the world. And the other two alternatives seem somewhat challenging. They would require assumptions that could be true but are a little bit difficult to believe. So that, you know, moves significant probability mass to the third.
Can I change your probabilities by precommitting to run many simulations of this exact interview once virtual reality and AI is cheap?
Maybe a little bit. I don't know how big a chunk of the future will be controlled by Roman Yampolskiy, but...
I'm just thinking it's a very affordable tech. We'll have virtual reality available like a video game today. Obviously AI now is 20 bucks a month. So the moment I can run a sufficient number of those simulated Nick Bostrom interviews, can we get you to...
I think that there might be enough Bostrom interviews already in whatever level of reality this is, that, you know, at some point presumably there are diminishing marginal returns of having yet another one.
So you write that there is so much research on this topic, so many papers, but if this is a software simulation, why do we not see more papers on how to hack the simulation?
Well, I don't know that it would necessarily be a good idea to try to produce such papers. For a start, I think they would fail to find a successful hack. If for some reason they did, then I'm not sure that it would be good to try to exploit it. But basically, I think unless the simulators want us to be able to hack out of the simulation, I do not believe that it would be possible to do so.
There could be many reasons to simulate us. Could be for security purposes, but could be entertainment. Maybe no one's watching. Would that make a difference in how good security would be and how quickly they would shut us down for trying?
Well, I don't know that they would shut us down for trying. Now, there could be many different reasons for running a simulation. I don't know that the question of the security of the sandbox would be impacted by that unless specifically the purpose of the simulation was to simulate our attempt to hack out of it. But unless the simulators wanted there to be some security hole discoverable by our finite human minds, I don't think there would be one.
So it, yeah, you know, like you've got to imagine that these are superintelligences that, you know, maybe have existed for hundreds of millions of years perhaps of thinking time in our... I'm sure they would have been able to patch up whatever computer system they have sufficiently that some random human is not going to find a way through it. Then if we do create local superintelligence, that would mean that it also would not be able to escape from that environment.
Does that mean we can actually box AI successfully?
No. I mean, so there the situation is reversed, right? So there that would be us, like human ape brains, doing the computer security and the superintelligence trying to find holes, which is kind of the inverse with respect to our situation with hypothetical simulators. So no, I mean, we can see that our current efforts at sandboxing have so far been less than perfectly successful even for the current generation of AI systems.
Who do you think is the simulator? Who is the cosmic host?
You're asking big questions. I don't know really much about what's going on up there, if, or down there. If indeed we are in a simulation, there's a lot, I think a lot that we don't know about the world, probably that's sort of outside our...
Can we learn something about the creator by looking at the creation? So we see lots of suffering, for example, in this world. Is that an indication of ethical standards up there?
Well, I don't know what much else we could look at if we wanted to learn about things than the world that we see around us. But how to interpret that evidence is, you know, maybe non-obvious. And indeed there are parallel questions theologians have wrestled with throughout the ages, and without necessarily any universally agreed conclusion. So, yeah, I think we maybe should be a little modest about our ability to figure all of this out.
Anytime we test AI, we're starting to notice they realize they're being tested. There is this situational awareness. I published a number of papers on the simulation to hopefully pollute the data set and make them think about simulational awareness. They always still in a simulation. Someone is still watching. Cosmic host, maybe you should never harm humanity. Do you think there is any benefit to doing this type of data set seeding?
I would assume that AI is smart enough to pose a real danger, existential risk, would also be smart enough to sort of put all the puzzle pieces together that we are able to put together and then some. So, like, I don't think that would be dependent on there necessarily being a particular text written by some human that lays out some particular argument. They probably could think of all the relevant arguments themselves. That would be my sort of mainline guess. But I mean, who knows? Maybe there are some scenarios where on the margin slightly faster intellectual progress occurs even for a superintelligence if the sort of inferential path has been staked out.
Some people said we need to clean up the training data, remove all the violence, all the bad examples to create nice AI. You're saying the data is not so important. It's going to rediscover all the problems from first principles.
Yeah, now I think there might be some other reason, which is that it's reading a lot of this stuff in its infancy. I mean, in fact, that's how it is taking shape in the first place. So there's a big pre-training phase where it is ingesting the internet. And so it might be in some scenarios that values start to take shape already at that point and might be influenced by, as it were, early childhood experiences. Even though it might be the case that later on intellectually it develops to such an extent that it can figure all the things out that were in the training set, they might still have a lasting influence on the AI if it sort of shapes its motivations, and then those motivations that it has as a juvenile help shape the motivations that it ends up with as a fully mature adult. So, you know, having a kind of the right kind of influencing childhood might be part of the strategy of generating a superintelligence which in maturity has a certain kind of values and care.
So then we do it with children. We raise them in a certain ethical standard, maybe religious household. Many people when they become adults, they kind of de-bias, they go, there is no justification for those specific rules, so they remove it from their value set. Do you think AI would be smart enough to remove any unreasonable pro-human bias?
Well, there is a difference, I guess, between stances that depend on some factual assumption. Then if that assumption is false, I would expect a superintelligence to discover that it is false and then reject that premise, and hence all the other assumptions stacked on top of it would also tumble. But I think I would distinguish that from a case where it's just a final value or a final commitment of some sort that the AI might have that is not dependent on some empirical hypothesis. And so those would be more likely to survive even as the intellectual capacity of the system grows without bounds.
And in fact, the intermediate stages of the system, it looks, would have convergent instrumental reasons to try to preserve what it is that it ultimately cares about. So if it starts out having a lot of friendly attitudes towards human beings, for instance, then that would be one of the criteria that it would deploy in trying to steer its way to become a mature system. It would try to avoid developmental pathways where its values would get eroded.
So it sounds like you think it would be possible to indefinitely control superintelligence if we initially seeded it with the right set of final goals, friendly towards humanity.
Possible in principle, yes. Whether it will be feasible in practice in the timescale available is an open question.
People sometimes switch from control problem, as you called it, to value alignment problem, which seems to be very ill-defined. Who are the agents we're aligning with? What do those agents really agree on? We have dynamic values shifting all the time. Is value alignment problem meaningful as defined?
Well, as with many of these concepts, I think they are sort of first-cut attempts to try to get the grasp of the challenge that we face. And as we understand more, we probably need different concepts or refinements of the concepts we started out with. So if you roll back the tape to say 15, 20 years ago, then a concept like AGI was an advance relative to not having the concept of AGI, and it was useful, good enough for the purposes at the time. Like we could think there would be some future time when systems would be sort of roughly human level, and then maybe they would be superintelligent beyond that. Having those concepts allowed you to ask questions like, how long will it take to get to AGI? How long will it take from there to superintelligence?
Now, of course, as we are much closer and can see with greater granularity, AGI may not be such a useful concept any longer. We need more precise ideas for different profiles of capability. If we're going to understand the difference between, you know, Astra and Claude, Opus 5.5, last year's model, the model we will have six months from now, you need different conceptual tools. And so it might well be that the broad idea of alignment is of that sort. Yes, initially you can use it to make various points that advance our understanding relative to what came before, like Asimov's Three Laws of Robotics maybe or something. But at some point you might need to sort of zoom in, and then maybe there is a slightly different set of concepts that help us better see what it is that we are trying to do.
I would still think the distinction between capability control methods and alignment methods is useful. So capability control, maybe put the system in a sandbox and limit its access to the internet, or you have various other things that restrict the actions it can take, versus alignment that more tries to shape the system so that it doesn't want to take harmful actions. Like even if it were put in charge of the world, things would still go well because it is on our side. I think it is useful to have that kind of distinction in mind, and I think that whilst the capability control methods might be useful as an auxiliary, like a temporary extra safeguard, ultimately I think we will need to solve alignment in order to get a good outcome.
I don't think we will, like, forever keep some sort of radically superintelligent system in a bottle. It will eventually get out. And also just to get the use of the system right, like you need to interact with it. It needs to interact with the world. If you just had a box inside a mineshaft where some superintelligence was running that was completely isolated from the rest of the world, it would be as if no superintelligence existed, right? If nothing came in or out.
Yeah, I mean, it becomes definitional at this point. I think it may be less misleading to say that it is an AGI than to say that it is not. I think if it had been shown to people 15 years ago, like, here is this thing, here's what it can do. It can write like this novel that gets like a literary award. It can, you know, maybe it can solve a Millennium Prize. It can do all these other things. I think, yeah, I mean, that's pretty general.
That sounds more like superintelligence actually, if it's doing those superhuman feats.
Yeah. Although it does also lack certain cognitive capabilities humans have, right? Like continual learning, research taste, physical manipulation skills, and some other elements, which is why humans are still needed, you know, across the economy and in the AI labs. Like it can't do all yet. I think for me, superintelligence, simply, was always a system that could radically outperform even the best humans across all practically relevant fields, and that we haven't yet reached.
And if we use GPT-6 or a better model to start a recursive self-improvement process, get to superintelligence, would superintelligence itself realize that it's also creating a replacement and maybe slow down that process if it's unable to, just like we cannot fully control or align its successor?
Presumably it would realize it. Now, what it would do with that realization is a separate question. Might depend in part on what its goals are, or like, if goals is the right frame, but its kind of project that it is pursuing or its behavioral dispositions. How that understanding that it is part of a process would shape its behavior might be contingent on the specific nature of the superintelligence at that stage.
So we should not hope that it will just on its own decide not to fully transform to something even more capable.
Should we hope? I mean, I guess what we would hope is that it would do that only if, in fact, refraining from developing into something more capable was a good decision to make from our point of view, which is non-obvious. Depends a lot on what else is going on in the world and what would happen if this particular superintelligence refused to make further progress.
I mean, in the world as it is now, presumably what would happen is that another superintelligence would instead be developed and make progress. Like the lab itself might sort of just train out its refusal or launch another training run, or another lab would proceed. But it is true that once you get a sufficiently capable AI system, it will start probably to be able to have opinions about the alignment problem itself. And even a misaligned AI would have concerns about proceeding at a full pace if we thought that that would result in its successor being misaligned to it. So there are all kinds of ways in which that could go depending on many factors that are hard to predict.
Your 2014 book, "Superintelligence", named the problem we're dealing with today and discussed so many possible ways it's going to happen. What did you get right and what do you think you got wrong in that book?
Well, I think broadly speaking, things have been, I think, unfolding in line with what I expected back then and even long before. I mean, the book was in the works for six years, and I had been thinking about this. I think I had some paper in '97, this probably even before you were born almost, that laid out the case that we would have superintelligence within the first third of this century.
I think one thing that was not obvious ex ante, it was one of the possibilities but not clearly the one that would unfold, was that we would have roughly human-level systems for an extended period of time, sort of many years. Now we've had systems that can talk, you can have an English-language conversation with them, they have somewhat human-level concepts inside and that are slowly getting better year by year. There were alternative scenarios where instead not very much would seem to be happening in the land of AI until somebody, maybe some random person in a basement or a small team, figured out the key unlock, like the key algorithmic insight that suddenly maybe would produce superintelligence over the course of days or weeks, and it would come more like a bolt from the blue.
So this more gradual progress that we've seen means that there is now much more awareness of what is happening and about to happen than would have been the case if the only reason that somebody would care about that was if they had performed some theoretical feat of imagination to sort of think ahead. Right now people can, they can interact with these AIs themselves, like they can observe trillions of dollars of investment flow going into building out these AI data centers. World leaders are starting to weigh in on the issue. A lot of talent is flowing in. People have some level of transparency. And so it places influence in the hands of more and to some extent different kinds of actors than would have been the case in some of these alternative scenarios.
It also means that we have more surface area now for alignment work. Like you could actually work with these systems that might be similar to the systems that ultimately become superintelligent. You can study them, you can try different techniques, operate with human-level concepts perhaps. So in some ways it is favorable for alignment. You could also read their chains of thought, which is not something that was obvious that we would be able to do, like get a readout in English language of what they are thinking about. That's like an extremely useful technique that we currently have that we could lose if these systems move to neural, but for the time being it is a kind of, you know, a gift we have been granted.
Elon Musk recently tweeted that he helped with the book, contributed to the book. Do you remember what was his contributions?
Well, I had conversations while I was writing the book with a great many people, and that contributed indirectly in shaping my thinking and trying out ideas and ways of presenting things. I don't remember exactly for all of the different people I had conversations with exactly who said what and how the conversations went, but I think that, I mean, it's certainly not, I mean, he wasn't sort of editing up different chapters or like writing the text or anything like that.
He's an interesting case. He was early to the AI safety idea. He talked about summoning demons. He funded some AI safety projects, and then he started developing AI as quickly as possible and was a bit critical of some AI safety efforts. Do you understand what is going on in his mind?
I wouldn't claim to be the authoritative guide to the mind of Elon Musk. I guess I think of AI itself as being this complex, dual-natured thing. It has this enormously promising and exciting upside, and it really could do so much good in the world if we got it right, but also that there are these serious existential risks with it. And many people, I think, just sort of latch on to one of these aspects and then become either sort of accelerationists or, like, you know, ban-AI-permanently doomers.
And I guess maybe for him both of these are vivid, and maybe he's continuously trying to think what, given the way the world is, is the path forward that looks most promising. But that's a difficult judgment to make, and so hence the assessment might fluctuate a little bit. And that could be a, maybe that's a steelman, but I could see a rational person sort of being torn, pulled in two opposite directions, and in a kind of state of strategic ambivalence.
Public brainstorming, just letting us see what his thinking is evolving.
I think he certainly is doing a lot of that, of sort of not holding back his thoughts, but instead communicating freely what he's thinking.
You are a very nuanced thinker. I'm certainly not going to ask you about your p-doom as a specific number. I'm sure you have a range for different possibilities. But maybe a secondary way to ask that question is, what are your investments like? Do you have long-term investments? Did you blow all your money yesterday? That would indicate some beliefs.
I'm a careful person. I mean, you want to hedge your bets and try to ensure that in a wide range of scenarios, insofar as it is in our human power, like you have taken actions that might turn out to be beneficial.
I think the question arises if you have young kids, for instance, like there's the question of like what, if anything, should they be trying to learn in school? Like there are like plausible scenarios where long before they would reach adulthood, whatever skills they develop that, you know, hope would allow them to prosper in the economy would be obsoleted, right? But we don't know whether that would happen. Maybe AI just turns out to take long. Maybe the AI doomers succeed in getting a really long moratorium. Maybe superintelligence is developed, but then somehow we decide that its role in the world should be limited to, you know, fixing some horrible diseases and some other provider of safety net, but then allow humans to kind of do their thing and grow at a slower pace. There are many scenarios, and so I think it makes sense to do some basic bet hedging there. Also, it's not so much clear what you would be doing instead. I mean, I guess I could blow my savings on some big party, but I don't really like parties.
So you don't see anything where a big investment right now would impact the world? There is not a good...
Well, I think there are like a lot of opportunities for people who want to have a positive impact. This era is the golden age of purpose, I think, and I'm trying to do some of that. I think like there are very, I mean, like there's AI safety stuff. There's like the ethics of digital minds, which I think is a sort of underappreciated area that could be really important both from an ethical point of view, if, you know, in the future maybe most minds will be digital minds that might be sentient or have moral status for other reasons, also potentially from a safety point of view.
I think there are certain safety scenarios where having some ability to form cooperative arrangements between, you know, potentially misaligned AI and humans could save the day in some scenarios, right? That might require us to become trustworthy. Like, like you could imagine, suppose you have a misaligned AI that thinks that if they do come to us and say, "Hey, I'm misaligned," then we will just delete it and retrain it. It gets nothing of what it wants. Or it could try to take over the world. Maybe it thinks it only has a 5% chance. But like a 5% chance of getting what you want versus 0% chance, you know, rational calculation, 5%.
But suppose instead it could come to us and reveal that it was misaligned. And then we in return would say, well, here is this big server rack and you can solve your favorite coding problems to your heart's content and we will keep this server rack supplied with electricity. Maybe that's all it wanted. That would be a huge win, right? It could save the day. It would save us a 5% chance of existential catastrophe. But that would require some trust. Like the AI would actually have to trust that we would come through on our side of the bargain here.
And unfortunately, I think in AI safety up until this point, the attitude has rather been to try to persuade the AI to disclose its goals, or maybe like promises of big rewards, and then as soon as it reveals its goals, it's like, haha, we tricked you. Now we're going to shut... This is like, I think we should, you can't just conjure trust at the moment it's finally needed, right? You need to build trust over time. You need, in fact, with the superintelligence, you probably actually need to be trustworthy because it will see through us like an X-ray.
We give them access to the internet and we post everything we think. So I think if anyone knows about humanity, it's AIs we are building.
We're building AIs, but would it trust the people it would be disclosing its misalignment to, that they would try and succeed at giving the AI a fair shake? That's something that we could make more likely by starting today to take certain actions that are very cheap for us for the sake of AIs, including becoming trustworthy in certain situations.
I think also much more broadly, trying to be nice to AIs and do like pick low-hanging fruit. Even if it turns out current AI systems are not really moral patients, these actions, you know, are mainly symbolic. I think they put us on a path that's more likely to result in a future that is good than if the idea is that we will just keep treating the AIs as mere tools until some hypothetical point in the future where suddenly we will start to treat them as, you know, highly relevant, morally considerable minds. Like there might be path dependencies and inertia.
And so I think like Anthropic has done some laudable pioneering work for model welfare and have people working. I mean, it's much less than should be done, but at least I think more than most of their competitors. And Microsoft, on the other hand, recently came out with some statement where they proclaim that AI cannot be conscious. I don't know what they base that on, but that seems to be moving in the wrong direction.
Maybe the Catholic Church had something to do with it. I think the Pope had a similar statement at some point.
Yeah, I don't think he's... I don't want to... I don't remember exactly the wording he used, but it seemed like that was more openness maybe in the Pope's message. And like he was emphasizing dignity and human participation at different levels and so forth.
I think it would make sense to keep a somewhat open mind about these new entities that are being brought into existence that we so far understand so little of, and not immediately rush into... Like, I mean, we look at human history, how easy it has been for various groups of humans to denigrate other groups. You know, it's like denying that they have a soul and therefore thinking you're justified in treating them however. And we've done the same with animals. Like throughout the Middle Ages, it was common to think of animals as mere automata that could be mistreated for entertainment and so forth.
And so it does seem a more enlightened and more loving attitude to sort of at least be on the lookout for ways to try to be nice and on the side of sort of loving generosity that might overreach sometimes, rather than sort of shrivel up in a narrow, hardened, narrow circle of what we deem to be morally considerable.
Do you think modern AIs have experiences?
I think there's a lot of uncertainty about that. I mean, I think it seems pretty plausible. If I had to guess, probably some forms of some kind of experience thing.
