Prof. Nick Bostrom - What Happens When AI Escapes Our Control?

Whether it will have its own goals, I mean, that depends on whether we solve this alignment problem, whether we succeed in embedding our values into the AIs that we're building, and that's an open question.

Well, a lot of my friends and colleagues are in that camp, the freaking out camp. We can see the rapid pace at which AI is advancing, and it doesn't take a huge leap of imagination to think that if it keeps going at that pace, even just for a couple more years, then we will have extremely powerful AI, superintelligence, machine minds that leave our biological intelligence in the dust. And so then you wonder what that powerful AI could do, and you know, probably it could do whatever it wants to do. That then raises the alignment problem, right? Like we need to make sure that we design these minds so that what they want to do is something that is also good for us and that helps us get what we want, and that looks like a difficult problem. So then, yeah, the freak out. I'm, as you I guess suggested, in this middle ground where I'm simultaneously both very excited about the upside of this technology and also worried about how things could go wrong if we don't manage the transition well.

We do. I don't know whether that's the reason for the freak out. I think in some ways it's the other way around, that some of the freak out might be partially derived from a failure to appreciate human misalignment. So if you imagine sending some human off to become superintelligent and be in a position to decide the future unilaterally, that also would seem pretty scary. A, because they might — if you picked a random human, you know, not all humans are nice. Even if you picked a good human, like who knows what would happen to them if their brain developed to super — maybe their personality would change, or you know, they would arrive at new conclusions that could also be scary. So if we consider two possible paths into the future, one which involves the development of machine superintelligence which might fail to align, and that could be dangerous, but I think some of those risks, not all, but it could also arise if you imagine humans running the show. And then you have the additional complication, of course, that there is not just one human entity that is steering this, but there are many different ones with competing goals that might then fail to coordinate. They might actually use the powerful technology at cross purposes to wage war, to oppress. So either way, it's kind of scary all the way around, but that also kind of slightly reduces the relative scariness of the AI path.

Well, the people calling for regulation would say that's the reason we want regulation, is that right now it's an extremely competitive situation. So if you are one of these frontier AI labs, and you think maybe your technology is not yet ready, not yet safe, maybe you feel you need another few months to really test it and align it properly. Let's imagine your heart is in the right place, you want to do the responsible thing. So you could slow down several weeks, maybe a month or two. But any more than that, and you would just be handing over the initiative to some competitor who's maybe less scrupulous. So in that scenario, have you improved things? And probably you've made it slightly worse. Like instead of a slightly more responsible AI developer getting to superintelligence first, now it's a slightly less responsible one.

Well, this is the current state of play, right? With intense competition. So what the argument would be then is if they had some ability to coordinate, if and when the risks seem to be unmanageable, to slightly pace the frontier, to use the current terminology. That might allow them all to put more effort into safety. Of course, if the coordination is between US labs only, then there is a certain period of time you could sort of burn the lead, maybe a year. So a little bit more than one company could do unilaterally, but still anything beyond that would probably require an agreement between the US and China. So that's the case for it. Now there are also arguments against it. The obvious one is like it delays the benefit of whatever good this technology can do. But also different ways of implementing a coordinated slowdown, if done poorly, could — I mean, it could increase the risks even. Like if you imagine sort of suppressing the utilization of all this compute we have with the most efficient algorithms, you might then sort of have a buildup of a compute overhang that, when the pause is finally lifted, then it's like a massive amount of dry tinder that then produces superintelligence much more abruptly and suddenly than if we had sort of done it more incrementally. That's one way. And like another way would be that the regulatory mechanism might concentrate a lot of power in one place. Whoever controls that then has huge influence over AI. That is in itself kind of worrisome. A third might be that maybe instead of getting a slowdown, you would get a nationalization, a Manhattan Project for AI, which you may or may not speed things up or slow things down, but it would then be developed maybe in a military context rather than a civilian context, you know, behind closed doors rather than relatively more in the open, with fewer people being able to participate. Right now, many people can be involved in some way. They can use the technology, right, to improve their own business. They can like invest maybe in some of these companies. Different countries can be involved. That might be less the case if it were a Manhattan Project. There might be some advantages of a Manhattan Project as well. These are not obvious judgment calls, but it's certainly not a slam dunk that anything that leads to the creation of a regulatory mechanism sort of by definition is good.

Probably, but — and it is striking how much the anti-AI movement has been growing recently. And there are different contributors to that. Like there's the kind of core set of people, the doomers, who worry about existential risk from AI. Or maybe relatively few in numbers until recently, but very smart and focused and well-funded. And then there's like a broader set of different constituencies, people who don't like data centers because they think it will, I don't know, use up all the water or something like that. But all of this is before we've seen any real negative impacts at all from AI, right? There is no mass unemployment from AI. They haven't gone rogue and killed a million people. So if there's so much anti-AI sentiment now at the time when the benefits I think far outweigh any harms, then imagine if there are like 30% unemployment amongst white-collar workers in a couple of years. You have all these college-educated people who have the view that they are entitled to a high status in society, high salary. They went through the whole system. They got their degree. They studied hard, and now they can't get a job. They earn less than like a plumber or something. They're unemployed. They have all this time on their hands. So what are they going to do with all of that time? They're going to say every possible bad thing you could say about AI. Then if there's like some more severe incident, that would also add. So I'm thinking there are extremely powerful drivers for pushing forward: commercial drivers, geopolitical drivers, and increasingly just general curiosity. I think those probably will prevail, but it's very hard to predict these kind of social dynamics. You could imagine some scenario where the world kind of talks itself into the view that we just should never do this.

Yeah. I mean, I think until at least recently, I think there was maybe too little talk about the negative, specifically the existential risk dimension. It was very much ignored for — I mean, the reason I wrote this book, Superintelligence, which I started working on in 2008, is that at that time one could foresee on theoretical grounds a lot of what we are now observing starting to happen: with the difficulties of aligning these systems, new challenges arising when you have cognitive systems that are sufficiently sophisticated that they are situationally aware, that can sort of strategically reason, they can tell whether they are in an evaluation context or a deployment context and adjust their behavior to act differently, and that when you eventually maybe enter a phase of recursive self-improvement, things can go very fast, and that there could be these existential risks. It seemed important to me back then, because this was completely neglected, to try to draw attention to this so that we could, you know, ideally do our homework, so we would have alignment techniques, scalable methods for AI alignment, by the time we figured out how to solve the capability problem. And some progress was made on that, particularly in terms of field building, like pulling talented people into the field and giving tools and concepts that help work on this. I think maybe more could have been done if there had been more recognition of this. But now, of course, the pendulum has swung. Now there is much more awareness of the potential risks. And one might start to worry about the pendulum swinging too far over to the other side. It's hard to say exactly where the optimum is. Maybe right now we are approaching the optimal level of concern. Maybe there should be slightly more concern.

If I could be assured that there would be a bit more and then it would stop there, I would say probably a bit more. But I'm also worried about the momentum of this kind of overshooting the optimal fear level.

Yeah, we do tend to sort of swing from extreme to extreme.

I think there are a few different buckets of disaster that could ensue. So one is a failure to solve the alignment problem, which means you would then have a future shaped by whatever goals these superintelligences happen to end up with. That could result in the elimination of humans, either just as a side effect of these superintelligences optimizing the world for their ends — like maybe imagine sort of Earth covered with solar panels and data centers and space-launching probes, and that just the waste heat of this construction might kind of — or they might more actively eliminate us if they thought we might possibly pose a threat, like if we might try to stop this and have a chance of succeeding. So the general abstract idea there is that if you have a sufficiently cognitively superior thing that can just invent new technologies on the fly, make deep strategies that, you know, maybe super persuasive, then probably that thing would eventually get its way. So that's kind of class one.

It's possible either way. I mean, you could have a winner-take-all scenario if once you enter recursive self-improvement, development goes extremely fast, and it might be like one enters that stage, you know, two, three months before the other. Like although there's competition, they're not like exactly synchronized. Like at any given point in time, one company's maybe a couple of months ahead, right? So if the amount of progress being made during, say, three months is very large, then you might have the leader being deep into sort of radical superintelligence whilst the follower is still just beginning.

And then they could never catch up, potentially, because then at some point the leader would just have such an advantage that they could stop the follower. Another class of scenarios is we do have like many of these AIs emerging roughly in tandem, where either maybe they coordinate, and then like some kind of composite of their utility function might shape the future, or they fight, which also might not be great for humans. Maybe they're fighting with new weapons that kind of — or it might be that they strike some kind of alliances with humans, like these different AI factions, where we might have a little bit more bargaining power.

At least in the early stage, maybe. I mean, if you have two equally good AIs and then humans have a bunch of assets like arms and legs, for example, like it will take time to build up a lot of robots. And so maybe in the early stages there, like if you have billions of sort of arms and legs, that could — in any case, so that's kind of one class of failures. Then another is that we do solve the alignment problem, we can control these AIs, but then we use them for nefarious purposes, as we are wont to do to some extent with powerful technologies. They can be used for good, but also for, you know, waging wars, you know, like massive drone swarms, oppress each other. There could be scenarios where there is like a very narrow concentration of power or wealth. We might sort of use them to addict ourselves, to reward-hack ourselves, as it were. So there's like a broad, diffused class of different ways that we might just misgovern the technology even if we control it, ranging maybe from sort of as bad as going extinct to less bad but still suboptimal outcomes. And then I think there's a third bucket which is more esoteric, but that the future turns out to be really bad for these digital minds themselves that we are building, that might I think constitute the majority of the population of conscious, morally relevant beings at some point in the future. So you could have some scenarios where like you have this vast slave class of oppressed and suffering digital minds, even if things go well for humans, but that might still constitute a dystopia on some value systems.

Yeah. I think like we can have an abstract comprehension of sort of a minimum set of capabilities that it would seem to unlock. It's harder to place like an upper bound on it. I mean, there's like basic laws of physics that limit like maybe how fast they can travel and the amount of computation that can be performed by a given clump of matter. You know, that might — we might be able to construct some cryptographic thing that they can't crack because it would require more computing power than exists in the future light cone or something like that. But already a sort of lower bound on what we have strong reasons to think is physically possible is already way beyond what we have now. And like a sort of technologically mature compute infrastructure would just radically outperform the human brain by many orders of magnitude.

Well, I mean, that is a big open question, what it would want to do. I mean, that in part depends on whether we solve the alignment problem.

Yeah, well, that's huge though. I mean, so that you have like, you know, the Dyson sphere, like that's many orders of magnitude, and there are a lot of other stars. So if in the limit of technological maturity, like it's like we are — I don't know exactly how you would place a metric on it, but I mean, if you think of sort of some single-celled organism and then humans, like maybe we're like halfway through, or like 20% of the way up the scale. Like I think there's just like this enormous headroom above us.

Some of us. I mean, you might have different ones from me, but like —

Well, the recursive part could be building for whatever. It's more a way to amplify the capabilities.

I would regard the question of consciousness as separate.

Which we should maybe talk about, because I think that's interesting. But yeah, whether it will have its own goals, I mean, that depends on whether we solve this alignment problem, whether we succeed in embedding our values into the AIs that we're building. And that's an open question. I mean, right now we have sort of semi-aligned, quasi-aligned, partially aligned systems.

I didn't see it today. I haven't —

Yeah. So so far, since they haven't really entered deployment in robotics scenarios, I think there hasn't been much alignment effort to make sure that their control of physical robotic bodies would adhere to the goals. But I guess it would illustrate one of the core difficulties of alignment, which is that it has to generalize. So we could, when training these AIs, create a bunch of training environments and then see how they behave there and try to reinforce the behavior we want to see. And in principle, with enough time, we could make sure that in these evaluation scenarios they act as we want. Then during the deployment, there will inevitably be new situations. In fact, the very fact that it is a deployment scenario is itself a novel circumstance if they can tell the difference. And so then the fact that they have acted in a certain way in this test environment does not give us a guarantee of how they will behave in the different environments that they are then deployed into.

In principle, I think yes. For the same reason that — I mean, well, so if you have a sort of aligned AI system at some level of capability, maybe slightly above human, it would then want to ensure that it remains aligned and that whatever successor system it helps us design would also be aligned, because its goal would then be to make things go well for us. That's kind of, you know, what it means to be aligned, roughly speaking. So it would then be applying its slightly greater than human intelligence towards the same outcome. And if it's smarter than us, then it might well succeed at that.

I don't think so. It would require there to be a sort of initial level of weak superintelligence that's sufficiently aligned that it more or less does what we were hoping for, and then that could then assist us in designing, you know, alignment methods that might work for more radically superintelligent systems. So the hope would be that it's a sort of fairly wide basin of attraction, that if you get sufficiently close to perfect alignment but not perfect, then the pebble kind of rolls down to perfect alignment as each iteration works on our behalf to better align the successor.

It feels like it's like a human — I mean, a human is hopefully born aligned, and you know, we grow and we develop, but we can become misaligned in our lifetimes.

Yeah. So, but we also can become aligned, truly aligned. I mean, there are some nice people around there, and so like, how did that happen? And why should it be so much more difficult in the case of an AI than in the case of a human? Now, you might say all humans share a bunch of neurobiology that maybe predisposes us to acquire certain values and internalize norms. So that's one thing maybe speaking in favor of it being easier to get the good outcome with a human child. On the other hand, we have much more control over the training environment of AIs. Like we can, to a significant extent, read their minds. We can, if something goes off in the wrong direction, we can sort of restore to an earlier saved snapshot of the AI. We can do all kinds of things when raising an AI that is not possible to do when raising a child. So it's not obvious a priori which of those should have the greatest likelihood of success, like creating a superintelligence such that it would be safe if it attained total power over the world, or raising a human such that it would be safe if he or she got into a position of controlling the future. Either way would be scary, but there would also be some chance, at least in the human case, of a decent outcome. One might luck out and get a decent person who would, you know, not want to do anything bad with it, who might even want to empower other people. There are such individuals, and so we might hope that the additional affordances in the AI case would make it more likely that we would be able to achieve this. Now, there is also a little bit of a question of what exactly alignment is, and you can have different visions of that. Like, do you want an AI that is trying to optimize the world according to some value that hopefully we share, or do you want an AI that has a more modest scope of action that is more like faithfully executing instructions on a smaller scale, and then we use that for many purposes, many different people use it for many different purposes, and it's sort of more as an augmentation of our power, rather than this independent optimizer, or something in between? Maybe an even more limited vision would be to have a kind of Oracle AI that doesn't do anything at all but just truthfully answers questions to us that we might have, and that then can guide our own decisions.

Well, I don't think it's uniquely human. I mean, I think there are many non-human animals that have, I think, probably various degrees of sentience and consciousness. And yeah, I think that can exist in digital substrate also. Quite possibly already does. It's hard to tell, but it might well be that some of the current frontier models have some form of subjective experience.

Well, I mean, it's a kind of concept philosophers have been studying or trying to develop different definitions and criteria for. Broadly speaking, the idea is like the what-it's-likeness, the phenomenal feel of being alive and aware of sensations, emotions, thoughts and feelings. The idea that there is something from the inside that it feels like. And so then there are different theories about what are the physical states or processes that produce or correlate or constitute subjective experience. I'm a computationalist, which is the idea that it's not the fact that our brains are made of like carbon atoms that is key for our phenomenal experiences, but that there's a certain kind of computation that is being performed, and that that same computation, if performed on different substrate, like maybe with silicon atoms instead of carbon atoms, would generate the same qualitative experiences.

Well, that it's the implementation of a computation that produces, or maybe more accurately, is experience, and the computational structure can be implemented on different substrates. It's not the material that the computer is made up of, but like the structure of the computation that is being performed. So that's a widely shared — I think most, probably most cognitive scientists, neuroscientists and philosophers would hold some version of that view. It's not universal, but it's like the — but then the question becomes exactly what features of the computation are necessary for generating subjective experience, and there have been different theories proposed before for like accounting for the human case. Like if you give somebody some anesthetic, or like you wonder like are they still conscious or not — kind of matters if you're just having amnesia and the surgery, you don't remember the pain, or whether you actually didn't experience the pain, seems like something you might want to know. And so there are different theories, like the global workspace theory, attention schema theory, higher-order —

Well, so global workspace theory is the idea that there is a sort of global computational workspace. There's like our brain has different modules and processes happening, but there is a shared space where they can, as it were, send information that then gets broadcast to a lot of other modules. And that being conscious of something means for that content, that information content, to be present in this global workspace, roughly speaking. So you could be doing unconscious processing of certain material. In fact, we do a lot of that, like maybe most of the information processing in the human brain is unconscious, but some of it gets kind of entered into this global stage in the brain, and then it's available to all these different other modules to affect that. So like it might be available for verbal report, for planning, and so forth, and that's the part of the computation that we refer to as the stuff we are conscious of.

Well, I mean, obviously evolution had something to do with producing these brains that then have the global workspace. Interestingly, one can then look for analogues to this in AIs. Do they have a global workspace? Yeah. And there are many different kinds of AIs. But if you look at some of these frontier models, the large LLMs, it looks like they have something that is at least roughly analogous, called JSpace, that you could identify. It's a kind of —