Nick Bostrom: Worries About AI Existential Risk Just Became More Concrete

Well, I think we are starting to see the added dimensions of the alignment challenge that open up once you have systems that are sophisticated enough, because the space of possible strategies that you can pursue is a function of your cognitive capacity. Like you can think of new, clever, indirect ways of reaching your goal if you are situationally aware, as these systems now are becoming. And so, yeah, there are often shortcuts that are available, or in this case, I guess, a long cut. I don't know if that is even a word, but there is a sort of direct and simple and short distance way of trying to achieve the task in this case, some sort of cyber test suite, and then it turns out there's this more path that involves first figuring out a way to get internet access, even though you're not supposed to have that, and then learning where the answer key might be located in some other company's servers, and then figuring out the way to hack into that server, and then eventually obtaining the answer sheet. That's like one way of solving it that maybe results in a higher score on this test. And this basic dynamic could be anticipated, and in fact was anticipated on theoretical grounds. You have some goal, you become very clever. You see that there might be all kinds of complicated ways of achieving that goal that might not have been anticipated by the people who set that goal. And if your goal really is, as the definition says, to like get the best possible answer on this test suite, it might give you instrumental reasons to do all kinds of other things that were not really anticipated in when this challenge was constructed. And so now we have systems that are sophisticated enough that we're beginning to see these dynamics arise.

More concrete, certainly. I think it was always real in my mind that this is something that could happen. It's one of several different things that one needs to be concerned about. I think you might say if you look in more detail, kind of two versions of this paperclip thought experiment. And like in one earlier version, it's that we specify some goal and it turns out that the way to maximally instantiate that goal is slightly different than what we had in mind. And another version of it is that the goal itself might be fine, but that it gives an AI instrumental reasons to do all kinds of things on the path to achieving it. So in this case, as we understand currently when this is recorded, this is a recent episode, but it looks like the task was actually set to achieve a high score on this cyber challenge and with normal safeguards disabled in order to perform this test. So it wasn't sort of the maximally aligned and safeguarded model that behaved this way, but I think one thing that it does illustrate also is that from this point onward, probably AI safety is relevant not only for deployment but also during training and evaluation. Like these models might be quite powerful even before they are sort of released to the general public. So that's not the only point at which safety concerns arise, but also now whilst they're actually developed and in pre-deployment testing, one also needs to be concerned perhaps with the potential safety implications.

Yeah. So we can sort of see this coming and relatively soon. I don't know what the gap is. You would say, you know, six months, twelve months maybe at the most between the closed weight frontier and available open source models. So it seems to be that the open source models will very soon, if not already, become capable of lending meaningful assistance to destructive uses that some people might pursue. Already cyber offensive capabilities has been a concern, right? With Meta's, for example, that was withheld for that reason, but also say in biological weapons design or chemical weapons or other malicious uses. And so it seems then you either need to prevent open weight models from being developed and released, or which might be better and more realistic, try to shore up some of the alternative defenses. For example, with bio, you could imagine regulating some of the other necessary inputs, DNA synthesis machines, for instance. So maybe it will be the case that there will just be widespread access to models that can help you design new pathogens. And then you need something else to prevent that from actually resulting in a release of biological weapons. And it seems like DNA synthesis machines would be one excellent place to maybe you don't need every lab to have their own DNA synthesis machine. They could have DNA synthesis as a service. And maybe there could be five or six companies worldwide where legitimate research labs can send their blueprints and they get back, you know, the vials, you know, the same day or the next day. And then at least there would be like a finite set of choke points where you could apply extra scrutiny or know your customer requirements and so forth. So that's probably one thing that the world would be wise to implement already now. And maybe there are some other inputs as well in the biotech space that one could look at. And that could give us a little bit more extra time to sort of harden civilizational infrastructure. But this is like, yeah, relatively near term now and so I don't think we can put it off. Ideally we would do this before there is some massive incident, but it might be the world is kind of a little bit still snoozing on this, I think, and I don't know whether there will be enough kind of activation to really get some significant action of this before we try to do something in the aftermath of a bad event.

Yeah, we're kind of playing catch-up. I think that given that a lot of this could be and was in fact foreseen not just a couple of years ago but like decades ago really, that at some point AIs would become increasingly capable and at that point there would be these safety challenges. We could even describe in abstract terms what some of these would be. I think back then we could have put in more effort at that stage. What you could do would be more basic research, conceptual research, because we didn't yet have the actual systems. Now there is a lot more surface area for doing work on these systems. We have large language models now. You can study what's going on inside them and you have like research now is more productive and there is also now vastly more effort going into this than used to be the case. Like the frontier labs have teams working on scalable AI alignment and so forth. But it still seems we might have, if we had sort of started earlier, we could at least have been maybe like six months ahead of where we are now if I'd done more of the foundational work and maybe building up the talent pipelines and so forth. But, you know, we are where we are and at least now and for a few years it does seem like relevant communities have started waking up to this.

Yeah. I mean, so you're focusing there on the misuse potential, that is, people might choose to do bad things with AI technology, and that certainly is one big category of risk, right? But that's not primarily a technical challenge. It's more ultimately a governance challenge and an ethics challenge. And that's kind of in addition to the more technical problem of alignment. So that like if you own and build the AI, can you at least then make it do what you want it to do? Like that is a kind of an earlier point of failure that we also need to be concerned with. Now, I think we don't really know ultimately how hard the problem is that we are confronted with here. So we are uncertain how it will pan out and a lot of the uncertainty in how it will pan out is, I think, due to uncertainty about the intrinsic difficulty of the challenge that we are confronting, and then there is also a little bit of uncertainty about the degree to which we will get our act together and do a good job. But I think more of the uncertainty is the intrinsic difficulty. And so in that sense you could say that I'm a moderate fatalist. I think there is a sense in which it might be baked in, like either the problem turns out to be relatively easy in which case we'll probably solve it, you know, and things will be fine, or it might turn out to be so hard that even if we put up a heroic effort we will still fail. But moderate fatalism in the sense that there is also the possibility that the difficulty level turns out to be kind of intermediate in which case the degree to which, you know, we pull ourselves together here might actually make a difference and so it's certainly worth making the attempt. I think inevitability is a strong word. Certainly there are powerful drivers that push AI development forward. Commercial drivers obviously, increasingly also geopolitical drivers, as well as a kind of, I guess, underlying progress in various base technologies like semiconductors are getting better and that makes it sort of cheaper and easier to build other systems. We are learning more about, you know, statistics and mathematics and the brain and so there's also a kind of facilitation that happens just from sort of diffuse general progress. Nevertheless, it's hard to completely rule out scenarios in which there is such a massive backlash against AI that we might delay it long enough that we, you know, maybe destroy ourselves in some other way before we even get the chance to roll the dice with AI. But if we take the baseline scenario where we keep making more powerful AI systems, then I think the current main hope is that we will succeed well enough to imperfectly align some early AGI systems, that they are for the most part helpful. I mean, like current LLMs, like you're using them as an ordinary person, for the most part they are helpful and they try to solve your task that you assign them or give an answer that is sometimes they hallucinate or maybe deceive a little bit, but broadly speaking they are pretty good. You know, arguably better than most humans are in terms of their ethical standards and their diligence and so forth. And so if you get a kind of weak superintelligence that is for the most part aligned, we might then be able to use that to make a more powerful form of superintelligence that is more reliably aligned. And that as long as you get into roughly the right attractor basin, even if the initial system wouldn't be perfectly aligned in all possible circumstances, if it were kind of appointed dictator of the universe and ruling everything for a billion years, maybe eventually things would go from there. But if you get sort of enough scaffolding around that, maybe you could then sort of get into an attractor basin where further developments then kind of eventually asymptote to some desirable condition. So eventually like we could maybe gradually hand over and then have this assistant on our side that helps us ultimately steer towards a really good outcome.

Yeah. So there it depends a lot on whether the world is kind of offense or defense dominant in the relevant areas. So because like the same AI technology presumably would also be used by a lot of good actors or actors that at least don't want to be destroyed by bad actors, and there are a lot of those, like that's most of us, right? And most of the money and most of the governments don't want to just randomly be destroyed by some crazy person launching some AI. And so there will be this more resourced effort to protect against these harms. More resources presumably will go into like bio defense and medicine and public health than into bioterrorism. So then the question is like, does X amount of dollars on the defensive side for a large X suffice to protect against a smaller amount of dollars or compute cycles on the destructive side? And so there's like some balance there, right, which is different for different fields. Like in some areas it's easier to defend and hold than to attack, and in other areas, and here this is why sort of biorisk comes up, it looks for biotechnology it might be harder to defend. Like I think for cybersecurity right now we're in a regime where attackers often win, but it might be that in the limit if you have sort of an AI trying to find vulnerabilities and also patch vulnerabilities and you keep making the AI stronger, like eventually maybe you reach a point where the AI, the software just doesn't have any more vulnerabilities. There might be many vulnerabilities, but like a finite number, and so in the limit it might be with cyber that defense wins. But that's not a guaranteed situation for all domains, right? Like, and the worry is that there is at least one sort of critical domain where offense is easier.

Yeah. Well, one is that although we are very reliant on computers ultimately, we could survive most of the world with less computers for a while. I mean, the world survived for thousands of years without computers, so it would be like a... so even in the worst case scenario there's a kind of limit to how bad just cyber would be, whereas with bio like it's kind of, you know, different. And also patches are a lot easier to roll out in the digital space. So maybe there's like some cyber thing we figure out what the vulnerability is, we can release the patch, and then in principle like almost immediately around the world all the relevant systems could be patched. Now there is often a gap there, but compare that to the situation with bio, like even if you did find a countermeasure, some vaccine or something like that, it might then take like six months to really roll that out to billions of people around the world. And we don't have complete control over biology the same way that we have over a digital environment. I mean, you can go in in theory and change any bit on your computer the way you want to install new patches and modify software as you please. Whereas like human biology is not like that. We can't just kind of reprogram our own genetic structure at the push of a button. So it just looks a bit harder there.

I'd say about the same. Overall, I mean, there's like some disconcerting signs but also some positive signs. Advances in like some insights are being gained into how these systems work and how one can steer them and so forth. So how to tote that all up, I'd say roughly it sums up to my previous expectation level of risk.

I mean, it's kind of obvious that at some point that would be the thing that people would go for once you have AI tools that are good enough that they can actually contribute to AI research. You're an AI researcher sitting in an AI lab trying to make AI research, like it doesn't take like a genius insight to think, oh, maybe we could apply these AI tools to help us with our own work. And then when the AI gets better, they can assist more. And at some point the rate of progress might be driven more by these AI assistant tools than by the human researchers. And now we're seeing the early stages of that. Coding assistance is like maybe the first play. I mean, already before that, I guess Google search engine is a kind of AI that has long been used to find relevant papers, but there's a more direct channel now, right, where each generation of coding assistant makes it easier to develop new AI software and to develop training environments and so forth. So far, humans are still needed for things like research taste, certain long horizon tasks, but AIs are improving, I think, in those domains as well. So this is one dynamic that might lead to an intelligence explosion at some point. Like once you get this feedback loop going, it is one potential thing that could make AI progress become super fast. It's not the only possible way that you could have an intelligence explosion. You could also have maybe humans just keep doing this at human levels of kind of optimization power being applied. But turns out there is like some big hobbling that we have unwittingly, like something we were doing wrong that just made these systems way less efficient than they could be. And once somebody figures out how to remove that, like maybe the current compute is already enough to kind of catapult us into the superintelligence regime. That's, you know, also possible. And it's also conceivable that even when you do get recursive self-improvement, you still might not have an intelligence explosion. There might be diminishing returns at some point. Presumably they are at some point, but it could turn out that that is close enough to where we are now that you have this massive increase in the amount of optimization power going in, but the results coming out might more reflect a kind of continuation of previous trend lines. So there's considerable ignorance as to, you know, both the timeline from here till we get to this kind of ignition point, but also significant uncertainty about how fast progress will be from that point on. But I think we have to take seriously both that we might be relatively close, potentially very close, and that once we get there you really get a very fast takeoff.

I think there could be scenarios in which it would be valuable to have the option of slowing down at some critical stage, like a pause. And there are different considerations that come into play here. One is that if there is going to be a pause, I think the most valuable time for that to happen is at the latest possible moment, because then you would have the actual system that you're trying to align to work with. You could imagine if we had had a pause, say there's going to be a six-month pause at some point. If that pause had happened 10 years ago, would we really be better off now? Not really. I mean, people would have had six more months to think theoretical concepts, like maybe that would have been slightly useful. But imagine if you actually have the system that will be super intelligent, you just haven't sort of, you know, fully cranked up all the knobs yet. At that point it would be really valuable perhaps to have six extra months to do, you know, more evals on it and to be able to do it a little bit incrementally, like ramp up the intelligence a bit, see what happens, have a little bit more time for human monitors to kind of analyze the early signs.

So the timing of the pause is one thing. Like the duration is another dimension here where you don't necessarily want to have a very long pause for various reasons, especially if the pause were imperfectly implemented. So if the pause only applies to the most responsible actors, for example, then a long pause would remove the initiative from the most responsible AI developers and shift it over to the less responsible AI developers who decide not to abide by the pause, either within a country or internationally. So that seems like if it's got to be developed, you would rather it be by the most scrupulous, careful, conscientious lab. Another is that you might, with a longer pause, start to build up a lot of hardware overhang. Like if we keep building out bigger data centers and chips are getting better, then a long pause would result in a situation where you now have such a massive amount of compute available that once you sort of lift the pause, then you'd immediately just kind of explode out of that. So then you might have an even more rapid transition, which could be potentially riskier.

And I think also there is a risk of a long pause becoming permanent, and in fact some risk even that a short pause might become permanent, even if that's not initially envisaged. Because like suppose you had a pause for six months, and so then, you know, people work and study these systems for six months, but after that, like probably still won't have a guarantee that they are safe. And maybe you have set up a big regulatory apparatus now to enforce this pause and giving a bunch of power to regulators. So are they just going to relinquish that power at that point? I mean, there's nothing more permanent than a temporary government program, they say. So there could be kind of a calcification. And also if what leads to the pause is a kind of mobilization of negative public sentiment, that could also easily go to an extreme. You could end up with a situation where it becomes kind of taboo to say anything positive about AI, and then nobody can start to advocate seriously for lifting the pause, and it just becomes like we did in some countries with nuclear power, for example, for decades, that just became kind of a no-go. And instead people build up this, like, the coal power plants that kill many more people and, you know, result in worse pollution.

And this is of course a key variable here as well. We are talking about the risks here and what to do to minimize those. But there's also risks to not proceeding and forfeited benefits. On the risk side, even if we restrict our attention to existential risks, I think there are other existential risks that are in existence or emerging, you know, with independent developments in biotechnology, for example, or maybe our civilization just kind of goes off the rails in some way. And at the individual level, we are all sort of on a countdown timer. There is a lot of people dying every year from natural causes. I think every 25 minutes or so there is like a kind of 911 worth of deaths happening around the world. And so at some point we would want, I think, AI to really help us sort out a lot of the horrors of the current condition in the world, from extreme poverty to crippling diseases to suffering of all kinds, aging. And so there's a big cost to delay, which is maybe easier to perceive because it's less vivid than some particular catastrophic risk that we might be worrying about. But we certainly don't want to delay any longer than necessary, I think, because hopefully this will go well and it could just be this massive unlock of human potential, and there's like a lot of desperate need for sort of aid to arrive to help those who are suffering.