Why Speeding Up AI Could Save Lives | Nick Bostrom

If we do develop this safely, then human life expectancy could go up hugely. If you imagine people having the same mortality rate as like a 20-year-old, our life expectancy would maybe be, you know, 1,200, 400 years or so. On the other hand, if the risk of AI goes down too slow, then you also don't want to wait because like we are just dying in the meantime and you'll lose more expected life years from waiting than you gain from having the risk. So, so it requires a sort of a rightly balanced intermediate case to get sort of longer wait periods of many years.

I'm happy to be here.

Uh, it's a, it's a strange time we're living in. Um, we'll see if it catches on. Traditionally, AI has been the larger, the umbrella term, right? You could have a little simple expert system from the 1970s, that would be AI. It's kind of a stretch to use the word superintelligence for all of that. Uh, in, in my terminology, superintelligence has been this thing farther down the road where you really have machines that radically outperform us in all cognitive domains.

But, um, I mean, you could, I guess, look at some of the current achievements, you know, solving all these math problems and so forth. And, uh, I mean, it's not completely ridiculous to start calling at least some of these, uh, performances superhuman and the thing superintelligent.

Well, it depends on relative to what expectation stage. So I think if you were looking at the future of AI 15 or 20 years ago, it was not obvious that you would have an extended period of time when we would have systems that could speak, like just talk to humans in ordinary English without sort of immediately after like just becoming radically superintelligent and leaving us in the dust everywhere. So, um, so the sort of emergence of, of language as a kind of natural affordance of these systems was, I think, somewhat surprising. Um, I mean, more recently, I don't know how surprising it was. I mean, coding and mathematics are very impressive, but it is also areas where you do have objective feedback that you can gather quickly and at scale. So, um, I guess it's less surprising but still striking.

Yeah, it'll be interesting to see how it goes because when the sort of pre-training phase was the dominant chunk of training, they did absorb a lot of human psychology just from reading all this stuff that we have written and they had these like kind of almost human foibles.

That was an early stage where if you, if you gave the AI a little pep talk like, 'This question is really important. Please try your utmost. I might lose my job if, if I don't answer this question.' Like they would actually perform better. Um, now, from 20 years ago or something, that would have been like an insane prediction, right? Like it's not the way computers work that you need to give them a pep talk to get them to perform. Um, but as you said, now with the increasing dominance of the sort of reinforcement learning post-training, we do seem to get more of the sort of goal-monomaniacal, goal-focused optimization style behavior again, and I think it depends where we go from here. It does look like this more goal-seeking in the reinforcement learning sense dynamics put a big strain on alignment efforts, and so given now that there is increasing awareness of the problem of alignment, you know, maybe that will lead labs to develop techniques that kind of counteract that a little bit. Um, so it'll be interesting to see how that shakes out.

Well, it's a combination. I think there are sort of limited versions of instrumental convergence that we can observe there. The idea of sort of even longer planning horizons and, and more situational awareness that, that think, you know, like also in terms of humans more explicitly, not just these automated graders, and like think if we really want to make sure we, we achieve our goal here, we also need to think about what, how are humans going to react to different things that we do. Um, so it, it comes in degrees, but I do think you start to see these kind of instrumental reasons for, you know, preventing premature shut off, gaining more resources, getting more intelligence, getting more power in some experimental settings, also goal guarding, like, like you can see the AIs in some experimental setup reason about how they might want to respond so as not to become misaligned with further training effort. Um, that there was like some studies on this from a year ago or so where you could see these dynamics start to kick in. Um, so I think it's a kind of very natural consequence if you, if you become smarter and smarter, you realize more and more considerations, and if you have some goal, then you will understand that there is a bunch of means that are very useful for a wide range of goals, and you will then have an incentive to try to pursue those means. Um, and so I, I think it wouldn't at all be surprising if, if we do, if we do observe that.

Yeah. I mean, I guess the optimal is to let them loose on the internet, but then maybe like monitoring them really closely as they go along so that you would be able to sort of stop things if and when, um, it starts to create damage. Um, that, that does trade off. I mean, you do get more reliable signal like by deploying in a real environment, but it also means you lose one safeguard, like keeping them in a sandbox, like give you one extra layer of defense. Um, so I think, yeah, it depends. I think, I mean, probably now labs will start to do some in-earlier testing, like, and during training and early testing, like really trying to tighten up their sandboxes. And like, I think this is one of the lessons from the Hugging Face is that safety needs to start not just when you're about to deploy it and you like checking that it's safe to deploy, but throughout the training process itself. That could be a risk of misalignment.

And in some senses is even more like, like the most kind of obvious target for a misaligned AI to, to want to interfere with would be inside the AI company, right? Like that's, that's where the juice is. That's where the next generation of model is being trained. That's where all of these monitoring systems. That's like a big center of power. Uh, if, if you were a misaligned AI, so that's kind of the, maybe the most concerning place.

Um, well, I mean, this is like an open research area. I think that once you have a sufficiently sophisticated system already in existence that is misaligned, it's very hard to rule out scheming. You can try with mechanistic interpretability techniques to read what's going on in its mind, but if the system is sufficiently sophisticated, it's hard to know that it hasn't figured out the way to reorganize its thinking so that it looks innocuous while still preserving the kind of pursuit of the misaligned goal. So ideally you would start at an earlier stage. So these systems don't start out superintelligent like during the training process. Initially it's just a random weight matrix and then gradually capabilities emerge. So it might be that if you sort of, um, monitor that close enough, you might hope that if they're going to scheme, that would first be a stage where they'd start scheming but are not yet competent enough to do it in a way that conceals the fact that they are scheming effectively. And only when they become a little bit smarter will they also learn to scheme in a way that conceals the fact that they are scheming. So if you're monitoring them throughout this developmental trajectory, you might be able to pick up that earlier stage of the sort of naive, that the naive schemer that, uh, that does it in a clumsy enough way that we can detect it.

Um, yeah, well, I should say by way of framing, so this paper only tries to look at one aspect of the overall question in front of us. So I think, uh, there are many considerations that would go into, uh, an opinion about whether we should be moving faster, slower, or pause or stop. Uh, this paper only looks at the question what would maximize life expectancy for currently existing, uh, individuals. Um, but you might also, for example, care about future individuals and other things. But like if we just zoom in on that sub-question, um, um, then it, it does look, if you think as, as I am inclined to do, that, uh, you like fully developed, I, I'm awkward with this terminology now, like I would say superintelligence, but now I don't know, I have to say super-duper intelligence maybe. Yeah. So that super-duper intelligence would be able to make great, uh, strides in biomedicine and, and solve aging and so forth. So that if we do develop this, um, safely, then human life expectancy could go up hugely. If you imagine people having the same mortality rate as like a 20-year-old, our life expectancy would maybe be, you know, 1,200, 400 years or so. Um, and you know, potentially you could go far beyond that if you imagine uploading with backup copies. But like if, if we just use that as a sort of conservative lower bound.

Um, and, and then you can sort of roughly calculate like, uh, how, how many humans are dying per year and, and then like you would then in order to conclude that we should be slowing down or pausing, you would want the risk from AI, uh, to be falling rapidly enough that the, the, the rate of risk decline kind of is, is greater than the annual death, uh, rate. Um, and so I think this could well be the case especially for shorter periods of delay. Like if you imagine, you know, we, like somebody figures out the key missing breakthrough here. Okay, now we have, we know how to make these superintelligent, like, uh, maybe it would be really valuable to have a few extra weeks there just to sort of test everything out properly, right? Like crank things up a little bit gradually or a few extra months. I think, uh, during that initial stage of delay that would be like a large reduction of risk. Um, now if you start to think about longer pauses then it gets a little bit awkward, right? Because, um, either, uh, the, the, the risk goes down fast and then you don't need to wait for many years because the risk goes down so fast that if you just wait for a few months it's already very low and negligible and we should go ahead. On the other hand, if the risk of AI goes down too slow, then you also don't want to wait because like we are just dying in the meantime and you lose more expected life years from, from waiting than you gain from having the risk. So, so it requires a sort of a rightly balanced intermediate case to get sort of longer wait periods of, of many years.

So I, I have this metaphor. I think of it as somebody with a, a terminal condition like congestive heart failure or something who could decide to go in for an operation and if they do that their death risk will spike. Like it's more likely that you will die tomorrow if you have an operation tomorrow. Uh, but then if you make it through maybe you, you cured the condition and you can live a normal lifespan. Otherwise maybe you, you probably will survive for several more months and, and then die after half a year from, from the condition. And so, um, at some point maybe you want to delay a little bit if you think you will get a better surgeon if you wait a couple of weeks or something like that.

Um, but you don't want to delay it so long that you're already going to be dead by the time the more optimal conditions arrive.

Yeah. So, uh, I would call it, um, differential technological development back, uh, in I think like 10, 15 years ago or something like that. And so I think it can apply, uh, quite generally. Like in biotechnology, you know, you might want sort of the, the vaccines and prophylactics and diagnostics to come before the, the bioweapons even if the technology itself will ultimately unlock all of those. Right? Thinking about the sequence of different capabilities, um, is I think often a more realistic approach than thinking which technologies are good and bad and then let's try to never develop the bad ones. Um, so with respect to AI, I think, yeah, there are different, I mean, so obviously broadly AI safety and alignment and sound governance and stuff like that seem to protect like relative to sheer capability gains, but you can zoom in even more narrowly. And so within things that might accelerate AI capability, there might be certain ways of doing it that are relatively riskier than others. So, um, like moving to looped transformers or architectures that allow for sort of more serial depth computation between each verbal token is emitted would be one way of increasing capabilities possibly that would then reduce our ability to monitor what they're thinking by reading their chain of thought. That seems maybe more problematic than like say achieving a similar level of capability scale up by just increasing the number of parameters, let's say. Um, and so, um, yeah, I think, I think that is a useful lens to apply and, uh, if, if there are things we can do to sort of, you know, frontload advances in, in medicine or other desirable applications, frontload the use of AI capabilities for learning more about alignment, for interpretability work, um, that, you know, for hardening our cyber, uh, infrastructure and so forth, that, that all seems, uh, for the good.

Um, yeah, I'm sympathetic to this idea of pacing the frontier, that is developing the capability now so that, uh, if and when it really starts to seem alarming, we would have the option of, um, going slower for some period of time. Um, I think that would be, uh, net good. I am a little less excited about the rhetoric of like stopping AI or pausing AI. I think, I think theoretically a right, correctly implemented pause at the right time done in exactly the right way could be very valuable. But it also leaves open the question, how long should the pause be? Like things sometimes they say there's nothing more permanent than a temporary government, uh, program. Uh, and if you set up this apparatus to sort of control AI and then presumably to get this happen in the first place, you have to really, um, create huge antagonism against AI. Um, and so if the sentiment suddenly switches and AI becomes this taboo thing that you're not allowed to say anything positive about or you get cancelled or like you could imagine things getting locked in. Um, I, I don't expect that to happen, but I, I don't think we can rule it out either if we just watch the trajectory of how much the sort of anti-AI faction has been growing. And this is all before we've actually seen anything bad happening, right? Uh, before there's any big labor market impact, before any like catastrophe has happened. So if, if something like that also were to be added to the mix on top of these existential risk worries, then, you know, we can't rule out. So, um, and then there are other concerns with a badly implemented pause as well, like it could lead to, uh, increased concentration of power risks if like people in general are not allowed to do this. You know, maybe it, it, it just shifts the whole development from the civilian sector into, um, you know, kind of Manhattan Project. Like if this is a really powerful and dangerous technology, like one natural way would be to stop it in the civilian sector and then push ahead hard, uh, in a sort of secret government program, which might have some advantages in terms of information security, but it also has some disadvantages in like it's kind of, you know, maybe concentrates more power in fewer stakeholders, make it less transparent, it's less kind of continuous and integrated, more, less opportunity for people to participate, uh, in different ways by using the technology or being investors and so forth. Um, so it gets hugely complicated. So, so basically the bottom line is that I don't think I have now a sort of fixed conviction for this is the way things should go from now to superintelligence, but rather I think we need to sort of feel our way through as, as information comes to light, as the political landscape shifts, and then like see what kind of makes sense and try to sort of, uh, navigate these, these waters, uh, like one, one row, one oar stroke at a time.

Proverb, something like that. Yeah.