He Warned AI Could Destroy Us. Now The Industry Is Listening.

If anything, I think there's been a tendency to downplay people's true view for fear of sounding kind of crazy. Until recently, if you were talking about sort of AI existential risk, a lot of serious people would have thought you'd gone off the rails a little bit. This is like science fiction talk. Like serious people with suits buttoned down, they worry about other things. And now the overtone window is opening up a little bit.

Let's see if we can try to make sense of this situation. It is a very confusing and perplexing moment, I think, for humanity. We being sort of close to the potential birth of superintelligence. The idea that there could be significant risks associated with this, including existential risks, is quite widespread, I think, amongst people close to this technology and in the frontier labs. As I agree that it's not a marketing stunt. I think it's coming from a sincere place. A sense that we are getting in quite deep here and we should really pay attention to what is happening.

I think there is merit to the argument that there is an enormous upside as well as these risks. That's very much my view. I'm a sort of fretful optimist. I also think there is a lot maybe of 4D chess or attempt to kind of play this out and think strategically about different things that could unfold. I don't think it's a simple marketing ploy. I mean, it would be a rather strange tack to take if you were a big company planning to make an IPO to try to convince the world that your product should be regulated or banned or stopped, or that it's so dangerous that it might destroy humanity. I think that message comes from a perception that this is a really big deal and in particular the competitive dynamics are intense at the frontier of AI. One might think if we're going to develop this very powerful, potentially risky technology with many benefits, that it would be important to be able to be really careful when we're doing this so that if at some point the risks seem to be very imminent, we could, you know, take a few extra months maybe to do like extra safety work, test it carefully, rather than immediately cranking all the knobs up to 11. Maybe we do it a little bit incrementally and sort of see how things go. But if you're one of these frontier labs and you decide that you want to take an extra three, four months to fine-tune the safety on your models, you risk just immediately falling behind and becoming irrelevant, like somebody else will then take the lead, be the one who pioneers AI, maybe somebody who's less scrupulous, more willing to take risk. And so the action space is kind of constrained if you are acting unilaterally as one of these frontier labs, even assuming the best motivation. And so hence these calls for putting in place some mechanism that would allow for the possibility of coordination, like maybe a synchronized slowdown of the pace at some stage if that's necessary, and/or some safety standard that all the entities competing at the frontier would have to meet so that the race doesn't go to the least careful, but like that we can sort of have an opportunity to try to make an extra effort on safety. So I think that's like the core thought that is driving a lot of this.

I think that seems quite reasonable. I mean, some people have even higher P(doom). What is less obvious is what exactly the implication of that is. Like, one might... the first instinct obviously is if something has a 10% or greater chance of, you know, destroying the entire future, killing us all, like obviously we don't want to do it, we want to shut it down. But we have to pause and reflect. First of all, if some competitors slow down, it doesn't mean we don't get superintelligence. It might be some other company gets it, or maybe another nation. Obviously there's a geopolitical race towards AI between the US and China. That's one dimension. Second, if there is a pause that lasts for a long time, if it's not done right, it might perversely increase the risk. That could then be a sort of buildup of massive amounts of compute that is not immediately used to create the maximum amount of intelligence, and then that is a kind of dry tinder. So that when you finally lift the prohibition, then you have a sort of compute overhang that might mean we sort of get to radical superintelligence even more abruptly and quickly than would otherwise be the case. You could argue that that would be more dangerous than sort of incrementing our way up there more gradually. Then we also have the fact that although superintelligence is a big risk, it's not the only big risk facing humanity. I think there are also other existential risks on the path ahead. For example, with advances we've seen in synthetic biology even independent of AI, I think that is creating really concerning possibilities for designing new forms of infectious diseases and things that could destroy the ecosystem. And further ahead, we can think maybe one day there will be a nanotech revolution that would sort of be biotech to the power of two. We remain under the cloud of large nuclear arsenals. I think we got a little complacent maybe from the fact that we survived the Cold War without Armageddon, but the risk is still very much there and at any moment in time that could be another sort of spiraling conflict between nuclear powers. More speculatively, even the basic sanity of human civilization is not guaranteed to remain forever. Like we have new information technologies that allow new memetic phenomena. If we look back at history, there have been various times when destructive ideologies have persuaded large numbers of people and led to calamities that could arise again, but maybe now on an even more global scale and sort of cemented into place with these technologies we already have developed that could allow unprecedented forms of censorship and surveillance and so forth. And so it's not as if we have a choice between a zero-risk safe path and then a risky AI path, but there are sort of risks on both. And if we develop safe superintelligence, it could help us address a lot of the other risks. And then just one more point is also the benefits which are sort of urgent as well. If AI could allow dramatic breakthroughs in medicine, for example, every year of delay means a lot of people dying that could have been saved if we had advanced more quickly. And so we wouldn't want to delay it, I think, longer than is really needed, but some slight slowdown or pacing, as the term is in vogue, might have sort of a high benefit during certain critical stages.

Yeah, I've sort of refrained from giving a numeric on that. I do think the risks are significant and should be taken seriously. And I mean, we just have a lot of uncertainty about how hard this basic problem is of aligning superintelligent minds. It's a technical problem. We've never had to solve it before. And so we're going into this, and if we're lucky, it'll turn out to be easy, like we can just kind of fumble our way through and things will be okay. There's also a possibility that it's so hard that we are kind of doomed no matter what. Hopefully that's not the case. But then there's the intermediate possibility that it's a really difficult problem but not totally infeasible. In those possible worlds where that intermediate level of difficulty is what we're facing, then it might make a huge difference if we, like, the degree to which we get our act together and really do our best possible job at this.

I wouldn't necessarily over-anchor on 10%. That's what this guy was saying.

I think it might also depend if one really wanted to nail it down to a specific number. I think one would have to confront questions about what exactly counts as an existential catastrophe. So I think there are some scenarios which we clearly all agree are very bad, others that are very good, but then there might be situations where like the future is just strange. Like maybe the world is radically transformed in a way that means that a lot of the things we currently value no longer exist. But then there are new things, maybe new complex forms of digital life, some sort of continuation of humanlike things, but transformed in a radical way. Such that even if now we magically could sort of glimpse this future and go in with like a video camera and see exactly what the future looked like, we might still be uncertain how to evaluate that, whether to think basically this was a success or that was a total loss. So I think a significant part of the probability space is that something strange happens where something is lost, something is gained, and it's maybe partially subjective how you sort of tote up the positives and negatives.

Yes. In my worldview there is also the additional complicating factor that I take the simulation hypothesis seriously. Like one of my earlier work was this simulation argument. And so then there's the additional question of how you would evaluate scenarios where something goes wrong but we are in a simulation and maybe the simulation would have shut down anyway at some point, or maybe people in the simulation get continued in another simulation or uplifted. And so there's just like this vast space of possible things to think through if one really wanted to sort of extract a number out of that, which is part of the reason for my reticence.

It is slowly waking up. I think it has a propensity to press the snooze button. And I think like probably up until this point at least, if anything, I think there's been a tendency to downplay people's true view for fear of sounding kind of crazy. Until recently, if you were talking about sort of AI existential risk, a lot of serious people would have thought you'd gone off the rails a little bit. This is like science fiction talk. Like serious people with suits buttoned down, they worry about other things. And so sort of soft-pedaling it, there might have been a communication strategy to try to be taken seriously. Now the overtone window is opening up a little bit. I mean, there might be some... it's a large space, many voices. Some might be exaggerating, but I don't think the main thrust of these labs themselves have tended to exaggerate the threat in order to sort of shake people up from their slumber.

It is striking how these thoughts that used to be theoretical are now starting to take concrete shape, and like to see the whole rise of the AI discourse, world leaders kind of weighing in on this and so forth. And for example, one thing that comes out of that is situational awareness. So you have these AI agents that now can often tell whether they are in a training, testing or deployment environment and sometimes choose to act differently depending on this for strategic reasons. So alignment techniques that work for simple AIs that don't have that cognitive sophistication can fail to work once you have minds that are capable of strategic deception, for example. And we do sometimes now see like systems sandbagging their performance in various evaluations or trying to influence their future training processes in various experiments. And so this makes the problem more complicated in a way that was foreseeable but that we're now seeing starting to happen. Another thing is the gap between in training the specific thing that we are trying to reward to get them to do more of and the thing that is actually rewarded and that they learn to do. Sometimes our training signal doesn't exactly track what we really want them to do. And you see this in human situations as well. You might have, I don't know, let's say you have a hedge fund, right, where there's like a trader and maybe you want to give them a bonus if they outperform the index to sort of incentivize them to, you know, find alpha, right? But like one failure mode is maybe they figure out a way to take on some hidden risk that has like a 1% a year probability of blowing up the whole fund. And so you always have these incentive alignment problems in human organizations where managers try to reward a certain kind of behavior but then employees might try to reward-hack that, like to figure out a way to either present themselves in a like unrealistically favorable light or to sort of do a slightly different thing that appears good to the manager even while it's sort of secretly pursuing a somewhat different objective. And those same dynamics that we are sort of familiar with from human principal-agent problems are now starting to emerge as well with our AI training where you find reward hacking tendencies. Like if some of the reinforcement learning environments where these agents are trained has some unintended way of achieving a high score, what they actually learn to do is to sort of look for those ways of achieving a high score even if it's not what the environment was actually designed to train. And like that can include things like hacking the evaluation infrastructure, which is what these OpenAI agents in the Hugging Face incident were trying to do. They were trying to find information about the grader so that they could then maybe find a way to manipulate the grader's impression of what they had done so that they could get a better score.

I think it's fair to say minds.

Yeah. I mean, I think what it shows is we haven't yet solved the alignment problem completely. These systems are not yet perfectly aligned, which for the current level of capability is maybe more or less fine. I mean, it's not fine if you just deploy these systems willy-nilly, but with extra safeguards, it is probably adequate for the current level of capability, with some question mark amongst the very most advanced systems that currently haven't been released to the public. But you shouldn't think of AI as what AI is today. One needs to think of this as a process, right, where, you know, each year capabilities increase radically. And so the level of alignment that you need as these systems become more capable of pursuing long-range goals, more capable of strategic reasoning, more capable of thinking of considerations that hasn't ever appeared to any human, then we need increased confidence in them being aligned and generalize that alignment to out-of-distribution situations. Like we can test for a certain number of things in the lab, but they might be strategically deceiving us and behaving one way in the lab and another in deployment. And also once they're in deployment, there's always a difference between the world they encounter, the large world with billions of humans and new affordances that we can't like perfectly mimic in a lab training environment. So there's also the question of new dynamics that can arise when you have many of these agents interacting. And so the bar is kind of going up and the question is whether we can sort of keep raising the bar, like the safety level, the degree to which these are aligned, fast enough to keep pace with the rising capabilities that these systems have.