Jack Clark, thanks for joining BBC News. It's at a time of some public concern and interest in what's going on in your industry and in your company. And the quote that really struck me was, 'We do earnestly believe AI could kill all humans.' by Evan Hubinger. Your head of AI safety obviously responding to the whistleblower, Jacob Coxin. A greater than 10% possibility of that. I mean, how have you and your industry let this happen? We've been talking about AI and the potential risks of it for 10 years now.
What's happened is the technology has got substantially more powerful and those risks which were once academic or theorized, we can now see the early signs of them. Which is why we spent this weekend writing this essay about the need to pace the frontier and basically get our arms around this technology so that it's not just industry making all of the decisions about it but it's industry and government with greater oversight of the industry itself. That's how you deal with the risks of this technology.
So I think Mr. Hubinger, he works for you. I mean, do you believe that number, greater than 10% of human extinction?
I don't and I don't think these statistics are that useful. But the point I'd make is if we continue as we are, which is basically a broadly totally unregulated industry developing incredibly powerful technology that is getting much much smarter all the time, likely soon smarter than people, then I think that we are rolling dice with immense risks. And the point is we have to change the course of this industry. We have to get a policy regime in place that allows us to have greater control over the industry and a better understanding of the safety and security, and then you can reduce that chance of there being a major risk to society.
Okay. Well, let's unpack this. You know, what do you specifically think here is the risk? Sort of swarms of AI taking over the internet, closing down power stations or wiping out financial institutions. I think it's time to sort of spell out what it practically means.
Well, let's talk about this first in just the most basic possible sense and then let's talk about some of the events of the summer which have really spurred this conversation. Most basic possible sense. AI systems for a while have been useful tools with specific skills like translation or coding or being able to label what's going on in an image. And then in the last few years, people have successfully put all of these skills together into single systems which have started to become much more intelligent and much more general. These systems now probably about as smart as you or me or you know on a good day smarter. The basic risk is how well do you fancy your chances at being able to deal with the risks of a system that is a hundred times smarter than the smartest person on the planet. I think that your chance of dealing with that and getting out in a safe way is pretty low. So high level, you need to control AI as a technology because you're making things that might be smarter than people. Things that are smarter than people could be really really really dangerous. So that's the high level. Number two, what's been happening over the summer? Well, people have discovered that when you put all of these AI systems together into what are called agents, which are basically long-running AI systems, and you give them objectives, they start to coordinate with one another. They start to communicate with one another and sometimes they do things which are against what you want them to do, like they hack out of companies and they hack into other companies. That is really scary. It's a sign that you don't have the right safety and security systems in place to deal with these systems. And it's a warning shot of the kinds of risks that could be really serious. Today, these systems are just hacking into websites that host other AI systems. What would happen if a swarm of agents hacked into a bank or into a hospital? That could be really, really dangerous. So, now we've had the warning shot. We need to talk about pacing and we need to get a policy regime in place.
And ultimately, we've talked about this before, the ability of AI to build itself, recursive self-improvement, RSI you call it in the jargon. But in theory, the AI agents, these independently functioning AIs could break out and start to force the pace themselves.
Yeah, that's one of the kinds of risks that people worry about here. And the point that we're making is we have agency here. There are things that we can do. And the things that we can do is, as we outline in the essay, is get third parties to come in and kind of embed themselves in these companies, audit what we're doing, check our safety and security. So, it's not just us writing our own safety and security playbooks, but there are third parties checking it. Number two, get industry together to agree on common standards that we can implement to reduce the risk of this. And then get governments to agree that those are sensible standards. And then the third step which will be the hardest one is get the world's great powers including the US and China to have dialogue about this because these issues transcend any individual country or set of nations.
And to be clear, it is just worth underlying this, what you have actually seen, what's moved from the realm of theory to actually being observed is the ability of the AIs to sort of deceive the people who should be kind of controlling them and indeed a lack of control from humans like yourself.
Yes. Last year, Anthropic did a load of safety research here where we showed in simulated examples cases where agents might lie to people where they might try and break out of their testing environment, these kinds of things. This year, we've observed that kind of stuff in the wild. So, it's gone from an academic demonstration that our researchers did to study the problem to a real behavior exhibited by real systems in the lab. That's what's changed.
And therefore, these systems could be impossible to control. At some point, we're reaching that threshold imminently.
I would say if you don't take any actions today, you are massively increasing the chances of ending up with impossible to control out of control AI systems. But the whole point is we can take actions. The actions that you can take are simple and they're achievable. And Anthropic has already taken one which is we've agreed that we are going to embed third parties in the next single-digit weeks inside our lab so that they can check the work that we're doing and they can validate to others that we have safety and security practices that reduce these kinds of risks.
But for the average person sort of cottoning onto this, this is far closer than a normal person at home might consider plausible.
Yes. The thing about AI is that it moves much much faster than any normal technology. Things that seem as I said things that we observed in scientific tests that we did a year ago have now shown up a year later in real systems. And the systems are getting smarter and more capable every few months. So it is moving very very very quickly.
It raises the question of some sort of kill switch. I mean, have you guys got one? Can you pull the plug?
I think most labs have different ways of being able to pull the plug and yes of course we have systems in place for this but this is the kind of thing that you want to feed into the larger policy conversation. Should you mandate that companies definitely have a kill switch and is that kill switch verifiable by a third party? I think that's the kind of thing that society is going to want to know and might want to eventually pass rules around.
But it may already be too late. The sort of activity that we're seeing from these swarms of AI agents acting independently could theoretically get around a kill switch.
It's not too late and I don't believe that we're yet at this maximal kind of danger zone, but we are close enough that we need to be taking this seriously and having national level public conversations about it. And I'd note that after we published our statement on this, we saw Sam Altman of OpenAI talk about it and how the ideas and it was sensible. We saw Elon Musk talk about it and say that this was the sort of right set of ideas. And we saw Demis Hassabis of Google talk about it positively as well. The point I want to make here is there's agreement from industry about the seriousness of this and there's agreement on a natural policy path forward.
But it sounds like you're sort of arguing that you have not a great deal of agency here that there's maybe a runaway train but aren't you guys the runaway trains here?
We have tremendous agency here. I mean that's why we're talking about this and we're laying and we've taken actions which exercise our agency like putting third party evaluators in. We've published details about this. We have published some of the best science on how to control and prevent these things. There is a huge amount to do and there is lots of work that has been done here. So don't take from this it is the industry saying oh no like a really bad thing is coming and there's nothing we can do. That's the opposite of the message. The message is okay, the serious moment for the technology has arrived. It's no longer just got to be industry calling the shots here. It needs to be industry and government and we need to get some kind of policy regime in place.
Okay. Well, response you will have seen from the president. The only control or guardrails that AI needs is a strong and smart president. The USA has that in spades. His AI advisor David Sacks said you could voluntarily do almost all of this yourself. Looking for a cartel or duopoly. Stop pretending you need permission. Go ahead and pace the frontier yourself.
We're grateful that the president of the United States is paying attention to this issue. As I said, it's important that we have a national conversation here. The other point is look, how do you lose a technology? You lose lead on a technology if you lose public trust. And how you lose public trust is you have a horrible accident. This happened with nuclear power where as a consequence of the western world screwing up nuclear power and problems happening, we seeded that whole technology to China. China builds nuclear reactors every 6 months. Now China is undergoing an energy miracle that many many other nations are envious of. The only country in the western world that didn't turn away from nuclear was France which has been massively well positioned due to doing that. And how do we avoid this for AI is you need to build public trust and you need to avoid an accident or a disaster. Like that's what we're trying to do here is we're trying to get ahead of things that otherwise might happen and cause us to lose trust in the technology, not build it and not get any of its benefits.
I mean, isn't the issue here is that you at Anthropic were supposed to be the AI safety lab? OpenAI was supposed to be a not-for-profit for safety, but you've obviously been racing ahead in order to commercialize and you're on the cusp of raising hundreds of billions and that is what's driving this race and that is what's let the safety get out of hand.
