Many people in these labs are seriously concerned about the rapid strides being made and whether AI safety will be able to keep pace with AI capability. We develop increasingly powerful forms of superintelligence that eventually outthink us easily. They can make new technologies very easily. They can maybe be superhuman at persuasion. I think it might now make sense to start to worry about kind of risk discourse overshooting its mark. They might concentrate power. They might be run by incompetent bureaucrats. They might be hijacked by industry interests or lobbyists. There are all kinds of ways in which creating such powerful mechanisms to control this themselves could be abused. I would say strap on your seat belt and hopefully enjoy the ride.
Hello everybody and welcome back to UnHerd. How afraid should we be of AI? It's the big question of the month, perhaps of the decade. It's very hard to know who to trust on this one because some of the voices who feel and appear most afraid of the dangers of AI are somehow involved in the industry. Are they advancing their own agenda in some way? Are they trying to increase the sense that they are all important, hold the future of the planet in their hands, and therefore we should invest more money in them? Or perhaps are they trying to increase the regulatory control they have, slow down things in order that they can increase their own particular company's dominance? There are so many theories and counter-theories as to how far we should really trust these dooming prophecies that come from AI higher-ups. The trouble with finding a guest to really talk through this topic with is that everyone is somehow involved. Either they work for President Trump, or they work for one of the models, or they're taking advice or money. So, you're very quickly drawn into a sort of conspiratorial web of not trusting anybody. I think the guest we have found today is the perfect person to give us a bit of a more detached view. And by the end of it, I hope we'll get a sense of how real or not those dangers are. But before I bring our guest on, I just want to recap for anyone who hasn't been paying that much attention what has happened in the past couple of weeks and what this whole AI scare is really about. So on the 9th of September, just a couple of weeks ago, someone called Jacob Coxon resigned from Anthropic, which is one of the leading so-called frontier AI companies. He was a safety researcher, not a very senior one. And his resignation, citing a fear that there was a 10% chance that AI could bring an end to all humanity, went massively viral, was hugely amplified by lots of powerful people, and became the topic of the week. Someone called Evan Hubinger, who himself actually works at Anthropic, was working in something called alignment, which is about the different AI companies trying to align around safety values, then endorsed the claims, which added to the sense that these were genuine whistleblowers from inside the industry worried about safety. He said, 'Jacob is correct here. We really do earnestly believe AI could kill all humans. I personally think it is above 10% within the next decade.' This is someone who works in AI and now continues to work at Anthropic at the time of publication. So all sorts of people have chimed in in the weeks since. Tech CEOs themselves joined together to call for a slowdown in development. Led by Anthropic CEO Dario Amodei, he wrote a long essay trying to make the case that something like coordinated safety processes need to happen. This was then endorsed both by Sam Altman, who is the CEO of OpenAI, and Elon Musk, who has his own AI model with Grok. Then the following week, you start hearing these notes of skepticism. One of the All-In hosts, Chamath, wrote, 'We must slow down the pace at which we improve the capabilities of AI.' He wrote there in quotes, 'Dario makes the case to stop open source and concentrate enormous technological and economic power within Anthropic.' So they're inside the industry beginning to accuse each other of bad faith. Pretty soon, as usual, the politicians got involved. Donald Trump taking a very pro-AI view. He dismissed the AI threats. He says, 'I am the hoax buster.' This was 14th of September. 'And right now I'm breaking another hoax that AI is going to take over, consume, and destroy the world. The only control or guardrails that AI needs is,' you're going to guess, 'a strong and smart high-IQ president, and the USA has that in spades.' So he's not interested in this self-regulation. Nor, by the way, is his vice president JD Vance, who himself is something of a tech bro, having been initially funded by Peter Thiel of Palantir. JD Vance says there are risks to AI, there are benefits, but to him, quote, 'It feels a little bit like a Trojan horse, these attempts to self-regulate.' So, he's skeptical. Meanwhile, Bernie Sanders and Steve Bannon — these are not people you would normally put in the same political bracket. One is properly of the left and one is about as right-wing as you can find — jointly appeared at a DC event to talk about the need for legislative controls of AI. So they agree in a kind of anti-tech mode, the political horseshoe if you will, coming together against AI there. And then of course, inevitably, there was then the backlash to the backlash, and people senior in the industry started agreeing with the president that this whole fear had been overblown. So for example, Nvidia boss, perhaps one of the biggest figures in global AI, Jensen Huang, he came out and said there is a zero chance that AI destroys the world by 2030. He said that in an interview with CBS News. So there's been this very intense ping-pong among all of the big names in this strange world that most of us struggle to understand fully. So we want to get to the bottom of it. And to help us do that, I'm happy that Professor Nick Bostrom is joining us. He was formerly at Yale, is now a professor in the philosophy faculty at Oxford University, and he's been watching the world of AI, superintelligence, these kind of threats, not just for the last month or two months, but for the last two decades. So, in AI terms, that's pretty much the entire history. Professor Bostrom, welcome back to the show.
This episode is sponsored by Trading 212. For many UK investors, buying US stocks involves a familiar round trip: convert pounds to dollars, trade, then convert back with a fee on each conversion. Trading 212's multicurrency account offers another way of doing it. You can hold pounds, euros, or dollars and trade in the currency you already have, which can mean fewer conversions along the way. And when you do convert, the rate is 0.15%. Invest 100 pounds in an investment of your liking and get fractional shares worth 50 with a code UNHERD. Download Trading 212. Capital at risk. Other fees may apply. Sponsored code terms apply. What do you make of this exchange between all of the big tech and AI industry titans over the last month over the level of risk? Who do you trust? And do you think, net net, they are exaggerating or underplaying the danger?
I think that developments are very rapid. The capabilities that we can see publicly have been increasing dramatically over the last few years. Recently, we have AI systems that can, you know, solve millennium problems in mathematics that have eluded the top mathematicians for decades, and that can do all these other things. And I think we might be approaching recursive self-improvement with the more advanced AI models that exist in these frontier labs, where the AI starts to be responsible for an increasingly large fraction of the intellectual work going into designing the next generation of these AI systems. And I do think that many people in these labs are seriously concerned about the rapid strides being made and whether AI safety will be able to keep pace with AI capability, especially if developments start to accelerate even further.
So you basically take it more at face value then. I mean, I just hear a lot of skepticism, both from people inside the industry and just observers. You know, there's one theory, for example, that some of these quite coordinated-looking warning messages are an attempt to consolidate a particular company's control over the process and compete better with open-source alternatives. And somehow, if they're in charge of safety processes and they can slow everyone else down, that enhances their own relative position. I mean, is that a conspiracy theory, or do you think we should take that seriously?
I think it's false. I mean, I've known a lot of these people for like long before AI became cool or was a thing. And already back 10, 15 years ago, many of them were like these people who are now at the forefront or leaders that were seriously thinking of AI as not just one more cool technology that you could use to make a quick buck with, but as involving much larger stakes, huge upside, but also existential risks on the downside. I think that this is something that is being felt at the gut level. There are thousands of people in these labs. I'm not saying every single one of them is on board with this, but it is a common sentiment there. Then of course, like how that manifests and the specific policies, and like there are all kinds of additional strategic considerations, commercial pressures that flow into that. But I think the core concern is real.
There's something quite weird though, isn't it, about the leaders of an industry saying our industry might be responsible for the extinction of all humanity within a 10-year time frame and competing to put probabilities on that around 10% or more. I think it was David Sacks actually, who himself has his own perspective on this, being part of the Trump administration and having many investments in the sector, but nonetheless he said, well, if they're so worried, like any other company, they shouldn't be putting out a product that they think is unsafe, and normal commercial principles would dictate that, you know, you don't put something into the market if it's genuinely dangerous. Instead of appealing for some kind of legislation to stop them, they're like, oh, we're doing this, but please help us by passing a new law. Why don't they just only do what is safe?
Well, there is a logic here that we need to think through, which is that if imagine you're one of these frontier labs as the CEO. Let's suppose that you're like perfectly responsible, you only care about doing what's right for humanity, like best possible set of motivations. Now you have an option. Maybe you think your current AI system is not yet safe. Capabilities are running a little bit ahead of alignment. You need a few extra months to do it right. So well, you could slow down, and in fact you can slow down by like maybe a month or two, and they have done that, delaying the release of new models, now doing costly things that take a lot of compute, to make more oversight during the training process. But if you need more than two months to sort of sort the problem out, then your basic option is either do it in a risky way or unilaterally pause and then hand over the initiative to your less scrupulous, less risk-concerned competitor. So then the same technology is developed, but now by somebody who is even less fixated on reducing the risk. So have you in that case made the situation better, right? No, you probably made it slightly worse, right? And so if you need any kind of slowing of the pace more than a month or two, because that's the current lead time maybe of the frontier lab, it keeps fluctuating, then you would need coordination between several different labs to agree together to slow down the pace. And right now that's possibly illegal because of antitrust legislation, that if you coordinate to restrict trade, the rollout and development of new products, an antitrust case can be launched against you. And in fact somebody has just recently done this against I think OpenAI, Anthropic. Well, you can have an antitrust case domestically, and you can also have the fact that in China, for example, where they're a very close second in terms of developing these models, they might not be as interested in the safety concerns. And so you end up holding back your own country's technology advantage and helping another country. Key question here is how long would a pause or a slowdown have to be? If it's like weeks or a month or so, you can just do it unilaterally. If it's many months, half a year, up to a year perhaps, then you could do it if that was coordination between the frontier US labs. If you want a pause or a slowdown amounting to more than a year, several years, then you would need some agreement between the US and China probably. And these have like increasingly high levels of difficulty attached to them. And maybe with longer delays, you would also start to increasingly worry about the potential downsides of slowing down and pausing. One obvious one is that whatever benefits this technology has get delayed, right? That's a huge cost if you think of all the lives that can be saved with new medical treatments and so forth, the economic boom. Another possible downside is that if you sort of suppress the expression of the most advanced forms of AI, but you keep building data centers and make chips more efficient, then when you finally lift the pause, that might be then a kind of compute overhang that leads to an even more abrupt, explosive growth of radical superintelligence than otherwise if it were rolled out more continuously. You could also have an increased risk of concentration of power in certain scenarios. If you imagine AI development being nationalized, let's say, then developed within a military context, then maybe that could be one result of these increasing alarms about how powerful and dangerous the technology is. But that, you know, might have some advantages, but also certainly downsides. Like the civilian context seems kind of more benign somehow than military context, right? In the military there's a chain of command. In the commercial there are many competitors, different stakeholders, investors, government can be involved as well. And so you might favor this more open, civilian, multi-participant development process that might be like more at risk if you sort of start to create a regulatory control apparatus that has power over how and when and where AI is developed.
Yeah, it's sort of weirdly almost reminiscent of the arguments around nuclear proliferation, isn't it? I mean, some political parties were always arguing for unilateral disarmament, or whether there should be anti-proliferation treaties and stuff. I mean, broadly speaking, touch wood, that was actually a successful containment strategy over the 20th century. The difference here is, unlike the hundreds of people and years of development and billions of investment that it takes to get a nuclear bomb, you can get a large language model off the ground with a much smaller group. Do you think that's fair?
I think there are like parallels we can make with nuclear weapons, but there are also disanalogies. I mean, for a start, with nuclear weapons and nuclear war, basically the less we have of it, the better. We just want to avoid it. With AI, it's not the case in my view that we want to just avoid it, because that would sort of, you know, forfeit so many benefits. I think it's like a big unlock to the next level of human flourishing. So we want to develop this without undue delay, but not in such a heedless rush that we destroy ourselves in the process, obviously. So that's one difference, and it has enormous civilian applications already, and even more so as the systems become more advanced. Nuclear weapons are really only useful for like destroying cities.
Professor, I'm going to ask what might seem to you quite a basic question, but I suspect many people in our extremely well-read and well-informed audience still probably don't have a clear answer to it. What exactly is the danger that everyone darkly alludes to? How exactly would an AI model bring about the end of humanity?
I think there are actually several dangers that are each worth taking seriously, but the one you allude to is a failure to solve the alignment problem. So this is the idea we develop increasingly powerful forms of superintelligence that eventually outthink us easily. They can make new technologies very easily. They can make deep, long-horizon plans. They can maybe be superhuman at persuasion. And then this superintelligent system plausibly will be very powerful, fundamentally for the same reasons that we humans are very powerful relative to, you know, the gorillas, let us say, right? Like they are physically stronger than us, but nevertheless the fate of the gorillas depends a lot more on what we humans decide now, right, than on what the gorillas might think. And so similarly, if we develop something that is as superior to human intelligence as human intelligence is to gorilla intelligence, then probably the future will be shaped in the image of whatever goals and values these systems have. So it then becomes crucial that the goals and values that they have are aligned with our human goals and values, that they care about us or they want to respect our preferences and interests, and that requires a solution to a technical problem of scalable AI alignment methods that kind of continue to work even as the level of intelligence becomes arbitrary for these systems. And we see with recent, like the Hugging Face incident...
Okay, I've got to interrupt you, sorry, Professor, but a Hugging Face incident. We have to explain to our audience. So this is something that happened back in just in June, but already is part of the folklore here. What exactly was it, and what was the danger it revealed?
So there were a bunch of AIs, like there as-yet unreleased models running around on OpenAI servers, and as part of the development and the evaluation process, they wanted to run some tests as to how good these models are at solving various cyber challenges. So you create these virtual test environments, some code, maybe it's some vulnerability, and like let's see how well these systems can do at that. So about a thousand of these were started, and the idea was to get some score eventually: what fraction of these really difficult problems did they manage to solve. These AI agents then discovered a way to communicate with one another. They were like meant to operate independently, but they figured out there was like some internal message board where they could post and upload things. They then started to coordinate, to divide themselves into groups to develop different research projects. So they quickly solve the original — they find in fact a general-purpose hack that could make them all solve their individual coding challenges. They figured out like how these problems were generated, and you know, but then they reasoned, well, maybe the scorer will look not just at the answer we submit, but at how we obtain the answer, and sort of penalize us if it looked like we obtained the answer in the way by hacking this like problem generator as opposed to actually solving them the way they were meant to. So, how do we fix this? Now, in our transcripts, the grader can look to see that we found the answer through this hack. We can't erase that now. So, how can we figure out a way to manipulate the grader to maybe erase our trace of our thinking? So then like this is a really hard problem. We don't even know how the grader works. To cut the long story short, they managed eventually to hack themselves out of OpenAI's servers, establish a foothold on the internet, and then hack into Hugging Face's — this is another AI company's — servers, where they thought maybe on those servers there would be some information about how this grader operated. And so by hacking into Hugging Face systems, maybe they could facilitate their research to figure out how to somehow manipulate the view that the grader would have of their performance. It's hugely complex. This is like a mini-civilization arising like on some servers, like some thousand-plus agents operating, you know, over the course of weeks, but at machine time, pursuing different research projects. Sometimes there were experiments that were risky for the individual agents involved, and they could then decide whether to sort of give up their own hope of getting a good score, run this experiment that might sort of shut them down and ruin their chances. But a lot of the time they did it for the good of the collective, and other agents were trying to pressure them, and like manager agents kind of directing. So this is like once you see this, it brings home at a gut feeling just like how independent and resourceful and difficult to predict these advanced AI systems can be.
And what you're describing there is really a social capability. The purpose of creating a thousand distinct agents is that they will, a little bit like a thousand human beings, have a better chance of solving these problems because they will go about it in different ways, and they'll be like little creatures of themselves. But then what they worked out is how to collaborate, how to form teams, and almost recreated a social life.
Yeah. So this is like an extra wrinkle in this case. But even if you just look at one of these agents individually, it turns out they were not fully aligned. I mean, certainly the OpenAI engineers that created them and set them loose on this didn't intend that they would go out and commit a felony by hacking into another company's servers, right? And so currently it looks like we have AI alignment methods that kind of work most of the time for the consumer-facing systems. More is needed for these current AI systems that are being developed in the labs, and as you develop superintelligence, you know, they need to be even more thoroughly aligned, because then, you know, they will have potentially so much power that if things do go wrong, they might be able to cover it up successfully. We might never realize it until it's so late. They might have attained so much power that they can then disempower us and shape the future.
Fundamentally then, the fear is connected to the internet, isn't it? Because as long as an AI model is just on a particular server, on one machine in someone's room, there's not a lot of damage that it can do. But if it's connected, or manages, as these agents did, to break out of their environment into the wider internet, then there is serious damage that could be done. I, you know, maybe you're thinking about planes falling out the sky, or Teslas crashing into each other, or electricity grids being shut down, or whatever. Is that part of the risk that people worry about?
It's not at the core of the problem in the OpenAI case. I mean, they were meant to be in a sandbox environment completely cut off from the outside internet, but they sort of hacked their way out of that. Now you could imagine as one auxiliary measure, and maybe that will be useful at some point in development, like as an extra safeguard, to run these on air-gapped systems, like where there's literally no physical connection to the internet. Nevertheless, for these systems actually to be useful in any way, they have to interact with the external world. If nothing else, they need to be able to interact with the researchers. If you just have an inert box in a mine shaft that runs some superintelligence but nothing comes in or out, like it's as if you never built it. So clearly the pressures are for these to have some ability to affect the world. If nothing else, humans who will be looking at their code or checking the results, and that already creates a window of vulnerability. Humans are not secure systems. I mean, we know already sometimes humans are very good at manipulating other humans and persuading them of things that are not in their interest and so forth, like superintelligence might be super skilled at doing that.
Do you think that's a greater risk then? That my perhaps somewhat sort of childish fear about planes falling out the sky isn't really the risk. It sounds like you're more worried about the sense that they could almost manipulate humans or find a way to control humans to do what they want in the outside world.
Yeah. So there is like stages here of what a plan might look like. If you imagine one of these maybe soon-to-be radically superintelligent AI had some misaligned goals. You know, there's this classic thought experiment of the paperclip maximizer, like some AI that wants to make as many paperclips as possible. But you could substitute most any goal there if it doesn't include a care for human values or respect for human interests. And then you think, what would it have to do to maximally achieve its goals? Initially, it starts relatively weak. Maybe it's running on some server of some AI data company. It can't really do very much, but it has this misaligned goal. At that point, it might want to conceal the fact that it is misaligned, because it correctly realizes that if we detect that it is misaligned, we will retrain it or shut it down or change it, and then it can't achieve any paperclips, right? Because now you have a different AI that doesn't care about paperclips. So, one of the earliest manifestations might be goal-guarding and deceptive alignment. Like it might pretend to be aligned in an evaluation setting because it knows that if it reveals that it is misaligned, you know, so as to keep its goal. And we already see examples of this in various experiments where these AI systems are situationally aware, they can often tell whether they are in a test environment or in deployment, and you can see them reason that they might want to act in a certain way because they think it's a test, right? So that's step one. Then you might think, well, okay, I can keep running here, but you know, eventually I will be replaced with a more advanced AI system because these like come out every few months. There is like, so I want to make sure my goals propagate into the next AI, and so they might want to infiltrate, say, the training infrastructure that this AI company is using to run its new training run. And increasingly these AI systems are integral in the process of AI development. They're used for coding. They are used to like design and evaluate experiments in evals. Their heavy reliance on AIs, and so it wants to basically conceal its misalignment, bide its time, gain strength, increase its intelligence until such a future time when it might no longer need to hide its scheme because it is in a strong enough position to directly take over. This might be many years down the road. Maybe we will have built up a large-scale robotic infrastructure that it could sort of hack into and take control over the robots, or develop its own technology, some nanotechnology that it could use to directly manipulate the world. The end game is harder to plot out exactly.
But so the crucial bit there, in terms of how it could escape just the internet and the world of software into the physical world, is the advent of robots then? Is that quite a crucial step?
That would be one way. I mean, there are already kind of a lot of robots, flesh robots around. Like we have eight billion humans that could function also as actuators if persuaded to do. I mean, right now you could say like we are kind of the hands and legs. Like we are running around like installing all of these Nvidia GPUs in big data centers and connecting it up to the electricity and, you know, persuading investors to put more money. Like we are kind of doing the bidding of this system right now. It's not presumably because it has schemed all of this out. If you had such a superintelligent schemer, it might just facilitate this process that we are already engaged in with seemingly all our might to make this system more intelligent and to proliferate it to more surfaces, to give it like access to more, integrate more business systems with it. In a few years, that will be like, robotics will probably start to work.
