How is it that the makers of AI, the ones who have been saying that AI is the future, have suddenly now discovered the joys of restraint and caution?
Possibility of a misaligned superintelligence. You could imagine a future, maybe, where the Earth is converted into a giant data center or space probe launching system, all in the service of the AI's goals. Increasingly, it looks like we are handing over more and more responsibility for these AI systems.
What would that look like in practical terms?
At some point, these will just be radically superior to humans. A risk of human extinction, of sort of permanently destroying our future.
For years, the AI bros sold us the promise of smarter machines and apparently the end of human drudgery. But now some of the same people building the beast are discovering an old-fashioned and rather human virtue: caution. Is AI doomsday a serious risk or Silicon Valley's new favorite horror story? How do you align a machine with human interests when humans can barely agree on their own? Tonight, we are joined by Professor Nick Bostrom, a philosopher who specializes in and has written extensively about AI risk. Thank you so much for joining us on First Post, Professor.
The AI doomsday scenario is all anyone can talk about. First things first, though: what does the doomsday scenario actually look like beyond the science fiction and conspiracy theories? You've warned about the risk for years.
Well, I think there are a few different ways that this transition to the machine intelligence era could go wrong. One is the possibility of a misaligned superintelligence — some system that has goals, long-term goals that it wants to achieve, eventually becomes very smart and very powerful, and then shapes the future to achieve its goals rather than things that we humans care about. You could imagine a future, maybe, where the Earth is converted into a giant data center or space probe launching system, all in the service of the AI's goals, where we have fallen by the wayside.
That being said, AI fears — the warning signs have been there all along. This time the fears have resurfaced, triggered by the makers of this machine, Frankenstein's monster, if you will. How is it that the makers of AI, the ones who've been saying that AI is the future, have suddenly now discovered a sense of restraint and caution? How and why now?
I don't think it's sudden. People in these labs have, for many years, I think, recognized that it will ultimately be a very powerful technology — not the AIs we had in the past or that we have now, but if you extrapolate this forward, you will see that eventually we will have machines that radically outstrip us in technological ingenuity, strategic ability, and so on. And that could be very dangerous as well, if it's misaligned, if it's used for bad purposes. I think what has triggered the recent more public activity is partly the Hugging Face incident, that was a kind of warning shot, and then just the fact that the systems are now starting to be capable enough to really do impressive things.
So, when you have that level of capability, it doesn't take a huge leap of imagination to think, well, what if this goes one more step and then one more step beyond that? At some point, these will just be radically superior to humans. And then, you know, from that, a lot of things could follow. Hopefully, a great unlock for human flourishing — this is the hope. But to not imagine that there could be some risks associated with that would just seem naive.
Right, but I guess I'm still trying to understand why this public admission and why now? AI researcher Jacob Coxon resigned from Anthropic, citing concerns that frontier AI companies are gambling with our lives and claiming that their employees believe AI could kill all humans by the end of the decade. These are glaring warnings and huge claims. How do you assess this?
Well, one aspect is the highly competitive situation amongst the frontier labs. So, if one of them, for example, feels that it wants to be careful and cautious as capabilities are rapidly increasing, maybe they think they need another few months to really put all the safety measures in place, to check everything. Right now, if they slow down for three or four months, it just means they hand over the initiative to a competitor who's maybe willing to take more risks. And so, in this competitive situation, it is very difficult, even if it were for the benefit of all, for anyone actually to go a little bit slower. It requires coordination.
And so, I think one of the things they're hoping for is that at the minimum, maybe the US government could waive antitrust enforcement to allow them to synchronize for the sake of safety, and perhaps above that, have some regulatory body that could set safety standards that all players will have to meet.
