The line at which these rights need to be granted is once you cross sentience, right? So Claude 1.0 clearly not sentient. You know, current Claude probably not sentient. You know, Claude several years from now potentially sentient. Is that the right line? Well, I wouldn't be so confident that current AI systems don't have some form of mentality.
Hello and welcome to The Most Interesting Thing in AI from Rethink, The Atlantic's creative marketing studio in collaboration with PwC, a leader in helping organizations use AI. Is AI a threat to humanity? We've all seen sci-fi movies about machines that become self-aware and then rise up against their human creators. And those movies are fun. But as AI becomes more capable in the real world, we're hearing more worries about existential risk. About 10 years ago, the Oxford philosopher Nick Bostrom wrote a book called Superintelligence. And I remember reading it and thinking, whoa, there were a lot of really concerning scenarios that he laid out in AI. And I was the editor of Wired magazine then and a believer that AI was going to come reasonably quickly. Of course, AI came a lot faster than even I thought. And Bostrom's thinking was profoundly influential on the group of folks trying to steer AI in such a way that it wouldn't lead to catastrophic outcomes. But here's the interesting thing. There has been a ton of work. There have been lots of conversations about what happens if this goes wrong. But Bostrom has shifted. He's now more interested in utopia than dystopia. I wanted to talk to him about his change. I wanted to talk to him about how his old work stands up and I wanted to talk to him about where he sees AI going now. So here's my conversation with Nick Bostrom. Nick Bostrom, welcome to The Most Interesting Thing in AI.
Hey, glad to be with you.
I'm delighted you're here. So I want to start with your book Superintelligence, which I read a decade ago. It was one of the most influential things I read. Helped shape the way I thought about AI. It shaped the way a lot of people thought about AI. And so I'm curious if the world that you see right now, the models, the tech, the improvements, if it maps to the world you foresaw then, what is the same? What is different? Let's begin with your core argument and what you see now.
I think to a striking extent, yes, the developments that we are seeing in AI, it was in my view foreseeable that at some point AI capabilities would start to match human general learning reasoning abilities and then at some point after that go way beyond that and that this would be maybe the most fundamental thing ever really to happen, certainly since the industrial revolution, but perhaps even since the rise of the human species, because we are bringing into existence another form of mind. I think one thing that was not obvious in advance was that there would be an extended period of time, many years, during which we would have roughly human-level systems. So we've had now for maybe five years kind of these large LLMs that can talk in natural language and that have many of the psychological quirks even of humans. They're quite anthropomorphic in many ways and that was a possibility, but it was not the only possibility. There were also possibilities back then of things happening much more abruptly, that not very much would go on and then somebody maybe, you know, in some lab would figure out the key missing ingredient and suddenly the thing would go foom. So I think the actual scenario we have means that it's easier for more people to sort of start to recognize that this is an important thing, get some experience with the kinds of systems we have before they are sort of, you know, some radically disruptive force in the world. And so more actors are kind of involved now than would have been the case in some alternative histories.
So I think this means you might be the only person in the world who feels like AI is kind of going more slowly than expected.
Probably not the only one, but I mean, and there was a distribution of sort of...
And like a distinction as well between the time from back then, say if you start the clock in 1990 or something like that, to how long it would take until we sort of reached some takeoff point, and then a separate question of how fast that takeoff would be.
Let me ask you about the takeoff because I think that is extremely important to the precise moment we're in right now. So, one of the methods for takeoff that you describe so well in the book and have talked about is recursive self-improvement, where a machine invents a better machine, which then invents a better machine. And you can imagine that process happening very quickly. And up until this moment, it's been mostly humans making the algorithmic improvements and making the changes. But now, Gemini is improving Gemini. Claude Code is writing the code that improves Claude Code. Like the fundamental AI research is being done by machines. So could it be the case that things have been slow for the last several years, but we might now have recursive self-improvement that makes things go very fast?
I think we are now sort of in the early stages of some form of recursive self-improvement in that the AI labs are heavily using coding assistants to assist with their work and it does seem to speed progress to some extent. The loop is still not closed. There are still many ways in which these highly skilled human researchers and engineers are needed. But the uplift they are getting from actually being able to use the AI systems to accelerate their own work is increasing and that is one of the possible mechanisms whereby at some point you could get a kind of further acceleration, like once you start to close that loop.
What would have to actually happen to, as you say, close that loop? Like what needs to happen between now, or what will happen between now and when we get the real recursive self-improvement?
Well, there are certain things that current AI models are still not very good at. One of these, I guess, is research taste, trying to come up with new ideas and having some sense for which new ideas are promising to try. And maybe relatedly, long-horizon tasks. The current systems work well for things that humans maybe are able to do in, you know, half an hour. But if you sort of unleash them on like week-long projects and stuff, they often still go off the rails or get stuck or go off in some wrong direction, particularly if it's not easy to tell whether they got the right answer or not. And so at the frontier of research, you need a lot of judgment that it might take a long time to figure out whether a new research direction actually will bear fruit. And so those are places where human judgment is still valuable.
Putting these things together, do you think that the curve of improvement in AI capabilities will at some point hit an asymptote because human intelligence is finite, energy is finite, like, or do you think it's more likely that you'll have a process of recursive self-improvement or something else that makes the curve go, you know, up, up, up, maybe asymptote briefly and then skyrockets up?
Yeah, I don't think human brains are at the ceiling of the kinds of cognitive systems that can be instantiated in our universe. It's pretty clear that they are not, if for no other reason than just looking at the sort of the basic computational physics of the brain. I mean, the brain is pretty small, right? It has to fit inside the cranium. Theoretically, of course, if you have something the size of a warehouse, you know, there'd just be much more oomph in there. And we know transistors can operate at sort of, you know, gigahertz, whereas neurons, you know, they fire maybe 100 times a second. And so there are these basic kind of computational limitations of the human brain. So ultimately, I think things can go way beyond any individual or even like our collective intelligence.
If the ceiling's not a human brain, but is there a ceiling or is the ceiling infinite?
There's like some ceiling. Like ultimately there are fundamental physical limitations to the kinds of information processing that any physical system can do. Yeah, there's like fundamental limits to how fast you can process stuff. At a very extreme, you have, I mean, if you build a big enough brain, at some point that's dense enough, it will collapse into a black hole, right? If you build it very large, then there will be speed of light limitations if you have one part of the brain trying to communicate with another. And it might also be that the returns to intelligence at some point hits diminishing returns. Like once you have a system that's smart enough to basically figure all the important stuff out and invent all the really useful technologies, then like even if in some sense you could increase information processing beyond that, it might just not achieve much. But I think those limits are very far from where humans operate.
Yeah, there's still a distance between as smart as Claude Code is and so intelligent it collapses into a black hole. So where, what do you think happens on the overall question of intelligence in the next, I guess, in the next year and then in the next five years?
It's not really possible to have a high degree of confidence on these things. If things were to stall out a bit, a possible explanation for that might be that up until now, as I said earlier, a lot of the advances have been driven by this very rapid scale out of the compute used to train and run frontier systems, which in turn has been driven primarily by just increased investment. There's some background rate of improvement in the actual chip technology, but a lot is just people just spend more money. And you could spend even more money than we currently do, but there's a limit to that, right? Like, so you could go from a desktop computer that cost a thousand, you know, up to a million. Now we have billions, you know, maybe you could spend 500 billion or a trillion, but above that, you can't just keep doubling every few months, right? So at some point, the rate at which hardware grows will slow down, just because it starts to use up a large chunk of the actual manufacturing capability of the foundries that exist. So if by that time we haven't already achieved superintelligence, then, you know, that could then start to stretch timelines out.
And when you say superintelligence, it's not like there's a line you cross where now we have superintelligence, like on Tuesday we did and on Wednesday we didn't, and once we have it, it stops. Isn't it just a continuum? Like you reach superintelligence, but the superintelligence you have on Tuesday is vastly inferior to the one you have on Wednesday and six months from then.
Correct. Yeah. So as you zoom in, then obviously you can make finer distinctions and like the AI we have today, I mean, it is superintelligent in some respects and not in others. So you have a sort of jagged capability profile. Ultimately though, I think that it will exceed us in all these different dimensions and in particular when you're able to sort of close the loop on AI research.
Actually, let me give you one form of intelligence that I'm interested in. So I play guitar and I can't read music and I can't sing the songs I've written. And so the memory of the songs that I've written is entirely embodied, right? It's finger movements, it's muscle patterns. Is that kind of knowledge and intelligence extractable by a more powerful AI in some way? And is there any form of human intelligence that could not be replicated by a, you know, superintelligent AI system?
I mean, everything that you do is ultimately a function of your brain having various neural networks that, you know, that the speech you produce, but also the motor movements. And we are seeing progress in robotics as well. It's kind of lagging a little bit behind, partly because it's just harder to get a lot of data and to rapidly try, like if you have to actually route through physical robots that they keep breaking, they operate on a certain timescale. If you do it all purely digital, it's a little bit easier, but I think we will see like sort of motor intelligence also advance.
So this is super interesting. So I gave you my guitar example. What you're saying is, if I'm hearing correctly, it's that, you know, my knowledge of how to play a song is embedded in my brain. I just can't transmit it through my mouth because I don't have the language for it, right? I don't know the notes. I don't know how to describe, I don't know how to sing it. And so if the brain is just math, it's just neurons firing in this way. And an AI system can replicate the math, then an AI system can replicate, it's just harder to train on because we don't know how to put the data into the machine because I don't know how to describe it. But it could be replicated.
Yeah. And we don't need to describe it to train it. Like, I mean, already if you just look at the ability to recognize like a cat from a dog, or, you know, one human face from another, it's often very difficult to verbally describe if two people are looking alike what exactly the difference is. We can sort of pick out some gross features like, does he have glasses? Is the hair white or black or stuff like that. But the kind of pattern recognition that we use initially is very hard to describe in words, but nevertheless AIs can become very good at that. That's most of what humans do is not really verbalizable. Even the things that we do verbalize, like what actually makes the difference for a sort of insightful comment from a dumb comment is really hard to sort of explicitly write down as an algorithm. It's like a hugely complex thing with many factors weighing in. And that's exactly what these neural networks are good at and what really made it work.
All right. So let's go back to, you know, one of the themes you've mentioned is how important it is to be at the frontier. The amount of cost that goes in, it seems to me like we're heading to a world, though this is still to be determined, where a very small number of players have control over the most powerful AI. How much does that worry you? And would you prefer a world in which it's much more democratized, where instead of five frontier labs, it's 100 open-source community shops? How much are you worried about the power concentration?
Well, I think that is one of several big worries that one might have when one looks at this transition to the machine intelligence era. It's not obvious that having a lot more frontier developers would really on balance reduce risk, because I mean, maybe it would reduce the risk of power concentration, for example. Another thing you might worry about is, say, AI systems giving uplift to, say, the design of biological weapons, which I think already current frontier systems are capable of doing to some extent. And you might not want to democratize the capability to create weapons of mass destruction. If you have a small number of frontier models, it's easier to hope that those have some level of sense of responsibility and they implement safeguards to sort of have, say, the AI refuse if you are asking it to design like an improved version of smallpox. Like currently, like, yeah, Claude and Gemini will refuse that. If you have an open-source model, on the other hand, I mean, maybe it's trained to refuse that, but it's quite easy to remove those refusal mechanisms through post-training. So you sort of lose any ability to shape or limit what these AI systems are used for.
But if so, if that's the case, then the optimal number of super powerful AI systems would be one, right?
Well, it would make it easier to control these potentially negative applications. It would, on the other hand, increase the risk of power concentration.
So, if I were to give you a couple of choices, all right, Nick, you can have a future world, say two or three years down the line, where there is incredibly powerful AI, and there's one central company that does it. There's where we're headed. It seems like we're headed now, which is where there's six, there's 30, or there's 100,000. Which do you take?
Yeah, it's a good question. And I mean, what we have now is really like we have a few systems that are at the frontier and the leader is very like maybe just a few months ahead of the second follower. And then we have maybe like a year behind or so open-source models. One vision that some people had, say, 10 years ago was that like maybe there would be one company that would be running significantly ahead of like maybe it would have a year or two of a lead time to like the second follower. And then at some point when it reached sort of the threshold where like, okay, now we really know how to do this. We just need to like scale it up and it will become a superintelligence. It would then at least have the option of slowing down. It could sort of voluntarily pause for a year or so to implement extra safeguards, to double check everything, and then sort of, you know, slowly crank up the power level of this right over a period of months or a year. So that seems maybe a little bit safer than if instead you have a situation where there are a lot of competitors running neck to neck and any one of them, if it decides to slow down or take some extra precautions, just immediately fall behind and the race goes to the most reckless. That seems like a kind of riskier setup.
Which is interesting because that kind of seems like where we are, right? And if you look at the amount of work being done at safety and alignment at most of the frontier labs, it is deprioritized in favor of work to go faster because there are billions, trillions of dollars at stake. There's a really complex ego problem among a bunch of men who don't like each other and are competing against each other in different ways. Like it doesn't seem like alignment and safety are a major priority. Certainly not if compared with the way people spoke about AI five years ago.
Yes and to some extent no. I mean, I think it is true that it would be nice if there was a higher level of interest in like really focusing on safety, etc. I think though, compared to what one might have expected, there is a relatively high level of awareness amongst these frontier labs and many of the key people there that AI is not just one more cool technology, it's not just like a new iPhone or some like thing like that, but that it has much broader implications. And many people are thinking quite seriously about that, like teams working specifically on scalable AI alignment techniques.
Let's talk about alignment for a second. So first off, why don't you describe what you mean by alignment? And then the second part of the question would be, how does your view of alignment differ from the view of alignment of the folks you just mentioned, the folks at DeepMind, Anthropic, OpenAI?
Well, so alignment is basically making sure that the AI systems behave as the people building them intend. Like a few years ago, you started to have limited versions of the alignment problem in, like, you'd wanted to sort of make sure that the AI systems don't go off the rails and start to insult the users or, you know, spread hateful content and stuff like that. Now there is a slightly more sophisticated version of this where you might want them to sort of reliably refuse requests to assist with, like, biological weapons design or cyber attacks and stuff like that.
So let me ask you this variation on the question, which is totally there. Definitely people working on alignment. I have friends who work on alignment, and my fear is that the power of the models and the complexity of the models is increasing much more quickly than our understanding of alignment, and even in fact of our understanding of how the models work. Right? Like, you know, our quest to get to explainability, to understand what's really going on in a model, is improving, but at a much slower rate of progress than the actual power of the model. And if those two things aren't synced, power increases more than understanding alignment, aren't we headed towards trouble?
