Nick Bostrom Says We Are Clueless About What's Coming | Interesting Times

Thank you, Spencer. Good to be with you.

I don't think my outlook has changed fundamentally. I see it more that the world has changed. And so back when I started working on the first book, "Superintelligence," I began writing on that in 2008. At that point, the world was in a very different place. People were just dismissing this whole thing as science fiction. And nobody, save maybe a handful of individuals, were thinking about the challenges that would arise if we one day developed really powerful forms of machine intelligence. And so at that time, it seemed most useful for me to try to draw attention to the potential pitfalls of the path ahead — and so that we could use the intervening years to prepare ourselves to do our homework, basically like to do the research on alignment, etc. Now, of course, the discourse has changed a lot, and now it's very mainstream to think that superintelligence is not just a science fiction-like thing, it's like a real possibility. And so then it seemed less useful for me to just keep harping on — pinging the same string, and that there was this other side of the coin, which already always was there in my mind — the upside and the downside — and just take a glance at what would happen if things go well. Like if we take the best case scenario, that also has philosophically problematic aspects to it.

No, there isn't. Like the number itself kind of comes out of a black box and different people — those are, I mean, meant to be expressions of a subjective degree of uncertainty and confidence. It's not possible to calculate the number based on some set of agreed assumptions — which is reflected in the fact that different people come up with very different numbers. But I mean, my view is that there will be real risks associated with this transition to the machine superintelligence era, including existential risks. We should take those seriously. If you want a more qualitative assessment, I would say that I'm a fretful optimist. As in, think that there's a real chance things go quite pear-shaped, but you're hoping, you believe that it's not — yeah, I'm really excited about the upside. I'm hopeful we could get there, but I worry that we might not.

Well, I think there are several challenges that we would need to meet in order to get a good outcome here. One is the alignment problem. This is the technical difficulty of developing scalable methods of A.I. control. Basically ways to ensure that we can continue to steer these systems, have them do the things we intended for them to do rather than something different, even as these systems become ultimately arbitrarily capable. So that's one challenge. And then I think a different class of challenge is what you might call the governance problem. So assuming we manage to control this technology, then there is the question of what ends will we humans put it to. And sometimes we use our powers for ill purposes — to wage war, to oppress each other, maybe to concentrate power, to do all kinds of nefarious things. So the second challenge is, can we at least do a reasonably decent job at shifting the use cases predominantly towards the positive.

Yeah, well, I should clarify that the paper doesn't advocate for any of those things. It considers this possibility if we're unlucky. And then I consider if that were the case, then what would have to be true of a civilization that invents that technology in order for it not to be devastated. And depending on the nature of that technology, if it's one that incentivizes first strikes amongst major powers, then it might require some kind of global coordination. If the problem with the technology is that it makes it too easy for random individuals to bring down civilization — some super weapons of mass destruction that you could concoct in your own kitchen using ordinary materials — then it looks like you would need to do this very detailed surveillance mesh to prevent people from doing that. But the world is not guaranteed to be pegged up such that we will have an easy time. And anyway, so that paper, it just tries to create a framework for thinking what the different possible situations could be that we might find ourselves in.

So if we develop this strong form of superintelligence, then I think what that means is that across the board, across all other technological and economic fields as well, we would have extremely rapid advance because these superintelligent minds would then be doing the science and the research and the economic activity, but at digital speeds. So I think you would have a kind of rapid galloping towards perfect virtual reality or a cure for aging or space colonies or all kinds of really almost science fiction-like technologies that don't violate the laws of nature. They're just really hard. So the question then becomes, in such a condition of technological maturity, what would a human life look like? What would a good human life look like? What would a future path for us look like under these, in some sense, ideal circumstances? I have this concept of a solved world, which is the idea of technological maturity plus some decent job on the governance thing. So let's imagine everybody gets a slice, like everybody is free. There is. Then what happens is that a lot of the constraints that currently give shape to our human lives would peel away. You can have the AIs and the robots doing basically all economically productive tasks. You then have a leisure society where there is no longer any need for humans to work in order to get a paycheck. Now, I think if that was all, then I think it would obviously require some cultural transformation and adaptation, but ultimately, I think we could easily imagine living in such a world. There are already various groups of people who don't work for a living that you could look at for inspiration. There are children — young children don't work for a living. They seem to have good lives, spend time with friends, playing games, inventing things.

Yeah, but fair enough. Then we have retired people now. They already have a confounder that many retired people have poor health and stuff. So that's. But if you imagine somebody who has enough money and perfect health, they often have great lives. They go around traveling, spend time with grandchildren, they have hobbies, they watch movies, they golf, whatever it is like. It's not necessarily a dystopian existence.

And then you have various other people who maybe have a lot of independent wealth. They don't need to work for a living. So I think that there would be changes. The education system right now is kind of configured to take in children and then output productive workers with quality grading attached.

I have, but I do have sympathy because right now the world does need a lot of productive workers. So in some sense we have no option. But if we no longer needed productive workers, then it would be perverse to continue to think of education as just a preparation for entering into the labor force and get the desk job at an office. Then you maybe have an education geared more towards preparing people to live a great life, cultivate the art of conversation, hobbies, spirituality, physical wellness.

Well, so. So then you think a little bit harder about this, and you realize that it's not just our economic efforts that will become otiose in a solved world, but a lot of other instrumentally motivated effort also. Somebody maybe they want to be physically fit right now. They have to go to the gym themselves. They can't hire an assistant to lift the weights on their behalf. So for another person, maybe they want to — much as I would like to — yeah, maybe another person enjoys decorating their house. Like they want to pick out the curtains, the pillows, go through the catalogs, and then in the end, they can get it in just a way that is right for them, that they really feel reflects their own preferences. But at technological maturity, you would have recommender systems that would know your preferences so well that it could just select the curtains and the cushions and the rest and do a better job than you would if you went through the trouble to do it yourself. And so you could still do it yourself, but it might seem kind of pointless. And so you can walk through the different things that people fill their time with when they don't have to work. And it does seem that there would be this question of, is there still a point to doing this if there was a shortcut, a button you could press that would result in the same outcome?

Yeah, like maybe the runner wants the endorphin boost at the end of it. Or we engage in some activity because it gives us joy. But at technological maturity, you could directly induce the endorphin or the joy. And so it does raise this more profound philosophical question of what ultimately has meaning. Like what is worth doing for its own sake, not because it is practically useful to achieve something else.

No, I think that's important to clarify. I think so there is this chapter in "Superintelligence," the earlier book, where I consider different possible paths that might result in superintelligence. So one is whole brain emulation, where instead of building these completely artificial synthetic minds, you would try to do something that is very closely based on how the human brain operates. Like maybe detailed brain scans and then implementing neural networks. So copy-pasted from this. That would be a second route to machine intelligence. Then you could imagine forms of improvements in our collective epistemology — institutions and tools that allow humans to solve much more difficult problems together. So that would be like a kind of collective superintelligence. Then you could imagine maybe some smart drugs, nootropics that would improve our memory or concentration, or you could imagine some form of genetic enhancements. But I think the story began a long time ago with somebody developing a better way to chip a piece of flint to create a better axe, and then better ways of planting seeds, and then all kinds of inventions, double-entry bookkeeping, all the way up to the current era. And then now maybe we're getting close to the point where we can automate a lot of this process of inventing. And then that could speed things up radically. And that would be the most plausible path if we do end up at technological maturity in our lifetime. Certainly it probably would go through the route of machine superintelligence.

Before answering that question, I guess it's worth just reflecting for a second on what it would mean if we said no. Even though we had the ability, it's the embedded telos in so much of the modern world. In science, we're trying to figure out more and more about how the world works. In technology, we try to increase our ability to accomplish goals more efficiently. The economy tries to invent new processes and ways of organizing so we achieve our goals with fewer inputs. And so if one rejected the endpoint, I think it would then call into question — it would back-chain from that and think like it's kind of perverse to be working so hard to achieve a goal such that if we achieve the goal, it would be very bad for us.

So having no constraints is the goal, right? And then there would be in some sense it's implicit in technology now. But then it does turn out that if you actually follow through that logic, it becomes a lot less obvious how cheerful you actually are about achieving that goal. So that's, I guess it's like a dog chasing a car. It's kind of — yeah, yeah. Once you catch the car, all the stuff — yeah, exactly. Thought what would happen if we succeed. So maybe technology's good. Technology's goals aren't the only goods or goals that are choice-worthy that we should want.