I found the 2014 book to be sort of... it was very interesting. It's clearly ahead of its time. I also found it to be sort of like, unrelentingly grim. The newer book I found to be very fun. You seem to make fun of yourself a little bit in it. You seem to make fun of the students. You seem to be having a good time writing it. I'm curious if, like in the ten years between the two books, did something change such that you were... I don't know. The new book just seemed like you were having more fun writing it.
And yes, it's not so much that my views have changed. It's just that, all along for me are these two sides of the coin, and back in 2014, it seemed more pressing and relevant to study the downside, because that was very much neglected, and it seemed important to try to introduce some concepts so people could start working on figuring out solutions. And, well, there's much more awareness and understanding of that. It's time to also look at the other side of the coin. I wanted, in addition to talking about this utopian condition and maybe also, sort of illustrate a little bit the spirit with which I think we should approach these choices that would open up to us if we make it through this transition, a spirit of sort of playful generosity.
I mean, playful generosity is not necessarily a characterization that I would associate with a lot of Silicon Valley and hyperscalers and the people who are actually modeling... Yeah. Or creating these models. I don't know, like, there seems to be this weird tension between futurists and transhumanists who seem to be very focused on saving humanity, but actually don't seem to like people that much at the moment. You know what I mean?
Absolutely. They like humanity, but not human.
Hello and welcome to another episode of the Odd Lots podcast. I'm Joe Weisenthal.
Tracy, there have been two... Well, there's so much AI news obviously every single day for the rest of our lives if we're lucky. In the good scenario, there's two big events or sort of things though, that have really struck out to me lately that, to my mind, represent either the very good version or the very bad version, depending on how you look at them. So the bad, the scary version. And we're seeing more and more headlines like this. Several weeks ago we had the OpenAI Hugging Face incident, and people are, you know, some light hacking. Light hacking. It raises all these fears of Skynet type scenarios, etc., where we can't control the model. And then in this sort of quote, good version, we had OpenAI recently coming out and saying it had made all of these breakthroughs in theoretical mathematics, which is good in the sense that it's like it's working, but seems to be creating an existential crisis for some of the most brilliant people on earth who work in theoretical mathematics.
Right. So we've talked about this before. When it comes to AI, the range of possibilities seems more extreme than some previous technologies. Right. On the one hand, you have death and dystopia. The robots basically kill us all, and on the other hand, you have this idea that maybe superintelligence, AGI, is going to solve literally all the world's problems to the extent that you and I won't know what to do with ourselves anymore. Right. It seems hard to imagine. Right? Like that is the good scenario, is that... and then the whole range of outcomes in between. The good scenario is we've literally solved everything. In which case the question was the case. We're sad and miserable and yeah, or grappling for a meaning to our lives. Yeah. So is there a good scenario? And they are actually like, I don't know. Or maybe it's somewhere in the middle. An AGI will figure it out for us. Maybe the middle scenario is we just have a huge financial crisis of all the debt, and that's something between human extinction and meaninglessness in our lives. And we just have something that resembles, like the 2008, maybe the 2008 news, is actually the best of all suboptimal worlds of the AI.
Excellent. This will be our most uplifting conversation ever. Anyway, we've been doing more AI episodes seemingly every year since ChatGPT came out, but people smarter than us have been anticipating all of these things for a very long time. Some of these exact scenarios, the existential crisis facing mathematicians, the model escaping from its sandbox and circumventing misalignment with the creators. People have been thinking about this and anticipating this for literally decades, even before any of us were talking about that.
It's funny how, like the Venn diagram of sci-fi writers and like legitimate academic futurists, all of it is just converging, right?
Totally. And then there's the question of are the models bad because they read all of the sci-fi, and that's what they were trained on. Anyway, I'm very excited to say we really do have the perfect guest, someone who has been thinking about the whole range of possible questions and outcomes with AI for a very long time. We're going to be speaking with Nick Bostrom. He's a professor, a philosopher, and his most recent book is Deep Utopias, which sort of attempts to answer the question, what happens if AI goes right? So, Professor Bostrom, thank you so much for coming on Odd Lots.
What do you make of the existential crisis of mathematicians, their sort of realization that maybe there will just be no more puzzles left to solve, because the AI is just very good at them and can find them all cheaply and answer all of our questions?
Yeah, hopefully we'll all be in that position at some point. I guess we could compare it to what happened with chess, where computers have always exceeded humans. And yet a lot of people are still involved in training themselves in chess and competing in chess and taking pleasure in that. So maybe mathematics will ultimately become more like a hobby pursuit. Right? Joe? You still play chess, right?
Yeah. I'm glad you brought up chess, because I still love chess. And why? It's fun. It's fun, you know, it's still a puzzle. It's still mental exercise for me. It's still something to do to pass the time. While I'm waiting for the subway, playing a three minute game online, even before Stockfish and so forth, I knew I would never be a grandmaster. Still fun. All right. So, Nick, you have basically two books that deal with these tail risk scenarios now. So one is this idea of deep utopia, and AI is going to solve all our problems. And then you also have Superintelligence, which seemed to paint a different scenario where AI poses a physical and possibly existential danger to humanity. Which one do you think we're closer to right now?
Somewhere in the middle, maybe. So the first one was Superintelligence. It came out in 2014, and I had been working on it for six years before that. And at that time, the whole notion of having human-level or even superintelligent machines was generally just ignored or something. Many people dismissed it as science fiction, and it seemed to me clear that this would one day happen if science and technology were allowed to continue on a wide front, we would eventually figure out how to make smart algorithms and build big computers, and that once we succeeded with that, we would then be faced with quite a bunch of challenges, including this alignment problem of how you can sort of design a mind, you know, more brilliant than any human, and still steer it so that it will actually have an impact on the world that is beneficial. And so that, I guess, like with many other developments, helped eventually build a kind of global conversation. And so now all the frontier labs have research teams trying to develop scalable methods for AI control. And the conversation really has kind of swung, and there's like this rise of people who are really worried about what will happen. And I thought it might be time to have a look at the other side of the coin, like what happens if things actually go well, if we solve this alignment problem, let's also imagine we solve, to whatever extent it can be solved, like the governance challenge. So like we don't use this technology to oppress each other or wage war. And we got like the best possible outcome, like what would a human life actually look like in this kind of, I call it a solved world, where all practical problems either have already been solved or to the extent that they remain are better worked on by these superintelligent machines and robots, and where there is no longer any need for human economic labor. And in fact, if you think more deeply about it, you realize it's not just sort of earning wages that becomes unnecessary for humans to do, but a lot of other instrumentally motivated efforts as well would seem to lose their point. And so that's the kind of topic of this Deep Utopia book, actually talk about some of that, other things that could be solved.
Because when I was like a third of the way through your book, I was like, you know what? It's fine if all of our economic problems are solved, I'll just derive satisfaction from raising my kids and then maybe helping to raise their kids, etc. But as you point out, even that in the truly solved world, maybe even something like raising children might not be an available avenue for us to find at least our conventional notions of satisfaction.
Yeah. So we need to, I guess, make some distinctions here. So when you say satisfaction, that could mean a subjective state of mind, like you're feeling happy and pleased and relaxed and joyful and so forth. I think that would be easy to achieve in this condition of technological maturity. It could also mean satisfaction in the sense of our preferences having been matched, like all the things we want from life. And there it's a little bit more tricky because one of the things that people might want is purpose, like having worthwhile things to work on and struggle with. And there there is more of a question mark which we can sort of dive into, but if we take it sort of, so first, like we automate the economy, okay. So you wouldn't need to sort of commute into the office every day just to sort of get your paycheck. That would require a big adjustment in society and in culture, because right now, like the whole education system is geared towards producing workers, like children are coming in, right? And then they are processed, learn to sit at a desk to receive assignments. They are sort of quality graded and then spit out at the other end. And then you have somebody who can go into the economy and perform some economically useful type, so that that would no longer make sense in this scenario where machines can do all of these jobs, and you might want to have an education system that instead of focused on, you know, developing skills for enjoying life, like cultivating the art of conversation, appreciation for nature and beauty, you know, sports, hobbies, the ability to form deep friendships, spirituality, all of that kind of stuff. But then, as you alluded to, you think through what it really would mean to have such immense affordances. And because here it's not just that we would have superintelligent machines, but the superintelligence could also bring along a host of other technological advances. Right. You have science and technology running now at machine speed rather than human speed. So you get the kind of compression of the future. All these science fiction, like technologies and inventions that maybe we would make if we had 20,000 years to work on it, we would maybe have space colonies and perfect virtual realities, cures for aging, like all of that might have been quite soon after superintelligence. And so then if you ask somebody now who doesn't have to work for a living, oftentimes they have very busy lives. But maybe what do they do? Well, so for somebody maybe they like fitness. Maybe they spend an hour a day running or at the gym. Right. But at a technological maturity, you could just pop a pill that would achieve the same physiological effects. And like the state of mental clarity and relaxation that some people get, right? And so then you can start to work through the different things that like maybe for somebody else, they like, oh, I like decorating the home, could be a fun project that people might do, right, if they don't have to work. So like they go through the catalogs, they go to the store, they pick out the perfect curtain. They do all of these things. That could be. But then, like with superintelligence, you could have this kind of system that really is good at knowing or inferring our preferences, and it could do a much better job than I do at picking out all the perfect items and purchasing them and like installing them with robots, and so that the end result would be much better if you just press the button and say, hey, decorate my home, than if you went through the whole trouble of doing it yourself. So you could still do it yourself. But is there really a point to it? That would seem to be this kind of purposelessness. And you can sort of walk through the different things that one might fill one's life with, if one is like on an extended holiday, or on retirement, or you're like sufficiently wealthy, you have just decided to quit working, and a lot of those also, it seems, could be automated. So you would have this idea like more like a post-instrumental condition rather than just a post-work condition.
Some of this sounds nice. And I myself enjoy a little bit of interior decorating and going to the shops and buying stuff. In the utopian vision of the future, how are economic gains from AI actually distributed? Because again, like all of this sounds very nice, but it also costs money. And if we're not working because the robots have replaced us, I'm not sure where that money comes from.
Yeah, so that's a more practical problem, shall we say, which is like deliberately set aside in this book that if you talk about, because I wanted in that book actually to get to the question, the more philosophical questions of what ultimately gets value. But it is, of course, an important question, I think. So a lot depends on how it pans out. But, I mean, we do have a taxation system. So you could imagine, like in the simplest scenario where things just continue as they are with better...
Technology, maybe some companies make windfall profits from serving the latest models, building the infrastructure, the data centers, etc. Companies keeping efficiency gains from not needing to hire all of these humans. And then some of those capital gains, you know, our dividend income gets taxed and that could be redistributed.
The good thing about these scenarios where there is massive automation and hence massive unemployment is that those would also be scenarios with extremely rapid economic growth. So the pie would be expanding humongously, which means that you'd still maybe need some mechanism whereby people who don't have, you know, assets can still benefit, but even a small slice of a really enormous pie could go a long way.
So I think that could be this tsunami of wealth that would sort of be flowing through society. And many services that currently people have to pay a lot of money for might become almost free. I mean, we already see it with some, like medical advice, for example. Right. Maybe before, if you don't have public healthcare, you would need to maybe pay a doctor, you know, $400 to get a consultation. Now you can ask ChatGPT and get, you know, in some cases a better opinion, basically for free.
And that could happen, you know, in many other fields as well, legal advice, you know, maybe have cheap robots who could do your cleaning and, you know, gardening and run errands. So, yeah, some combination of sort of deflationary effects, and then maybe some sort of general spillover, either from philanthropy or taxation, government redistribution.
I found the 2014 book to be sort of, it was very interesting. It's clearly ahead of its time. I also found it to be sort of like unrelentingly grim, and you sort of, there's this very narrow path that we seem to have, and we'll talk more about it maybe later, of like avoiding the doom fate. The newer book I found to be very fun. You seem to make fun of yourself a little bit in it. You seem to make fun of the students. You seem to be having a good time writing it. I'm curious if, like, in the ten years between the two books, did something change such that you, I don't know, the new book just seemed like you were having more fun writing it.
Yeah. So it's not so much that my views have changed. It's just that, all along for me, these are two sides of the coin. And back in 2014, it seemed more pressing and relevant to study the downside, because that was very much neglected and it seemed important to try to introduce some concepts so people could start working on figuring out solutions. And, well, there's much more awareness and understanding of that. It seemed time to also look at the other side of the coin.
And so it's more just like both sides were there all along in my thinking, but, you know, in any one book you can't talk about everything. The style is, as you suggest, a different topic. And that's partly I wanted, in addition to talking about this utopian condition, maybe also sort of illustrate a little bit the spirit with which I think we should approach these choices that would open up to us if we make it through this transition, a spirit of sort of playful generosity that I think might make us more likely to sort of explore this large space of wonderful new modes of being and the wisdom. And I think so I wanted to sort of illustrate in the right are these like fictional characters, for example, in the book, and they're kind of living a little mini version of a utopia of sorts as they are sort of discussing and debating these topics.
I mean, playful generosity is not necessarily a characterization that I would associate with a lot of Silicon Valley and hyperscalers and the people who are actually modeling or creating these models. I don't know, like there seems to be this weird tension between futurists and transhumanists who seem to be very focused on saving humanity, but actually don't seem to like people that much at the moment. You know? You know what I mean?
