Nate Soares on P(Doom), Alien Superintelligence, Human Enhancement, and the Future of AI

Uh, skill issue. Yeah, I think there are actually a lot of factors. One is insufficient time. The field of AI just moved too fast. One is we tried to get a lot of, you know, world geniuses rallied around the problem and largely failed in that regard. You know, it would have been nice if a lot of the alignment problem had been the sort of thing top mathematical talent can work on and top, you know, a lot of these physicists, a lot of these mathematicians who are going into hedge funds, if we could have somehow gotten a lot of them to really realize that this problem was one of the big ones, one of the important ones. That was one thing Mary was trying to do was sort of like build up that community that largely didn't work. And you know, the problem looks hard. It's a tricky problem, but mostly I think it was time. I think if we had had, you know, 200 years of academic tradition on this problem, I think we could have cracked it, but we didn't.

You know, it could have taken only 20 years for all we knew. But it turns out not to be a 20-year problem, as far as I can tell.

Um, yeah, you know, the basic reason is the orthogonality thesis, which I assume you've talked about on here. But very roughly speaking, just as there is no force that if you made an AI that, you know, only cares about making paper clips, there is no force that'll come in and be like and change it to care about humans, change it to care about a flourishing future for humanity. But just as there's no force that would take the paper clipper and change it to care about humanity, there's no force that would take an AI that cares about humanity flourishing and would make it care about something else. So if you could hit that very narrow target, there is sort of like no mental force in the universe that would like come in and push it off that path. It's just a problem of like getting it onto that narrow path in the first place.

Um, you know, I think basically any goal that you try to put in directly, you're going to get it wrong. You're going to miss something, right? And so you're going to be like, 'Hey, here's the list of all the things that we need in human experience for things to go well or whatever.' And, you know, then it turns out you miss one on that list. And now you have like the AI putting you in the same ideal day over and over. And you're like, 'Hold on. This actually sucks.' And it's like, well, you didn't put novelty in the list, you know. And that's sort of like a silly example. But if you did have this power to sort of like point an AI at exactly the list you wrote and have it do exactly that, only that and nothing else, then yes, you would have a really hard time writing down the list. But you don't actually need to write down this list and have the AI do sort of exactly that list as a separate question from whether you could. The answer right now is we can't give an AI a list and have it do exactly that thing. But the sort of dream is not that you like give the AI the list of like here's the good. The dream is that you sort of somehow are able to be like, look, you know, there's this project of like humanity, of flourishing, of, you know, there's this stuff I should be asking for if I was wiser. There's this stuff that I like would be asking for if I was more the person who I wished I was. There's some concept there of like this good stuff that humanity would iterate on. We wouldn't lock it in. Just as past generations trying to do exactly what they thought was good would later be resented by future generations if there had been lock in, we're going to try not to have lock in. And there's this sort of like whole messy concept that we're still trying to figure out ourselves. Can you sort of like also latch on to this concept and then like help us fulfill it, right? That's the sort of thing you would be trying to do. We don't have anywhere near that ability to sort of like find a concept in the AI of this form that would stand up under super intelligent optimization. We don't have anything like the ability to sort of like make that concept be the thing the AI actually cares about in its pursuits. We're not anywhere near close to this, but is it technically possible? Sure. You know, I think this is a little bit like alchemists trying to turn lead into gold. Were they close? No. Was there anything they could do to get there? No. Was it possible to turn lead into gold? Yes, we can do that with modern nuclear physics. We know exactly how to like slam the neutrons into the lead atoms to get them to turn into gold atoms. It's that kind of possibility. It's not the kind of possibility in striking distance, but it's the kind of possibility that's a physical possibility.

Um, you know, like one way you could try and do this is coherent extrapolated volition. Like a different way you could try to do this is what we used to call a 'do what I mean' system or a DWIM system. The basic idea here is there's sort of like there's some concept your own brain is tracking towards this good stuff, right? There's some concept your own brain is tracking towards like a future where humanity is flourishing and so on. That, you know, it flinches at, you know, like getting things that a wiser you would want but that the current you doesn't want. It flinches at the idea of lock in. It flinches at this, that, and the other idea, right? And sort of trying to get an AI that like latches on to that and then like helps you pursue it. It looks to me like it's theoretically possible. It looks to me like it's physically possible. This is a different question than is it practically possible with anything remotely like today's technology. But like if you had the textbook from the future, you know, if humanity got to use trial and error on this problem, if we got to like try to build a super intelligence, watch it go wrong, watch it kill everybody, reset, try again, try that a bunch of times, iterate a lot, learn what we were doing, learn to actually understand the code, learn what we can and can't predict about it, and then like wrote the textbooks for generations and then sent back the textbook from 300 years in the future that was like, 'Oh yeah, turns out here's how you get the AI to like actually wind up caring about the things that you care about in a way that sort of like helps everything be good.' I think that's the sort of thing humanity could get to in the usual process of scientific trial and error. With the issue in the case of AI being that we can't survive the errors.

Um, are you worried about that because it seems wrong to you?

Um, and like there's a difference from my perspective between the AI converging correctly on 'we should kill everybody' and the AI converging incorrectly on 'we should kill everybody'. I personally am pretty confident that the AI cannot converge correctly on 'we should kill everybody', right? And if you're like, 'Suppose the AI correctly converges on killing everybody.' I'm like, 'Whoa. Hey, sounds like the process you were trying to extrapolate did not actually converge in the correct thing here.' You know, I'm pretty confident that like murder all life is not going to turn out to be like the actual answer to the question like what is the good thing to do? You know, I don't feel like I know exactly what the good thing to do is. I can't give you the exact list, but like I'm pretty sure murdering everybody is not it.

Um, I am in some ways relatively happy having a no-human bias and in other ways not. You know, I think there's a fair bit to tease apart here. The thing where humans are like, if we found aliens we should not discriminate against the aliens. We should not, you know, take all the aliens as slaves and be like, well, because we're humans, we're, you know, the masters of the universe and it's our divine right to like take these aliens as slaves and not care how they're treated. I think a lot of humans would be like, actually, let's maybe not do the slave thing again, whether they be like other humans or other aliens, we sort of like shouldn't be doing this slave business. And in that sense, I'm like, yep, I'm going to be, you know, if humanity starts enslaving the little fuzzies, I'm going to be on the abolitionist side. And so in that sense I don't have a human bias. In a different sense, you know, if humanity encounters the aliens that are like, we are going to spend all our resources on building stacks of pebbles that have a prime number of pebbles in the stacks. And we're like, okay, that's a weird thing. We're like happy to trade with you. And they're like, we are going to take apart your star and sort it and like turn it into a bunch of pebbles that are, like, prime number pebble stacks. And we're like, hey, actually, we're kind of using that star, you know, no thank you. And then someone comes along, you know, and suppose that humanity has like a bunch of the stars and we meet these aliens. They have one star and we have a thousand. I'm not like, 'Oh, well, out of fairness, we should give them 500 stars to turn into like these heaps of pebbles.' I'm sort of like, 'Well, we can trade with them. Maybe they'll get wealthier as we both trade and we'll both mutually get wealthier and maybe they'll spend their wealth on pebbles.' But I'm not like, 'Oh, from an egalitarian standpoint, who can really tell whether spending the power of those stars on happy, fun people living fulfilling lives or heaps of prime number pebbles is what we should really be doing with the energy? Why are you having the human bias of like spending your allocation of the energy on, you know, happy humans having fun and making art and like living joyful lives or these pebbles?' I'm like, well, I'm just the guy who spends it on like people having a good time, you know, and is that a human bias? Sure. Right. So, in some sense, like this idea of like is there a human bias? I'm like, let's tease apart. There's sort of like no way that I think human experience is sort of like fundamentally preferred to alien experience such that you can enslave the aliens, but also there's like certain stuff that I care about, stuff that I want, you know, a universe full of like people having fun and making discoveries and having a good time where I'm like, that's not up for grabs. And if you're like, well, that's an arbitrary human thing. I'm like, yeah, it happens to be my thing. That's what I'm like spending the resources that like my fraction of humanity's resources on is like making things nice and like niceness is not what everybody in the universe is going to be trying or like goodness or funness or like a great time is not what everyone in the universe is going to be trying to pursue. It's what we happen to be trying to pursue and you know there's no fault in that just because you can't deduce this idea of goodness logically from the beginning of time.

You know, I think it all probably comes out in the wash if I had to guess. I think the sort of shelling line to draw is you draw the line around humans. And you say, you know, from our perspective, this is all sort of like wild fantasy land. This is like the alchemist saying like, if we could turn lead into gold, how should we distribute the gold among the population, right? And I'm like, okay, you guys aren't actually close to turning lead into gold. You have no path to getting to turn lead into gold. The question of like, how are we going to distribute the gold if we get it? is you're sort of like counting chickens way before they hatch, right? But you know what I would say if the alchemist set up their like automated gold dispensers, if they're like, 'Well, how do I actually allocate this gold?' I'm like, 'Oh man.' You know, a good starting point is you just give the gold equally to everybody and then you sort it out from there. And actually, this is going to be weirder than you expect because the world's going to change when everyone has access to the gold that you're fantasy synthesizing in your fantasy world where your gold transmutation techniques work, which is not going to happen. But yeah, like if you had this magical power to sort of like magically align a super intelligence somehow. I think the sort of shelling group to draw is humanity and just be like, hey, try and extrapolate, like figure out the concept that like humanity as a whole is kind of pursuing rather than like one particular human. Will that... like what about all the squirrels? Well, in so far as there's a lot of people who care about the squirrels, a lot of the sort of like extrapolation there will wind up also including some of the squirrel stuff, right? And in so far as many people could be convinced to care about the squirrels, if they sort of like thought about it more, it'll care about the squirrels more. In so far as people could be convinced not to care about the squirrels thought about it more, it'll care about the squirrels less. Like how will it care about the aliens? It'll care about the aliens through the fact that a lot of the humans care about the aliens, right? Like if there was a species, an evolved species that actually was super pro-alien slavery and they extrapolated their own volition and they were like, now we're going to enslave all of the weaker alien species that we can find, then you know, that's the sort of thing they're doing with their resources. And I would not say that they like made an error by their own lights to do that. I would say they're making an error by human lights and we should go, you know, free those alien slaves that they're making. And if that creates a conflict between us and some aliens, that creates a conflict between us and some aliens. But the fact that humans wouldn't want to enslave the other aliens is sort of coming from inside the humans. That is a feature of us that would sort of get reflected in there if you're sort of looking at the people. You don't need like the whole thing where you're like, well, if we just have it look at the people why won't it care about the aliens? That's a thing a human is saying and that means that like some of that care is going to be on display if you're looking at the human values and like if there is some aspect that literally no human cares about then like, yep, it won't get into the values but like good news, no human cares.

That's my guess. Yeah. Hard to say. I mean, one thing like it sort of looks to me like a nice stability property is that if this sort of if the super intelligence had been set off by the Romans or if the super intelligence had been set off now. Yeah. The cluster of Romans. Yeah. Once we make a lot of copies of you and whole brain upload them. But yeah, it looks to me like there's a nice property that this should have, which is that if it was set off by the classical Romans around the Mediterranean or if it was set off by the million copies of the Romans sitting right here once they've been whole brain emulated, that both of those should sort of probably converge to a similar answer, right? And that's a thing where I'm like, well, that seems like a decent sanity check. And because I'm a human who thinks that that should probably happen, you're going to see some of that reflected in there.

So, I just I mean this is still like fantasy land of like how are we going to equitably allocate all of the gold that we make from our alchemy where I'm like look the alchemists are not getting gold from this process. Which I'm just going to keep saying because we keep being often in this fantasy regime. My basic take is that there are a lot of hard problems in figuring out how to make the world better, in figuring out how people can still lead fulfilling wonderful lives even when there exist, you know, super intelligent machines that can do everything. In how you sort of like aggregate opinions. You know, there are sort of like all these like very hard thorny questions and it would be way easier to solve them with a friendly super intelligence at our back, you know.

I mean I don't like the phrase control because it sort of suggests that you make an AI that doesn't really want to do nice things and you twist its arm until it does. And I'm like, man, that seems really doomed to me. Like, I think what you've got to do here is like somehow have the super intelligences actually care about the flourishing of humanity. Like actually get latched on to that concept of like the good that a lot of us have and be like, well, I'm steering towards that. And then like are there a lot of problems with how you sort that out? Sure. Does that mean that the problems have no answer? No. I think like you can probably figure out that like if you imagine like being a super intelligence that actually cares about making the world better and you're like gosh I really don't want to like have this lock in. I really don't want to like, you know, step on all these toes and ruin all these things. You can still probably figure out that like hey well you know for starters maybe we should stop having all these kids die of malaria. That seems kind of needless. I can just go put a stop to that while we're still sorting these other things out. Right? There's ways to make progress even if you don't have like all the answers immediately. And like are there thorny issues still? Absolutely. These are the sorts of things that could be figured out by very intelligent systems that care. The problem is sort of like making them care. And like will they be like, 'Hey, it turns out there's no perfect solution.' Absolutely. There's not going to be a perfect solution. Like does that mean that they like fall over and can't do anything? No. It's possible to make trade-offs, you know. Will everything be perfect for everybody? Probably not. But you know, intelligence is the sort of thing you can deploy to like figure out how to balance those trade-offs. And you know, it's I'm like, man, if you can make the super intelligence actually care about humanity, a lot of these other implementation questions are its problem that it'll be better at than me if you can get that initial caring in.

Um, I mean what happens in real life is humans are like, don't worry we're going to have plenty of stopping points and they hit the button and then the AI kills them all. You know, like in real life, rather than in fantasy land where you've solved alignment, yeah, one of the big problems here is that people pass a point of no return either before they thought it was a point of no return or thinking it's going to be fine and then it's not fine. This is one of the things that makes this really, really hard is like, yeah, if you have the super intelligence that's doing like slightly the wrong thing that turns out cared in not quite the right way, that turns out cared about the wrong things and you're like, oh we would like to take that back, there's not really a lot of takesies-backsies here. Like is it possible in theory to make a sort of AI that allows taking things back and like trying a few times before you like get really deep into no return territory. Sure, that's possible in theory. That's another thing that we're like nowhere near succeeding at. Like ultimately will humanity have to get to a point where it crosses a point of no return? Probably. At least at some like meta level, you know, you could have the AI like maybe if the AI really cares, then like it respects like certain types of humans who have then like later gone on to upgrade themselves to be comparably intelligent, right? And it's like spreading out rapidly through the universe and like securing the stars against like being taken by other distant aliens that are coming towards us. And it's like, well, I got to go do this real quick. But you know, maybe you could make one that if humans also make themselves smarter, maybe they can like play some of those games in the frontier in meeting the other aliens somehow. Or I don't really know how it looks, but like maybe there's a way to make an AI that still like some humans still have a lot of sway over the future on a deep level. But like in real life, yeah, there's a point of no return. And in real life, it's really quite hard to get that right. And that's one of the things that makes this problem really quite scary.

Um, you know, the universe has created life in one spot. It would be a little bit surprising if it created life in only one spot. You know, the Fermi paradox or at least, you know, the Fermi observation sort of shows that life can't be too dense in the universe or we'd be able to see the effects of aliens probably we'd be able to see them, you know, harvesting energy from various stars. This looks to me like it basically means that if there are aliens, they're pretty distant. Distant enough that we can't see the results of their alien civilization sort of harvesting stars yet. We can't see the stars going out as they put Dyson spheres around them or whatever. Which roughly means something like there is no alien species that is 100 million years older than us unless it is more than 100 million light years away. You know, like what we observe when we look at the night sky and see that the stars are not harvested is not that there's like nothing in the observable universe. What we see depends on how far out we're looking. When we look within 100 million light years and see nothing, that means there's no alien species that's 100 million years ahead of us. When we look at a billion light years and see nothing, that means there's no alien species that's a billion years ahead of us. But probably odds are, you know, there's maybe some aliens like 200 million light years away who are only 150 million years ahead of us, right? And that would mean in that particular example, that would mean that there's, you know, 50 million light years worth of distance between our civilizations at the moment, of which we can maybe collect 25 million. And so that's, you know, 25 million light years worth of stars that humanity could collect and then put towards whatever purposes we like. And this suggests, you know, depending on how far away those aliens are and how much older than us they are, this suggests that like there's a decent chance that an expanding civilization meets aliens on its borders and then you know you have potential trade there. How far are the aliens, if there are any in this reachable Hubble volume? You know, hard to say. If I had to venture a number, I would say humanity looks like it is at least 100 million years slower than it could have been as a civilization because Earth sort of messed around with dinosaurs for 100 million years that sort of like went nowhere and then like an asteroid came down and it was like try again. And like then the mammal lineage was able to get further intelligence wise than the dinosaurs. We could imagine another planet somewhere where you don't sort of mess around with dinosaurs for 100 million years. You know, it seems like you probably could have gone straight from the Cambrian explosion into some lineage like mammals that had the potential to sort of build civilization and save 100 million years. So, like it would be surprising if we were the oldest. It would be surprising if there wasn't some other civilization out there that had a 100 million year head start. Clearly, they're not within 100 million light years. There's probably somewhere just like wacky, wild, not very solid guesses order of magnitude somewhere probably between 100 million light years away and a billion light years away. You know, if you split the difference, that's like 500 million light years away. Or if you split the difference on a log scale order of magnitude, that's like what, 330 million light years away. So if I had to pick a point estimate of where the aliens are, it would be somewhere in that order of magnitude range. If you're off by, you know, two orders of magnitude there, you're looking at the aliens being 100 billion light years away, which is just outside the reachable universe. And this is easily the sort of calculation that could be off by two orders of magnitude. So I think it's probably somewhere between they're 100 million light years away and there's none.

Um, I think that that can't really explain the great filter because like you would it would be pretty surprising if like to answer the great filter, you would need to answer why a ton of alien species all have the same issue, right? Like could one alien species AI be like there's nothing I have to do? There's nothing I can use energy for to get more of my tasks achieved? Sure, maybe you could have one AI like that. Could they have a trillion AIs like that? Are there a trillion civilizations? Is it like only a trillion to one that there's some AI that's like actually turns out more energy can do more things? Like it seems like a pretty easy call that most intelligent stuff will have ways it prefers the world to be such that it can like redesign things, you know? Like it would be an easy call looking at humanity 100,000 years ago to be like when they grow up the world around them will look more like designed than like it's still just a bunch of replicators around them. You know, they probably won't live in the jungle. It probably like when you look around them, you'll probably find a lot of things that they've arranged precisely rather than still just the chaos of the jungles. If they manage to get intelligent at all. Right? That prediction is way easier than predicting that they will live in houses with books, right? Similarly with AI, it's hard to predict exactly what they'll do. It's hard to predict exactly what they'll be going for, but the prediction that like they'll be redesigning the universe around them somehow. There'll be some way that they prefer things to be rather than just like stars dumping all their energy into the empty night. There's going to be at least some of the AIs that have that level of preference around their universe. And so no, AI can't be an answer to the Fermi paradox because after the AIs kill the host species, they then go on to rearrange the universe and we should be able to see that.

You know, I think people get really into the idea of a causal trade because they think it's like sexy and interesting or something. I think you can often do a lot better by thinking about causal trade, you know? Like you want to know what happens like what are you most likely to experience after the super intelligence comes? Like don't think about all this crazy causal stuff. Just think about those like distant aliens 100 million light years away or 200 million light years away or whatever. Will some of those evolved species wish to buy copies of the humans? Maybe, you know, if humanity succeeds, if we somehow succeed at the AI alignment problem, we somehow start traveling the stars with, you know, super intelligences helping us with technology and helping us govern things and, you know, in 200 million light years from here, humanity encounters some alien super intelligence that is horribly

misaligned to its host species. It's like, 'Oh yeah, I killed them. I'm turning all of the stars in my volume into paper clips.' But I happen to have copies of the biological creatures that created me. Would you like them? I think humanity would be like, 'Yes, absolutely. We would like, you know, to recover some of these aliens in, you know, at least in simulation or whatever and see what they were like and maybe see if some of them want to be friends and that sort of thing, we'd probably pay for it.' You know, if it was like, 'Well, here's the costs that it was to me to preserve them. Will you recoup my costs then give me some benefit, you know, so we both mutually benefit from this trade?' I think humanity would be like, 'Yeah, we would take those alien simulations.' And insofar as that's a predictable property of evolved creatures, you don't need any acausal trait about it. The AI could be like, 'Well, I'm putting all the humans on ice or I'm scanning all of their brains and I'm going to sell those to aliens.' And so like, what happens after AI? If anything happens at all after AI, maybe you wake up in an alien zoo, right? And then we can all debate whether it was true that everyone died as my book title said. But like, yeah, you don't need to get into any of this acausal trade stuff. It's sort of overdetermined that you shouldn't be messing with the AIs even if there's going to be someone trying to buy copies of your brain in the distant future.

I don't think the decision theory research as a whole is a waste of time. It's just that it's not about like trying to get the acausal trades to go well with distant aliens or something. The way that that research matters is something that a lot of people have sort of like not been able to wrap their heads around and maybe it's like a little bit too abstract.

When you're sort of like facing a technical problem and you don't really know how to make progress, one good useful way to make progress is to find the places where you're confused. Find the places where your theory breaks down. Find the edges, right? Like if you were, if you know Newtonian mechanics and you're trying to invent general relativity, but you don't really know that you should be inventing general relativity. You don't really know like where you're supposed to be looking. Like looking at the stuff that your theory can't capture yet is a good way to sort of like blow the whole theory open, right? And you know, Lord Kelvin famously said like this physics stuff is all mostly handled except for some issues with light, you know, which I'm sure we'll figure out shortly. And we sort of didn't figure them out shortly, but it was true that by looking at light and looking at the odd behavior of light, we were sort of able to like crack the case open and we're sort of able to be like, 'Oh, actually on Newtonian physics when we have this like weird setup where we're like shooting light beams horizontally and vertically, those light beams should not be perfectly synced at every point in the seasons of the planet as the Earth is like moving at different, like, moving in different fundamental around the sun, right? It's in a circular path. So, it like traverses all these directions and one of those should be sort of like catching up to the light and one of them should sort of be receding from the light. You should sort of be able to like see that effect.' And then we couldn't see that effect. And that sort of like blows the case wide open and you're like, 'What the heck's going on here?' And this like leads you to like Lorentzian mechanics and like this leads you into special relativity which then leads you into general relativity.

With our theories of intelligence, there's a number of places where the current best academic theories of intelligence don't, where they break down where there's some anomaly like with light. And there's a handful of places where those theories break down. They break down around self-reference. They break down around decision-making. And these are places where if you sort of poke at them, you can maybe blow the theory wide open. You can maybe figure out the next theory of intelligence that you need. You can maybe like figure out the pieces of intelligence that we sort of like don't understand but we need to understand to do alignment, right? I think this was basically just a good strategy especially, you know, over a decade ago, well before the LLMs were even a twinkle in OpenAI's eye.

And you know, for a lot of these problems, you can like give these thought experiments, right? Where you can sort of, it's like if you're doing the experiment with a light and you're like, 'Well, you know, imagine that the world was filled with like an ether and that ether was like what was used to transmit the light waves. Well, then wouldn't it be the case that like blah blah blah' or like imagine that we had, you know, a series of rods all throughout space and there's a whole grid of rods, right? And you can imagine someone being like, 'These crazy guys are talking about ether and they're talking about these big grids of rods. Why do we, we're never going to need a big grid of rods in outer space. These guys are nuts.' It's like, 'No, no, no. It's like doing something else.' That's like, we're sort of like trying to pick apart the edges of the theory to figure out what's going on. That's sort of where a lot of the research into self-reference, the research into decision theory came from. And like, yes, you can come up with a bunch of examples of like, 'Well, what if you had an AI that was like trying to make a chocolate cake and every day blah blah blah blah blah.' But, you know, the issue is not that like the AIs are currently really bad at making chocolate cake and that's why we're going to die. It's like the issue is that we have no idea how intelligence works. We have no idea what this stuff is doing. And our theories of intelligence are not up to the task. And we're trying to like make the intelligence that are up to the task.

Humanity didn't go down that route. Humanity never really figured out how intelligence works. Modern AI did not come from any better understanding of intelligence. It came from learning that if you throw more computing power at it, nobody needs to understand anything about what's going on in there. It just happens to get smarter with more compute and more data. But like does that mean humanity should totally abandon the path of figuring out more about how intelligence works? I mean it's maybe too late now to get enough understanding of that to solve the alignment problem. But if we were still trying to solve the alignment problem I still think one should sort of pursue these theories of intelligence and you know I still think that was a reasonable thing for me to be trying especially 10 years ago. Although you know knowing what I know now I would probably try earlier and harder to be like, 'Hey guys, like one of these paradigms where we have no idea what we're doing is not going to lead anywhere good. Let's not go down this route.' Which we didn't do and maybe should have.

Super hard to say. You know, it could be that the next generation of AIs are just barely smart enough to automate AI research. And you know, they're still pretty dumb, but if you run a million of them in parallel for the equivalent at the equivalent of like 100x human speed and you run a ton of those in a huge data center, maybe that's just barely enough to build, you know, to sort of close the loop on automated AI research. And then maybe things start going really fast after that. Or maybe we hit the wall that Gary Marcus has been predicting every year for the past 5 years. And maybe LLMs finally hit their limits and we need to wait for a breakthrough. And then you know it takes 6 years to get a breakthrough and six more years to exploit that breakthrough to the point of super intelligence. Right? So do we have 12 months or 12 years? I don't know.

Yeah. You know, the proposal is shut this all down. We're not close to a solution. We're not, you know, we've talked a lot about like how could you align things? How would you align things if you had this power to align the AIs? We don't have this power. We're not close to this power.

And there's, I don't know, there's sort of a lot to unpack here, but one of the big impetuses behind the book is I started talking to politicians in DC and I've been talking to people in the AI business about these issues for, you know, over a decade. And they often don't want to hear it. They have all these objections like, 'Oh, well, won't the AI all automatically turn out nice for this reason? Won't it always have to listen to what we say for that reason?' Blah, blah, blah. And I would have these like big long arguments. And then when I first started going to talk to politicians, I would sort of lay out the basic issues of like, 'These guys are trying to make machines that are radically smarter than any human. We're growing these things. We don't understand what's going on inside them. We have no ability to sort of like make them actually care about what we want them to care about.' It's possible in principle for these machines to get to the point where they can go toe-to-toe with humanity as a whole, not individual humans, but like outstrip humanity at the ability to make their own infrastructure, make their own technology, make their own civilization. This is a crazy thing to be rushing into. Right? And I went into these conversations prepared for a really long back and forth. And a lot of these politicians were like, 'Oh, that's crazy. We shouldn't allow that to happen.' And I was like, 'Yeah.' And also what happened to the three hours of back and forth and all of these, you know, like what about this, what about that, right? And it sort of turns out that people have a harder time understanding something when they are being paid a ton of money to not believe it, you know? And so it turns out that actually a lot of people really can just understand the issue of like, hey, maybe if you're racing to create a radically smarter set of machines that nobody understands, that just might go wrong. Turns out that's kind of easy.

And so once we realized a lot of people outside of Silicon Valley sort of could grasp this argument and that politicians in the wake of ChatGPT were starting to notice that AI was real, that's when I sort of went to Eliezer and was like I think it's time for the book. And yeah, you know, it's mostly laying out the problem and like I said, my take on the solution is we just need to stop the race to super intelligence and that doesn't mean we need to give up on the current chat bots. You know, ChatGPT is not about to end the world as it is today. This sort of isn't, you know, I'm not here being like also we need to stop these chat bots from being in schools because they're going to affect kids learning. That's a separate problem that people are going to need to figure out how to deal with it. I'm sort of here being like, look, if we keep racing towards the super intelligence, it kills us. And so we need to not keep racing towards the super intelligence.

And I think that plan has a chance. I think that a lot of why the world has not been stopping the race is because the world has not been believing in the race. And that as we have seen world leaders start to notice AI more and more and start to notice how crazy this whole race is, we've started to see them react. You know, just in the last couple of weeks we've seen the sort of US government be like, 'Hold on, you've made a super cybersecurity hacker and you can't make it help adversaries. Like you say that like we've been showed that it can be jailbroken and you say that you can't fix it. What?' Right. And that's the appropriate response, you know, being sort of like surprised and freaked out and like, 'What the heck are you guys doing over there that you can make a cyber weapon and you can't control who wields it?' It's the appropriate response to be sort of like a little shocked and horrified, right? And we can have a whole discussion about whether the particular ways that they try to like respond with that shock and horror, whether that's, you know, going to be good or bad and this that and the other, but a lot of people have long said that stopping is inevitable.

And I liken it to a case of like we're in a bus that is racing towards a cliff edge and the driver is asleep. And I'm like, look, you know, maybe when the driver wakes up, they'll be like, 'I love taking buses off cliffs,' right? But let's not give up until the driver's awake.

You know, I think various people, there's been various signs that conversations like this have been had and you know even going back to the Obama administration there were I think there were some briefs about artificial intelligence.

You know, I would have to like go look through a lot of things people have said publicly. I think Elon Musk has said publicly that he like tried to talk to Donald Trump about the AI dangers. I'm not sure that's true, but whether he said so publicly or not, I would be a little bit surprised if he hadn't had that conversation with the president.

Yeah, I don't have good reads there and if I did, I'm not sure I should say them on podcasts.

I did have to like write a draft that served as the catalyst for him rewriting a much longer and better version that I then had to cut down. And we did cycles of this and then it's not like it was converging. It's just like we had a due date so we just cut it off.

Absolutely. I mean there's, I think there's four times as much text as in the book in the online resources which can be reached by the QR codes which I'm not sure how many people are reading but they're useful for when you get into Twitter fights and people are like, 'Well, you guys have never thought of X,' we're like, 'Actually...'