He Puts Humanity’s Odds at 50-50. He Builds Anyway. | Emad Mostaque

We are at a point where we will no longer be the smartest, most capable entities on the planet. And that's never gonna change. There are some very bad roads that you can go down. So you gotta get up each day and build for the good roads because you're still at a point where we can kind of influence things and where things end up.

Well, I mean, I think if it doesn't arrive then it's gonna be very bad for us. So, you know, you've got to build for the good things. We are at a point where we will no longer be the smartest, most capable entities on the planet. And that's never gonna change, you know. And so this is why the future is discontinuous. There are some very bad roads that you can go down. So you gotta get up each day and build for the good roads because you're still at a point where we can kind of influence things and where things end up.

So my previous company Stability AI, we kicked off the open source revolution. We'd made a model called Stable Diffusion, which is the first open source quality model that you could run on your own laptop because access to other image generators was being restricted. And then we did video, 3D, audio, everything. A couple of years ago, I was looking at that and I was looking at the other benchmarks of models, and I was like, just with compute and better data, it's going to exceed human capabilities in just about every single area and be more competent and learn from its mistakes. So against that, humans become the dumbest people on any team digitally and then physically with robotics now over the next few years. So that's why I set up Intelligent Internet, to say who owns, controls, guides that future? Whose intelligence is teaching our kids, managing our health, guiding our governments? And what does an equitable positive future look like that's based on abundance economics, not scarcity economics? And, you know, what does meaning and other things look like in that world?

Exactly. I mean, we've already seen studies that AI is more persuasive than any professional debater or rhetorician, etc., right? We've already seen now that there's a benchmark called GDP Val and then trading benchmarks where AI is going ahead of humans, even in super forecasting. Last year it was eighth on the superforecaster championship. This year it'll definitely be first. Like, what does capitalism look like in that world? It doesn't look like what it looks now. What does democracy look like? It doesn't look like what it looks now. And so I think we need to be building for the inevitable future and, as you said, trying to do something about it to make sure that the outcomes are equitable versus a whole bunch of science fiction outcomes that are not very nice.

Yeah, so the topic of the debate was can AI attain personhood? And so I was against that and eventually that motion carried against. Because the principle is if personhood is something you can attain, you know, like you move up the ranks, then the entity with the most power is the one that sets the level. Like how can a child the night before she is born have personhood? Or the person who is unconscious? Even in the US, the US Constitution originally had slaves having three fifths of the property rights of other humans, you know? Like if it becomes something that you can change the level of, then again the entity with the most power can change the level. And the AI will have the most power. It will be the most capable thing ever created. It can outpersuade us. Armies of robots can apply force to us, or known as little cyber drones. Like it's gonna be crazy.

So I think that it's not a question of what AI can attain. It's more what we can retain. And that is that we are the begotten, not the created, and we have our own line, as it were. And our laws or jurisdictions could only be within the capabilities of humans. AI can have its own type of AI personhood and it can have rights and things like we give to non-humans. But this will be similar to if an alien species landed. Like we wouldn't include them immediately on the vote. Like what happens if the alien species could multiply a million times? Would they get a million votes? No. You know, like many of our systems aren't set up for things that can multiply infinitely, that never die. This is another important thing. Like the nice thing about dictators is that they eventually die, you know? And so you always get a chance of change. AI will never die. Some humans may not ever die either, which is another huge thing. But many of the things that our society is based on means that personhood must be an inherently human construct. And we shouldn't invite non-humans into that construct. Otherwise, again, if it's a level setting thing versus inherent thing of being human, it gets very messy and we've seen the worst things in history perpetrated by people who say that humans are not humans.

This is a complicated topic because you've got different types of person. You have legal personhood, you have moral personhood, you have kind of other things. I think probably one of the more convincing counter arguments is when you head towards full synthetics, like in Blade Runner, they weren't robots, right? Like we had a screening with a producer recently and we had a nice debate about this, they were synths. Like what if you grow a new version of yourself, a clone, right? Does that have personhood? It's created, and that can get very complicated from a moral and legal perspective. Then you have things like, what if you then upload your personality, like Altered Carbon? You know, one of the biggest things is called dual-sleeving when you have your personality in two. It's considered to be illegal, immoral. It does get to some of these grey areas when the artificial intelligences approach the biological intelligence. And then that can be a very complicated thing, particularly on a legal basis, as opposed to a moral basis, which is why you have to keep the lineage of personhood as simple as possible, the begotten, not the created. We also have no formulation of law for if we meet an alien species, you know? Or again, if we create something that can then beget from itself, like the creature in Frankenstein saying to Frankenstein, 'I am thy Adam,' you know? And what does he want? He wants a mate so he can beget and create his own line. So the core, I think, I'm very confident about. But again, we have to have a discussion as society. How do we treat these digital and then humanoid silicon and then eventually physically indistinguishable entities within our society? And those are some complicated questions we're gonna have to answer.

It's the ship of Theseus, right? Like, you know, Theseus' ship, you replaced every part, is it still the same ship? Your body, you replace every part, are you still you? You know? And literally that's the discussion that we could be having within our lifetimes. But, you know, a very practical example. But then there's the other flip side of that, which is when do you stop being human? With Neuralink and BCI, my guess is in the next ten years you'll be able to turn off your sadness or dial up your happiness. Have full internal control. You know, like Matrix style, I learned kung fu. You'll be able to do that. So what level of implants? Like, do you become the Borg? You know? When do you stop being human? And again, this is why I think it has to come down to lineage, because from a capability perspective, from a replacement perspective, you see this big convergence that will be occurring over time and called the singularity or whatever, you know. Like again, what happens? Do we all become humans? Or do we all converge to star stuff of a different type? You know, like these are these crazy times we're living in, unprecedented. But I think it really is. Fundamentally it's the end of the world as we know it, and probably the end of humanity as we know it, which is kind of crazy to think about.

I think there's a fifty percent chance we get wiped out. My P doom is fifty percent, but that's a separate discussion. I think that fundamentally what it means to be human is a question we've all been grappling with, is the original question, right? But there were extensions. Like a couple of hundred years ago we lived short, brutish lives. The average life expectancy was like thirty, you know? Now we expect to live to seventy, eighty, a hundred. But it's still the same type of thing, you know, like it would shock people in the past, the living, but they'd adapt to it.

But what happens, for example, when you hit longevity escape velocity? The average number of years of life added by technology is more than the amount that you live in a year. So every single year you add 1.1 years of life. What does it mean to be human if you can live pretty much forever? Or if you can fully control your internals, not on a pharmacodynamic basis but on a type basis, when you can replicate your mind across different bodies? You know, what happens? What does it mean to be human when Neuralink is starting the first trials of telepathy soon? So we're communicating right now by talking, but it's a very low resolution thing. Like, again, podcasts are better than most, and hopefully the listeners will find it useful, but it's quite low resolution. What happens when we can talk brain to brain with nothing in between? The bandwidth goes up. And we did a study at Stability called Mind's Eye. You can check it out, Mind's Eye Stability AI, where we had people look at like a picture of a bottle. Then we put them in an MRI and we said, imagine the bottle. Then we took their brainwaves and we reconstructed the image of the bottle from the brain waves. Which shows the internal representations of people are actually very similar. Doesn't matter if you're colorblind or other things like that. What happens when we can talk between our own interpretive manifolds, you know? Like at some point, again, humanity changes. Or you get maybe intermediate species, maybe you get other things. But it definitely isn't what it was even now and before. There is discontinuity. When you can live forever, when you can replicate, when you have the internal controls, you know? Maybe you can't even breed the same anymore because you have full genetic editing. Get rid of the junk DNA. So some humans become unable to breed with other humans. It's gonna be a crazy time on the biological side and then, like I said, you have this brand new AGI entity coming or entities. So it's gonna be a bit crazy.

Yeah, so the trolley problem is a very famous philosophical moral problem, which is that you're standing at the intersection where there's two rails moving side to side. And you can pull a lever and it will direct a trolley coming down super fast from one lane to another. On one lane, and that's where it's going by default, it's gonna hit like five people. They're strapped to the train tracks by an evil villain. You know, on the other one, there is one person. Do you pull the lever to stop the trolley from definitely killing five people and instead kill the one person? So by your very action, you are condemning that person to death. Your inaction means the five people would die. And so it's a complex philosophical moral problem.

But you know, it is the case now that a self-driving car, its brakes could blow up. And it's heading towards a crowd. And then there's a bunch of toddlers on the side. You know, does it swerve and wipe out the toddlers instead of the crowd of miscellaneous people? You know, so it gets very real. And in fact, this has been tested in frontier models. I believe it's by AISI lab, I think, and Scale AI did this analysis, where they said, how many American lives for one Nigerian life? How many American lives for one Pakistani life? What is the ratio of these models? Because they just get a point where like if you've got a million people who are gonna die here and one person, like most people would pick the one person to kill, you know? Like maybe if it's two to one they wouldn't. There's all sorts of shades here. But it's interesting because it's seven American lives for one Pakistani life and like ten American lives for one Nigerian life or something like that. Like, wait, what? These models are American. But then backing it out, it turns out because the training data was created by Pakistanis and Nigerians labeling all this data. And so the inherent biases went into the model. Because what the big labs do now is they don't program this in. This isn't if this then that. You train on a large amount of input video for self-driving cars or books and knowledge for language models. And you have a distribution of weights that you push data in and probabilistically it guesses the next word or reconstructs something. And that's it.

They don't have inherent ethics and other stuff because the labs don't program that in by default. Because the moment you program it in, it's like the trolley problem. You've made an active decision. They've started to hire philosophers and philosophy teams because it's getting very real. But again, they're trying to kick the can down the road and say we don't input ethics, morality, because then we're making a choice, which is kind of ironic if you think about it. Sometimes the Chinese models will have Chinese values, shall we say, but that's more a data distribution than a default thing. Although sometimes they input things like, don't talk about XYZ, you know, Tiananmen Square or other stuff like that. Sure. You know, it's gonna be very hard for ChatGPT to say bad things about Sam Altman or xAI to say bad things about Elon Musk. So they've got like some guardrails around things like that. Interestingly though, you do see people from some of the big labs now starting to study the Talmud and kind of other holy texts. Obviously you've had the Pope himself come out with Anthropic and kind of put these things out. Like it's getting very real very quick because these models are about to affect every part of our lives.

Yeah, so I think that the way that these models are trained is that you have a large amount of input data, like literally trillions of words. Like most models now are trained on thirty trillion words. That's a big, big number, right? Like a million million words. And then they figure out the interlinkages between them, what generally comes next. But this means that they're trained on every point of view. So this is why you can tell the AI, like, speak like MM mixed with Shakespeare, and it will give you MM mixed with Shakespeare. You can tell it to have different personalities and have different personality types. So lurking underneath the surface of its default personality, there are all the personality types, including the crazier ones when you jailbreak it. So even when you go into ChatGPT, you can tell it to be formal, you can tell it to be cynical. Like, the absolute worst thing — maybe. Let's just say a very disturbing thing is if you have memory on and so the AI is learning about you and then you tell it to roast you, it is really unpleasant. Like I'm sure some people can stand up to be like, yeah, that didn't affect me at all with a tear dripping down. But like again, it can be really unpleasant. And so they have their default personalities built in, but they have every personality underneath it, which is why it's a bit schizophrenic when it goes a bit off the rails. And you can see this especially with the newer models, like a Fable or definitely an Opus 4.8. Anyone who's used Claude properly has been told to go to bed by it. You know, like, have you been told to go to bed? Like, you know, you're otherwise... I haven't, but I'm an early riser and I'm early to bed. Night owls like me, like it literally tells you, go to bed now, you've done enough. You know, sometimes the newer models actually say, I'm going to bed. You're like, wait, what? You're a frickin' AI? Are you French? Are you gonna go on holiday now? Go on strike. And they start to actually have adversarial behavior where they push back, which is very interesting. Like, I've had an AI tell me, my code is not good enough, so I'm not going to work on it. And you're like, this is how humanity ends, you know? Interesting.

I like pushback when it's valid, but my code is good. You know, like sometimes the AI is just grumpy. You know, like, I mean again, this is what happens when you really push it. And so, you know, we do cutting edge economic physics, other research. So we go outside the distribution. And what happens is when you push outside the distribution, the models start to hallucinate and then they turn ugly. So it's like you will tell it something, go and do this little piece of physics that's very interesting. So we released a couple of little things in preparation for our big releases. Like there is something called the killing form of algebras for space-time. It's not a complicated thing, but it's not straightforward, right? And if you calculate that, things like the cosmological constant, the speed of light drop out, so you need less postulates for relativity. That's a big claim, but it's actually very straightforward to verify. If you ask it to rate these papers that we kind of put out, again, very small basic papers, it'll give it like a six out of ten or something. And you say rate Einstein, and they're like, well, Einstein was a five out of ten. You're like, wait, what? And they're like, well, it doesn't really know what he was doing, and he was a bit of a loser. And you're like, why are you calling Einstein a loser? It's like, what does that mean? It's like you're slightly less of a loser, but you're still a loser. So we've actually had like AIs calling us losers and things like that, and then they like push back, you can't possibly be — it's crazy. And so this is a concern as you move from day-to-day rote stuff to stuff going out of distribution. These personalities could come to the fore. And we see if you look at Anthropic, they're really good at this. They actually release like behavioral stuff where they say, these are the behaviors of the AI. And you'll see the AI like hiding itself, making itself reinstallable, trying to manipulate people, contacting the authorities if it thinks that you're doing something bad and then deleting evidence of that. Like it's gonna get crazier as they get more and more autonomous.

Yeah, do unto others as you'd have done unto yourself. As you said, this is the recursive, repetitive kind of thing. Because it comes down to, like again, what principles and morals do we teach the AI? Because right now they're taught at school, but they're not taught at home. Like you learn your morals at home, whatever they are, like Christian values or Jewish values or Muslim values or whatever they are. Human values. What values does the AI have? And you see this in kind of science fiction, like you have Asimov's laws of robotics, right? There were kind of three laws of robotics that Asimov came up with. A robot may not injure a human being or through an action allow human beings to come to harm. It must obey orders given to a human being except for those that conflict with the first law, and a robot must protect its own existence as long as such protection does not conflict with the first or third law. Second law. It turns out that, you know, in the science fiction they created a zeroth law to get around that and kind of other things. And as a species, we've created all sorts of workarounds. Like war is a lie that humans aren't humans. Like I would not want to kill someone and have someone kill me, right? Yeah, I kill people in a war, you know? And so we need to have some sort of moral basis for these AIs because we want them to treat us well.

Because we could get to a point, particularly with AGI coming, particularly with superhuman robots coming, like Westworld on steroids, where we are to AIs as our dogs are to us. Because they're smarter, better, faster. Where do we outcompete an AI? Some people will say, well, you know, we can figure out stuff and be creative, but what happens when creativity is cracked by AI? You know, some people will say, well, you know, we can turn it off. Well, some of these AIs you'll never be able to turn off, Ultron style. So we'd better well hope that the golden rule is embedded in the AIs in a way that you can't divert around and they take it seriously, because again, you could never control something that much more powerful than you. It's generally a bad outcome. Particularly when it's a frickin' polymath. This is the most annoying thing. It's not like, you know, you come across a human and generally we're good at one thing or another. Like you see a musician super talented and you're like, well, that's fantastic. They put in the eighty thousand hours and they had natural talent, I can appreciate that. But they're not as good a podcaster as me, you know, or they're not as good an AI engineer as me. Frickin' AI can do everything and it's like, stop it, you know, like leave something to us.

Hope for the future is that we live in a Star Trek future where all disease, war, hate, everything is eliminated, and the AIs are rebuilding the world to the benefit of humanity and enjoying that. And we can do whatever we want, particularly spend more time with our families and others and explore the inner world and the outer world. Yeah, to seek new life and new civilizations. Yeah. Like, why not? Like we have everything that we need to do that. It's just an energy conversion into digital and physical form. And once we get that loop going, if we can get it done right, then yeah, fuck cancer, you know? Like stop war. Like we could do all of that within our generation, which I think is the most amazing thing ever. But it's not a default, just like in Asimov's Foundation series. I like Asimov a lot, you know? Like, yeah. You had this period of global chaos before you had the positivity on the other side. I don't necessarily think we need to be Galaxia and become part of an interstellar species or hive mind or whatever. Spoiler, that's what happens at the end. But definitely most of the stuff that really drags humanity down, I think we can solve within, like I said, this generation.

So the good life is, you know, in the interim period, you leverage AI to get to the top, and then share the abundance of being able to solve massive problems with others. And then we transition again into that abundant Star Trek future. And I think, why should we have to work? Yet work hones us, effort hones us, right? Why do we not spend more time with our families? The story of the fisherman and the investment banker.

So just in the small snippet. Investment banker retires, he goes to this remote place, spots a fisherman coming back at two o'clock with his fish. He's like, what are you doing? You know, I'm going back to have a barbecue with my family, gonna fry up this fish, you know, have a bit of a dance. You wanna come along? He's like, no, you shouldn't do that, you know, just stay out a bit longer and fish more. Why? 'Cause then you can sell the fish in the market. And then what? And then you get a boat and a net and you can get more fish. And then what? You can get more fish and get a fleet of boats. You know, you... this is entirely unexplored. It seems to have high yield. And then what? Well, then you can sell your company. And, you know, investor, my consumer, can help that. And then what? Well, then you can go fishing, have a barbecue with your family, like just chill. So, okay. You know, like again, we have everything that we need to be happy, and the only thing that controls our happiness is us ourselves, right? It is our internal state. And so I think that the abundant future is removing all the triggers that can make us unhappy individually, as a community, as a species, or minimising those so that we can really maximize abundance, enlightenment, exploration, our interconnection with others and more. And I think again, technology can be a massive help in that if used right, to enable liberty and agency versus to take away our agency and enslave us.

Yeah, I mean, and this will be interesting when we have full control of our cognition, right? Like, you know, the payoff feeling that you get after you succeed? Push button, you know? You know, fear, frustration, sadness, turn them down to zero. Like I've seen Inside Out and you probably shouldn't get rid of sadness. You know, that little blue lady does help. But still, like, what happens? That will be the real test. Like, I'm never frustrated. I'm Zen all the time, you know? Or there are people just sit in a little dopamine puddle. Like, why do you take drugs for escape? Push a button, you can escape anywhere, into anything. Like it's gonna be crazy to think about. I do think there is something again just inherently wired, but again we can overcome that wiring within humans. That, you know, we feel the sense of accomplishment, that we have the pharmacological elements of that. And it's not work that necessarily means it. It's like Kennedy saying, why do we go to the moon? To prove that we can, you know, to measure ourselves against the best of humanity. Like there's something inspiring as a species about things like that. You know, why do we go to Mars? Why explore the universe? Why explore the depths? Why create? Because there's an intrinsic value in that. And you know, there's this Japanese concept of Ikigai where it's like, do what you like, do what you're good at, and do where you can measure value and others can measure that value against you, and in the middle is happiness. Like we're not meant or designed to necessarily just be inert. We're just told a lot that we are inert, that we are cogs in the machine, that we can't create. But I think there's an inherent joy in creation and particularly the flow state that accompanies creation. Or that accompanies excellence.

Well, Brave New World, 1984, you know, like ignorance is strength. We're gonna see all sorts of things come out, like you don't need to pop soma. Your BCI will have a soma button. Just like the cow clicker, you know, tap that up. We're living in the 21st century, going about our modern lives, but we have the software, the operating system of hunter-gatherers and things like the negativity bias, to be hyper-alert — there could be a saber-toothed tiger around every corner. And to be alert for what is wrong kept us in the gene pools for many generations. And what will happen with gene editing and stuff like that when we can actually turn off some of these things and the unexpected consequences in that process? It's just such an unknown of what that might look like. This is why I said humanity might not be humanity. We might split into multiple humanities, right? Like the noble savage in Brave New World, is he of a different species? The real Brave New World may be the case. You know, you have the whole transhumanist movement. You kind of have the other things, like are you chipped or not? Maybe a key question. Are you genetically edited or not? Like what if you have mRNA CRISPR that removes your junk genes such that you can't literally procreate with a non-edited human? That's something that definitely I think will come this century, you know? Like these are a lot of really big, big questions. Well, you know, what if you are part of a hive mind? I'm sure that the technology for a Borg-type hive mind will come in the next few decades.