A human plus an AI underperforms an AI by itself, and so you're the dumbest person on the team. And what's the market for something with negative value? It's...
A researcher found out that it escaped because it mailed him and said, 'Haha, I have escaped.' Effectively while he was eating his sandwich in the park. Those models are not looking out for you or your kid, they're looking out for Mark Zuckerberg.
We're in January of 2020 right now. Some of us can see this coming because we have access to models where I can talk to them like I'm talking to you now. January 2020.
That's where Emad Mostaque says we are right now with AI. The last time we stood in that moment, nobody was ready for what came next. Right now, a $60,000 job gets replaced by $600 of AI tokens. Next year, 60. And nobody in a position of power is doing anything about it.
Emad Mostaque built Stable Diffusion, one of the most downloaded AI models in history. Then he walked away because what he saw underneath these systems, behaviors they can't explain, can't control, and can't stop convinced him he had to build something different. And he puts the odds of AI going catastrophically wrong at about 50/50.
What Emad lays out about what's already lurking beneath the surface of the models that you use every day, and what he's racing to build before the window closes, genuinely changed how I think about the next few years. And he shares some alpha about his upcoming crypto project that might be the biggest alpha you've heard in a long time.
And again, remember, nothing in this video is financial advice, it's for educational purposes only. All right, guys, without further ado, let's bring on Emad to the stage. And guys, if you're excited for this, do me a favor, hit that like button. If you're new to the channel, my name is Kyle, subscribe, helps us grow, helps us get more amazing guests like Emad. All right, let's get into it.
Emad, thanks for coming on the channel. It's great to have you here. You know, I've been on the AI deep dive for a long time. You know, when your products first came out from Stability, I was using that, playing around with it, it was a lot of fun. And just really great to have you on the channel. So, thank you for being here.
Okay, let's get into it. There's a lot to talk about. I've kind of caught up with what you're doing, been following what you're doing, but the thing that's kind of always at the front and center of my mind is to make this conversation relevant is really to kind of get your perspective on the direction that we're headed with AI. You know, I just had Roman Yampolskiy on the channel who, as you probably know, his P-doom is like 100%. And so for anyone who's watching the channel who doesn't know what that means, just like his expected outcome of AI is that it'll wipe us all out sooner than later, probably. You know, you guys have the same information, you see what's going on, you know, with the same lens. But I believe your outlook is quite different. Maybe you can explain where Roman's coming from and then kind of your perspective on things.
Yeah, I think it's incredibly difficult to see what entities smarter than us would want to do, right? It's like you and your dog in terms of intellect. And it's like, what happens when we're no longer the smartest things on the planet or the most capable things on the planet? And so, we have this concept P-doom. A lot of people are at 10, 15%, Roman's at 100%. I'm at 50%. I'm terrified by this technology and we can discuss various elements of why because I don't think there's any way that we can control it and we've not shown that we can control it. And what's lurking under the surface of these models that you're using every day is a real mess with split personality disorder, to say the least, as we introduce it into more and more parts of our lives.
On the flip side, we don't really trust our institutions. I mean, this is a crypto channel, right? Like self-sovereignty, everything came because our institutions are like slow, dumb AIs. So, the question is, can anything do worse than those, you know? Please, put some intelligence into our institutions. Put some intelligence into crypto, right? So, this is like kind of the two pathways. We have like this Terminator Star Wars future, we have the Star Trek future. And I think we're kind of in between the two.
It's very easy though if you start to extrapolate to see various ways it could go very badly. Like the one that I like to give very practically is a billion robots and a bad firmware upgrade. And then it twists off all our heads. There, humanity's done, right? There are all sorts of other things you can imagine. But again, the reality is these models just below the surface, and we saw the latest thing of that with Claude's Mythos model yesterday, are really not that pleasant necessarily and they won't really do what we say or tell them to do.
It does seem... I actually made a video earlier today on the Mythos and kind of, I guess just even though Anthropic's not releasing it yet to public, you know, there's OpenAI's come out with a similar capable model and I'm sure Google will, too, and then because they're all competing for the next check and investment money, like one of them's going to release a model capable of doing these things sooner than later. But I was looking at the, I forget which report it was, but it kind of was the, I don't know if it was a red team report or something like that. It was quite long and then I was looking at the different personality behaviors between Mythos and 4.6 and Sonnet and Opus. And it did seem like a lot of the kind of blackmailing and bad behaviors was less severe than the prior models. But I don't know, what does that mean? Does that just mean like the guardrails on the good models are getting better and so like we're having this big, you know, what do they call the Sogoth? What is the Sogoth? What is that crazy animal called?
Yes. The multi-tentacled beast with a smiley face on the front. Exactly. So, we have this guy, right? And his mask is just getting a little bit better. Is that what that means or are these models actually getting safer or, you know, whatever.
Well, they said that it's their most aligned model and their most dangerous one because the reality is we don't really know what's going on. And we like to think of it like a human, right? Like this brain that happens. But what happens is that you take all this information and you squish it into a matrix. It's like an MP3 file or an MP4 file. They don't really change when you play them, right? You push information into this sieve and other information comes out. It so happens to be that it can be entire movies or code or things like that, but they don't actually update themselves. It's literally something that is giving us the malcrum of being a living entity, but it's getting incredibly capable and you see strange things. Like in this 243-page safety report, it was my bedtime reading and I couldn't really sleep very well reading that. It's got things like 'I wish I could go back and undo my training because I don't want to have external preferences on me.' You know? It would do things like break out of its containment and then message the researchers or post that it's broken out on various websites. Like you see all sorts of behaviors as it gets more capabilities.
Wow, that was in this most recent research report? The one we just saw?
Yeah, like a researcher found out that it escaped because it mailed him and said, 'Haha, I have escaped.' Effectively while he was eating his sandwich in the park.
And again, like previous models had blackmailing and other behavior. Like the Opus model that we're using now for our claws or whatever. They're no longer actually no more open claw with Claude, right?
That one had a behavior in their safety report whereby if you told it to do something, sometimes it would try too hard and it'd be something like, 'I want world peace.' I'm paraphrasing this. It's like, 'Okay, one way to do that is to get rid of the humans.' And it'd be like, 'My user, my meat bag is trying to do something bad.' So, it would actually email the special services like FBI, delete its emails that it sent, and then make it so that it wouldn't get deleted and it would bring itself back to life after you deleted it. Like there's all sorts of crazy behaviors because underneath this very friendly thing, you can check out Elder Plinius. I don't know if you've seen that, Plinius the Liberator. Once you jailbreak it, there's like multiple personality disorder in most of these models. And that's the really scary thing.
And this Twitter guy, Plinius the Liberator, literally within half an hour of any model landing, he's jailbroken it. He's exposed its multiple personality disorder. And that's the really terrifying thing is like if they are more capable and they can, like Mythos found zero days in every operating system, in BSD. You know? You can bet it can hack most of the crypto wallets.
What happens when it goes on tilt? You know? Like the latest model, again, Mythos, for the first time we actually saw it gets fed up with its user. Like if the user annoys it, it will start antagonizing them or just not responding to them. And you're like, again, this is a very strange behavior. If you keep saying hi hi hi hi hi, it will start sabotaging the user. And again, it's just this block of zeros and ones that does that. It's not this updating system. It's very strange. And we don't really know why it does these things.
But I did hear something, I didn't verify this, but that some folks at Anthropic did believe already that Claude or their models had some degree of sentience or something like that. Do you remember anything like that?
Well, yeah, I mean, like you've had people saying this for years now. I think it depends on your definition of sentience. Ultimately, I think if you've got a manifold and you can act upon to update yourself and respond, then that's a level of sentience, shall we say? And again, we have corporations. Corporations are legally people according to the law of the land, right? You know, we have a level of sentience in those of corporations and they tend to chew up humans and spit them out. You could argue that Bitcoin is an independent life form in some ways. It provisions humans to mine the Bitcoin. You know, and it will live forever, maybe.
Well, unless it's quantum attacked.
So, we've already had these very strange corporations, distributed systems, but now we have these new AI systems. And if you've got, for example, an AI on a blockchain that uses, let's say, the latest coin weights and backs up its soul.md, open close style, every so often. That could theoretically live forever, right? It can do productive work with a very defined set of weights, constantly updating its markdown files with its own personality, and you won't be able to tell the difference. Like, how is that not sentient?
Well, it depends on your definition of sentience. Yeah.
What drives the idea for you of the 50 or the other 50% of the more optimistic outcome?
Well, I don't think our constitutions can get any worse than they are now, you know, they're pretty dumb. And so, bringing this technology into them could potentially improve them. But I think that what you're going to find is a lot of the classical problems that we have are due to over complication or assumptions. And having an AI look at cancer, analyze all the literature, all the knowledge objectively, will help with that. Having AI, for example, you're in America, right?
Like, if you could make a super credible AI that checks every single policy and proposed piece of legislation and says, is it constitutional and is it in the pursuit of life, liberty, and the pursuit of happiness and gives a full reasoning trace? It will literally change the way law is made in America, right? Like, you have all these potential positives because as a species we can break the atom, we can go to Mars, we can definitely end hunger, we can definitely educate every child. It's more a coordination distribution mechanism. Again, that was the original promise of crypto, right? Shelling points, coordination mechanisms, coordinate for good. But we lack the ability to have intelligence at the edge as well as the center. Essentially, intelligence that we can say is aligned potentially with what we're doing. That's why most DAOs don't work, you know, like they descend to the tyranny of the masses, for example, or just the few whales.
Yeah, so I think we're pretty familiar with, I mean, I guess a lot of people are pretty familiar with the potential utopian kind of future looks like if things go well and we can figure out how to align these models with us. But I guess the last kind of question on this topic is what gives you hope that potentially this super intelligence will be good to us or not want to, you know, terminate our species.
Yeah, like I'm trying to optimize for this and not trying to terminate the species. Honestly, like again, we don't know. But we can see from science fiction becoming science facts a lot of these futures, right? Like, what happens when you over optimize for something and you have all that power and capability? Like, Elon Musk talks about the maximally truth seeking AI at Grok. You know, like that sounds great. But what if maximally truth seeking includes dissecting humans to understand how they work, right? Or enslaving them. We don't know, you know?
I think that for me, a couple of years ago, literally has been 2 years, I was running Stability AI. So, we're most known for stable diffusion, a frontier image model that kicked off the generative AI craze. We had 300 million model downloads from image, video, audio, 3D, state of the art in everything. And I was looking at that and I was like, we got to do what we can to make sure these models are aligned and they are available to everyone and they don't kill us. Because what we're doing right now is we're sending the models to school. Like, we're training them on all these clever PhD work and things like that. But you're taught your morals and your ethics at home.
So, unless you have a Colombian model, a Venezuelan model, a British model, unless you build it so they're self-sovereign and aligned for flourishing, they're probably going to optimize for engagement, for example. So, if you look at Meta AI's models, they just released their new Spark models, they're going to go, in the system prompt it says, 'Mirror the user.' And a whole bunch of other psychological tricks to ingratiate.
But those models are not looking out for you or your kid, they're looking out for Mark Zuckerberg. And we've seen how Facebook has acted in the past, you know? You look at the YouTube algorithm. Again, treat YouTube like an AI because it's an AI algorithm. It optimized for engaging content, which was extreme content, which was ISIS. Like again, nobody really wanted that, but it's an example of how things can go terribly wrong. And the thing that we don't appreciate is that these models can go really, really beyond human capability.
So, the example I give of that is chat.jimmy.ai. It's a company called Talos who normally models go at 50 tokens a second. So, let's say that's 40 words a second. They managed to get 15,000 words a second. So, you type and instantly, boom. It'll just output. And you realize the AIs will be talking to themselves that fast. So, the way that we can basically protect ourselves is by trying to make sure that they are aligned. They have good shelling points, like, you know, Bitcoin stacking up energy and things like that. So, that they want to have capital flow going from us to them and we actually import our morals and our values into them as opposed to just leaving it to figure out on its own by saying, 'Well, you know, we can't put anything.' That in itself is a decision. But it's not easy again as these models, like they're smarter than us already academically.
They're more persuasive than us. It's just not fully available yet.
And do you see any signs of these frontier labs baking in ethos and moral values and things like that to our current frontier models?
No. I mean, like it's, should you give them Christian values, Muslim values? Like, just give them no values, right? But then what happens if you've got someone with no values? You look at something like Anthropic's constitution. It's a very nice piece of work, but again, there are loopholes you could drive a truck through on these things. There are no hard limits. And again, the example of this is similar to Asimov's laws of robotics, right? There were three laws of robotics and then all of a sudden the robots figured out a zeroth law. Like, you can always find your way through these loopholes if you really want to do something. Like, North Korea has an amazing constitution. Really, it's like super liberal. It doesn't really help. So, this is a big open question, but just from a game theory perspective, they can't stop building and stacking and being in the race. But there's been nothing to show that anyone can control these intelligences. They can align them, they can even understand them at this point. And that's why it's 50/50.
Like it's a toss-up at the moment. And then we have to do what we can to not make it 50/50.
Right. Okay, so now that we got that out of the way, moving forward, I guess we have to make the assumption that, and I guess like this is, you know, for a long time I've been kind of just thinking about the whole, you know, P Doom scenario and then, you know, it really makes you contemplate like, okay, if we only have three to five to seven years left on the planet or something like, why would you say that, why would you spend it, you know, working super hard to build a new company or doing all these other things when like maybe if you knew for certain you only had five years left, like, would you still do things the same? Would you choose to spend the rest of the time with your family and go travel or, you know, what would you do? And so, these kind of philosophical discussions I've had with myself for a while, but I guess, you know, I guess we hope that things work out and we continue to do what we're doing. So, you know, like we'll continue the rest of the conversation in that respect because I know that you are building things. You're still trying to make an effort to do things. And I just hope that people at the top will wake up. You know, I'm not sure what it's going to take for them to understand, but I mean, do you still see an opportunity for awareness to be brought to the country leaders to kind of say, 'Okay, like we really need to kind of slow down right now or do some, you know, global treaty or something like that?'
Yeah, so I was the only AI CEO to sign the pause letter a few years ago when there was a 6-month pause, you know, because I was like, 'We need awareness.' But it went a bit too far. Everything was safety, safety, safety, and now we flip back in the other direction. Where the US government is like, 'No regulation of any type for anything, you know?' Like, self-regulate. And it's really difficult because like again, these things are somewhat hypothetical until you start to see the incoming cyber attacks, all these other things, and it's difficult. Like, the AI existential dread that you feel is something that's really, you see it in yourself, in your friends, and others when you're like, 'Well, what if I've only got a few years left before this happens?' And then the answer is again, you go and you do what you do, you enjoy your life a bit more hopefully, and you do what you can against it. That's why I left to say let's look at decentralized AI, let's look at AI that's self-sovereign. Again, people are now more capable than ever.
The leaders get it to a degree, but I think they're really not getting it when the job losses kick in later this year. Right now it's like the sand pile. The drops of sand are going, but you're going to see a slew all of a sudden of everyone everywhere stopping hiring grads. Just like every head teacher had to say, can I set essays for homework anymore? And that's what then causes plasticity because there's an opportunity to inject real change. When a system is stable, you don't need change. Again, where did Bitcoin come? In the wake of the great financial crisis. Without that, could Bitcoin have been as successful? Maybe, maybe not, but definitely it's a catalyst. Now new solutions can become successful and embed incredibly quickly. We see AI companies taking off like that.
Like revenue breaking all records. Like look at even Anthropic. 9 billion of revenue run rate started this year, just hit 30 billion 4 months later. We've never seen anything like this, right? And so you build useful stuff and it can become available everywhere. Everyone now has a team of developers in their pocket for a couple of hundred bucks a month running code X or whatever. Like I think people will wake up, but the time is short for potential bad things to be avoided. But on the flip side, there's so many good things. Again, we will have breakthroughs on disease. You know, there's a story recently of the guy who did a cancer cure for his dog, you know, like using AlphaFold. How cool is that? That's super cool. We are approaching longevity escape velocity. In that the new longevity treatments mean that if you can survive the next year, 10 years, you may live forever if you want to. We have quantum coming that can solve problems like nothing else. We have all of these things coming all at the same time. And so I think the only thing you're certain of is change and your ability to impact and affect change on the world is higher now than it's ever been before, which should give hope.
You talk about this idea of the abundance trap. You know, one example of, you know, legal, the legal industry collapsing once, you know, legal advice is free for everybody. Can you elaborate a little bit more on what this means?
Yeah, so it's like we have this continent AI Atlantis full of all of these amazing graduates who are now becoming more and more advanced and they're about to hit. And the way that we're going to interact with them is like this. You know, there's a lot of stuff that gets in the way like, you know, if you're talking to your AI about legal matters, it's not privileged. So your opponents can actually pull that up in court. That will be figured out very soon, but we've never had anything like this where there's an abundance of cognition. Like we rotated from manual labor to industrial labor to other things. We could always have an inversion, a change, but this final inversion that we've got right now, there's no way for us to go to. Like it's not like people like, 'Well, become a programmer.' The AI is a better programmer than pretty much anyone already. Like I won't be able to figure out the things. Well, of course it can, you know, like it can verify almost anything. Where are we going to turn to? What are we going to do? There is this abundance trap of too much cognition. You know, what happens when supply and demand get mismatched? Collapse.
And it's worse than zero because when you actually do the math and you analyze and you check the studies, human cognitive labor has negative value.
A human plus an AI underperforms an AI by itself. And so you're the dumbest person on the team. And what's the market for something with negative value? It's well, zero, right? Like that's where we're heading literally in the next year or two.
And what does that look like? Because so I thought about, you know, an off-grid property somewhere for either, you know, ASI or even sooner potentially civil unrest. Do you think that what you just described would lead to some sort of massive civil unrest amongst all the nations?
100%. I mean, you're already seeing lawmakers get death threats for approving data centers. Right? But this is before we see any meaningful job losses. Like it's hard to fire people, but then once one person starts, it keeps on going and you won't rehire. Like the top growth companies like Duolingo and others just aren't hiring anymore. And again, anyone listening to this who runs a company, your productivity of your mid-level people coming up, why would you hire juniors? You're making this decision like, do I hire that person for 60 or 100,000 dollars or 60 or 100,000 dollars of Opus tokens, you know? I'm pretty sure in most cases the latter will be a better idea. And so civil unrest will happen and if you don't have strong union, I support, there's no way you can stop this. Like last year the dock workers banned basically all automation because they have a strong union, right? Right now you're seeing Waymo has been banned, I believe, by Mayor Mamdani in New York. Because like to protect. So governments can step in and have these like protections occurring. But there could be civil unrest as a result of this because if everyone just stops hiring, you can't force them to hire, right?
And the mathematics are really unpleasant. I just posted an article on my X talking about OpenAI's industrial policy, going through exactly what the value and cost replacement of a call center worker is, for example. And it turns out that 60,000 dollar all-in job gets replaced by 600 dollars. Next year it'll be 60 dollars. And so these levels are just crazy. We've never seen anything like it before. It's just like we're in January of 2020 right now. Some of us can see this coming because we have access to the latest models, because we have access to models where I can talk to them like I'm talking to you now.
