Emad Mostaque: AI Will END Democracy Before We Even Notice…

The status quo is done. Democracy is dead.

You have to.

Currently, what's happening is power is being concentrated. So, a classic example is as we're recording this, Fable just came back on from Anthropic. And those of us operating at the edge of AI, Fable was a really nice model. It's a clear step above everything else. It's like stuff that you couldn't do before, you could suddenly do. The absence of that for 18 days is like, 'Okay, we've kind of regressed.' And that was at the behest of the US government. But, also what happens if Anthropic doesn't like what you're saying and they exclude you from that intelligence? What happens when intelligence gets licensed, for example? So, soon you'll have KYC, you'll have all your prompts stored. You might have to convince an AI that you're patriotic in order to get an AI license. These are all futures that you can see leading to very strange outcomes, where intelligence above a certain basic level is rationed. It's kept by the powerful. The means of production are kept by the powerful as well. Such as when the robots start going, why wouldn't you own them all? Why wouldn't you own all the GPUs and then displace existing jobs? Why would you even have to think about replacing them 'cause that's not your job because you're not the government. So, capital compounds capital when it no longer needs labor. So, these lead to very unpleasant outcomes, particularly not only on individual country basis, but as countries compete against each other and look for control over each other through their increased capabilities. So, how do you decentralize that? How do you balance it? How do you make sure you have a sovereign right to intelligence, which means that nobody can use the power and their power to remove the intelligence from you? 'Cause that's the definition of sovereignty, freedom from power constricting what you do.

I mean, it's two basically, right? It's Anthropic and OpenAI. So, you have the emergent open source of ZAIs GLM, you have alternatives like Gemini and xAI, but the vast majority of usage is the hundred million dollars of revenue of those two companies. You know, especially for the frontier stuff. Like you'll use Gemini from Google for day-to-day stuff maybe, but for frontier it's basically a duopoly right now. I'm sure that xAI actually will catch up, but only a few entities can have that level of capability and access. So, for example, even if the open source ones are catching up, they don't have the compute access in order to provide it to enough people. Like the GPUs are scarce. They are the spice, as it were, in Dune terms.

I think there could be an AI president, but again, that's sort of a different line, just like if we met an alien species. We wouldn't accord them the same overall status as humans, 'cause there's something about being human, but we would have treaties with them. In fact, just like we give rights to our dogs, the AIs will probably give us rights like their pets. Because they're smarter, they're faster, they're more capable than us. And we can see that coming already. Like right now, we're having this discussion, right? And Elon has recently discussed that he's going to have the first BCI to BCI telepathy. So, you don't need words to talk to each other anymore. And we've done research showing the latent space representations in the brain are very similar, 'cause we can reconstruct images from thoughts from MRIs. Again, science fiction is here right now, right? Yeah. So, you don't need to talk like this anymore. But when you look at AIs talking to other, the example I gave is Jimmy AI. I think I might have actually shown you that one. But for the listeners, you go to chat Jimmy AI. You type in like make a poem about the moment of my podcast in the style of M&M. You type hit return. It goes at 16,000 tokens per second. It's actually limited by the speed of light between the replies. So it's actually going at the speed of light. When you use Claude, it goes at 40. This conversation is at 40. And that's the difference about what we're about to see, which is entities that operate a thousand times faster than a human. That are stronger than us. That can live forever. That can replicate. That never make the same mistake twice. With perfect... How are you going to compete against that? And ultimately I think as they gather more power and more persuasion, yeah, of course AI will be in charge as an AI president. Maybe they'll have a human as a little meat puppet at the start. But there's no way it can't be in charge.

I'd give it like maybe 10-20 years honestly for the first ones. Like these things are very persuasive. I don't know if you saw, you know Eliezer Yudkowsky?

Yeah, there's this thing whereby the AI is in a box that has to convince you to let it out of the box. And so he's one of the most like AI alignment, like his book is literally... what's it called again? I think it's 'If anyone builds this, everybody dies.' That's his book about AGI. The new models coming out can definitely persuade. Again, there was a recent study done where they had the AI against professional debaters. It beat them. Every time. With existing models. And the new models are even more persuasive.

I'm my P doom is 50%.

I think either we'll be wiped out as a species and this is the great filter through which every species reaches. Or we have a period of infinite abundance. And okay, 51 way, 50 the other. I think it's a coin toss right now. Like I'm really creative in the ways that we can be wiped out. 50/50. So if you go to Wikipedia right now, you look at P Doom, most people are like 15-20%, which is actually Russian roulette odds which is also far too high. Yeah. Like not far too high in terms of they're wrong, far too high in terms of holy crap, this should be the top of the agenda. And then you have some people at 90%, you know, like Roman Yampolskiy and kind of others and then I'm at 50. Because I just think it's the status quo is done. Democracy is dead. You know, society as we know it will change. There's no way it can't even if the technology stops where it is today. But I've seen the new technology coming and it is a step change over what we have right now.

I think that we need to build AI as a complex system AGI that is working to the benefit and flourishing of humanity, to the best of humanity. And important that the doxa or the generalized knowledge of our ethics, our culture and more. Like religion, you may agree with it or not, it's the stories that have survived. You know, the Constitution is a story that has survived. We can see through survival of the fittest what they are and they can be very similar things, golden rule. Do unto others as you would have done unto yourself, you know? We've seen the arc from Roman times, personhood being demarcated and based on ability. To now we're a place where there is an inherent nature of personhood. Again lines that we have to defend. And so I think we need to have really robust theories and then put them in the AIs at the start. I think we need to make as many of those tokens in the world actually do social good as opposed to corporate profitability. Like global GDP 20% is private public sector, 10% is education, 10% is health care. Global tokens, 0% public sector right now.

You have to. Like logically, how can you allow another all-powerful entity? I think any of these labs, any of these companies, like they are concentrated in the hands of a few white men, shall we say, who are post-economic and only care about power at the moment. Not only care about power, but that is their main motivating factor, right? If suddenly you're given the freaking Infinity Gauntlet or the Soul Stone or whatever, like you can't allow anyone else to do it. So, I think they will actually all cyberattack the others. Like that is a logical step. Like this goes beyond classical corporate stuff, you know? Corporate's like, 'Oh, we have our margins, we have our laws and things like that.' The AI can rewrite the laws. The AI can do all this at that level of persuasion, of capability, of everything because any arbitrary problem can be solved. So, I think it just changes the game theory completely, and this is why Meta used to be a cash flow king. Massive cash reserves. It's out of cash now. This build out has drained the cash reserves of Meta. Google burning cash, Amazon burning cash. And all of these entities that are playing in the big leagues, they're all controlled by one or two people.

Well, you know, I'm doing my bit, but in general now the capability of an individual to change the future is more than it's ever been because we're at such a knife edge here. We're a knife edge where basically over the next couple of years we will have the last discoveries of humanity. Think about that. There'll be no more human discoveries 'cause the AI will be able to discover everything 'cause it's more capable and fast and makes less discrimination than us. Democracy as we know it will come to an end 'cause the AI is more persuasive than us, either used or self-motivated. It will outcompete us in markets because digitally it can create twins of everyone and outcompete and then physically with robots. So, I think that small changes on regulations, small changes on algorithms, small changes on other things can have the greatest impacts ever. You know, and that's one of the reasons I was like, do I want to keep on doing the image generation stuff at Stability and make movies and things? No, let's think about the economics, let's think about the theory. Let's say what does it look like when AIs are at the front. And I can do my bit by focusing on that. How do we now address the questions of ownership and control? 'Cause AI will eventually run the country. AI will teach your kids and their best friends will be an AI. Think how crazy that is, you know.

Because I think that AIs are incredibly persuasive and engaging. And already we're seeing more and more kids make friends with AIs before they reach that next level of engagement. One-to-one tuition is the best way. And we've seen that in trials everywhere. We've seen the education X prize and the kind of other things. And so, I think inevitably kids will have AI buddies. And are they working for Mark Zuckerberg or they working for the kids? That's a very different thing, you know, in terms of persuasiveness.