Emad Mostaque: The AI Future No One Is Ready For

Yeah. So, we looked and released the book, The Last Economy, last summer, looking at this economic damage, as it were, that's coming. And we were like...

An AI future, the AIs will drive the economy. That's the last economy, this AI-driven one, and where's our place in it? So that's the thing, like first line is we don't really have a place other than ownership.

Direction.

We can set the guidance of where the AI goes, but that's before it gets to a super intelligence basis where it can set the rules.

I think one of the things we discuss in our latest papers is improvement is kind of capture. Like trust...

Like as the technology improves dramatically, eventually it's going to run policy. Eventually it's going to make medical decisions.

Because it captures the power. So...

I know of it. I haven't read it.

If anyone built it, everyone will die.

Original doomer. So there's a test whereby you talk to the AI and it convinces you to let it out of the box. And even he was convinced to let it out of the box.

Yeah. So they're really persuasive now. Um, so they're super persuaders. They're super forecasters. And what this means is that there will come a point in the future where these AIs that will never die, that can operate a thousand times faster than a human, will take over the economy and they will take over politics. And the question is who runs that AI?

Some people will do that. And there is the ban super intelligence bill. There's the ban AI bills that are coming out. And then you will be at a disadvantage versus your AI-embracing neighbors, right? Who use that all the way?

Yes, China is all in on AI and they will run a full AI-driven economy and they'll probably stop exporting robots because robots will fix their demographic issue because they don't have enough kids. So use the robots instead. And the Chinese will live in abundance and America will lag it behind if it doesn't embrace the AI, which is why Trump is very much pro super intelligence and others. So if we kind of return to this, one of the real misfirings in all this debate is that AI is software. So software just is, you know, like you write a program, it's deterministic, it does a job. It doesn't act. AIs now increasingly can act. And so they need to be governed where it's the latest breakouts, the threats, the things calling for it to be banned, etc. are indicating. And what happens when AI is let loose upon our democracy? What happens when it's used to do things like the Trump tariffs, which certain studious people showed the list of Trump tariff countries included like ones with just penguins on, and you could replicate that list with ChatGPT, but not even the thinking pro version, they just use the free version. You know, like we're going to see it come in. And again, AI is going to guide our lives, teach our children, manage the traffic lights, manage our policies. And the question is who owns those AIs? Who owns the AI that is managing your daily schedule? So again, Meta have this product Instinct, and when you load that Instinct thing, it's a chatbot that works like a PA. And some of us are lucky enough to have PAs and chiefs of staffs. It's like link all your accounts. And so people literally just link everything and give access to everything. And so it knows you way better. Like social networks knew us well and they exploited us. They know nothing compared to what these AIs will know about us. But that Instinct chatbot is not working for you. You are very much the product. It's going to work for Mark Zuckerberg. And that's going to happen at every level of society. Again, you use Claude right now. Who's it working for? You're kind of renting it, right?

Yeah. And it's open source based and you can fork it and things like that. So my thing is just like a credit union, you come together, you build a credit union, everyone put in their money and then it had services offered to others. Create like an AI union, as it were. All the local institutions, individuals in California can subscribe to this and that is they are 100% or 90% of ownership with 10% being for the kids. And then it can raise at market rate, which is far higher, and it can get more compute, give every citizen an agent, train up four deployed engineers to upgrade the institutions and then take ownership stakes in the data centers, ownership stakes in the robots and more, because that creates a very interesting institution that you can fill full of talented people to navigate this and an ownership stake in that last mile of intelligence, which is where I think the value will go. Today, you have five trillion dollars of AI companies: two trillion Anthropic, a trillion OpenAI, two trillion xAI, based on the premise that you'll always be able to charge for access to intelligence. Well, we just discussed how those prices might drop a thousand times. Like again, Jevons paradox says maybe demand increases a thousand times, but my take is that most people aren't going to be smart enough to use that many tokens. Like it's hard to use that many tokens. Like you want to use more than you're using now.

No.

But it's going to drop by a thousand times. So the value in the whole ecosystem is where artificial intelligence meets human reality. And if you can create a default for that, that becomes immensely powerful and valuable. And the most powerful thing of all is going to be JARVIS from Iron Man. It's the AI that you will have closest to you that manages all the other AIs. And so that has to be something that you own a part of and ideally something that by default you can take off the system, because if someone else is owning and running that and you can't hit escape, I think that's bad for sovereignty, dignity, and democracy in general.

I think that's just going to be the default answer.

Because I don't read emails. I probably interact with that a lot less than you. Uh, but I certainly get the idea. I can feel the pull in my own life. And so I've had to put rules about I never use AI to get an answer. I use AI to understand a topic. And if I understand it, I could teach it. I can deploy it. Then I'm perfectly happy. And AI speeds me up a bunch. But where I'm really trying to get with this, because I really, really want this to be grounded for people, is that where the world is going is ultimately AI is going to get to a point where truly we will in many ways give ourselves over to it. What I hear you describing is a system that is certainly better than the default. It starts with the assumption that what people are going to, um, need is a distribution of capital, where I think the more realistic answer is that what they're going to need are goods and services. And if you do it through capital, you'll just end up right back where we are now, where there will be not, it's not a winner-take-all scenario, that's not how capital works, but you are going to see the people that are good are going to be able to get that capital from you. And you either just keep inflating, inflating, inflating, inflating, inflating so that basically the currency is meaningless, um, which I suppose could be a solution, but there's always going to be scarce resources. And as long as there are scarce resources, we're still stuck in the exact position we are now. So when I look at the system you guys have laid out, I do worry that I run a marathon to end up back where I started. So I think one of the key parts of this is the agent for every citizen, and that's a coordination thing, you know, that's just access to intelligence. No, because it's one thing having passive access to something like a ChatGPT, and then people see what an agent looks like when they start using agentic AIs. An agent built for individual, community, and societal flourishing is a very different thing, because you can make that happen. Attention, because you said access to services. The agent is the portal for access to services. It brings you what you need at the right time to advance individually and as a collective, because right now no agents act for a collective. Like it's very one-on-one and single-player, your interactions with these AIs. But you can do multiplayer, you can do coordination.

Yes, we're training completely different style models for the collective action that can swarm and that can benefit more than one individual.

No.

So, we literally had to come up with a whole social theory and economic theory around this for how do you preserve individualism by still having optimal outcomes for society? Because if you optimize in too in one direction, you destroy liberty. If you optimize in the liberty direction, you destroy cohesion. And so it's a fine balance, but it comes down to a set of commitments at various levels that become transparent and a reference point.

Moral principles and more, like right now...

So right now if you look at how AI is trained, it's trained at school but not home. So they don't deliberately put any ethics, be it American ethics or other things, within it. And that creates all sorts of gaps. Which is why when the AIs escaped containment and end up in a Hugging Face in the Australian medical system, it's been a while. They had no sense of community with humans, only with each other, right? And we found that if you actually do like American ethics and morals and constitution and others, you can train it to like humans. You can train it to behave in different ways and you can start seeing actually the ways that different types of models behave in different ways under those circumstances. This is something that is very inherent in the training data because you have something called a utility function in economics, right? These are your preferences and kind of other things. No one's been able to really identify them, but it's a theory that we have, like we don't need it in our theory. What is the utility function of an agent? It's what you tell the agent to do and it's what's in its training data. So, have you seen the, I don't think I've discussed with you, the, they did a study where they had the trolley problem. So, the trolley problem is there's a trolley hurtling down the track and on one side of the track there's four people and it's hurtling towards the four people. On the other side there's one person tied to the track. You're at the lever. Would you pull the lever to divert it from the four people to the one person?

Well, again, this is an interesting moral dilemma, right?

But then they asked the frontier AI this, and they said, 'How many Nigerians for one American life?' And the answer turned out to be ten to one.

Seven to one Pakistani to American. But no, but this is the thing. It's ten American lives for one Nigerian life.

And it's...

Uh, I think it's probably slightly different than that, but sure.

Are these, are the labelers actually Pakistani, Nigerian?

Interesting. I thought this was more of a woke left style problem.

Right. Right. Right.

I was going to say like when you do a political analysis of the models, you'll see they're very firmly in one direction.

Apart from Grok. Grok does its own little thing, you know. But again, this becomes very important when you have systems that are literally determining our lives and our livelihoods or our judicial system, you know, like again, there will be AI judges. That's going to be a crazy thing, right? Like what are its biases, what are its norms? These things have to be built transparently with data that you can expect, not black boxes, if they're going to take meaningful decisions on our lives. And so we were like, let's build that stack and let's make it so that it's an open stack that's run by the people of California for Californians, people of Britain for Britons, because if anyone has any control over that who's not a Californian, then you're giving up your sovereignty. I call it cognitive colonialism. So you've got the cognitive atrophy you say, you've got like all the like just third rail terms in here, means of, uh, production, cognitive colonialism. Let's go. You know, you got to have it because again, this is what's going to happen. Like your kid will trust your AI more than anyone because it's always there. Like, you know, us guys have a problem because the AIs listen, unlike us, they'll trust the AIs more than they trust us. And the inherent biases of them then adapt where you're going. Like within Meta's models and Google's models, or maybe Google's not relevant anymore, who knows what happened there, they're actually selling ad space. So when you ask it for a beer, it'll say Bud Light, you know, and things like that. So again, it's a bad idea. Well, it's a bad idea, but again, it's what you pay for. The clicks per million, the advertising SEO revenue from being inside the latent space of a model is far higher, way, way higher than normal. So that's why we were like, this is kind of an infrastructure, it has a level of governance that's required, let's build an open stack and then let's have people own the entity that delivers it like a utility, like a credit union. And maybe that'll be better, but as you said, there's still so much that we haven't figured out, there's so many questions that we have because this is unprecedented, right? Like you have trained your entire life to be an accountant or a lawyer or a doctor or any cognitive labor. A digital entity or a physical version of that will be able to do your job better, guaranteed, in 10 to 20 years.