One thing I find crazy about these scenarios is just how much of the world's future labor supply ends up in very few companies, and also how fast the labor supply grows year-over-year. So if frontier compute, in FLOP terms, is growing 4x or 5x a year, and further the compute required to achieve the capabilities is decreasing 3x a year, the basically effective AI population size at the frontier lab is increasing 10x year-over-year. And so that doesn't really matter that much right now because AIs are not good enough to do full jobs or be as autonomous as people in their capacity to do work or pull off schemes or whatever. But if the current trend continues, you have a world where OpenAI goes from having, say, 10 million basically AI laborers this year to 100 million the next year, to a billion the year after that. And then pretty soon, even if compute scaling slows down, it doesn't take many more years before each company individually has more labor equivalents than there are people on Earth.
I think that's a thing that is very plausible by the end of this decade: that there's more AI labor, more effective population within a single lab, than there are people on Earth. So we talk often about centralization of power because of nationalization or whatever, but we don't think enough about the fact that we're actually moving very fast into a regime where most AI labor—or sorry, most people, in terms of the work output—is just concentrated within two labs who are consuming more and more of the world's compute. And so if these AIs are misaligned, then most of the world is misaligned basically, because most of the world's minds are there. But even if they're not, it's just very few companies have a lot of influence or a lot of control.
There's the whole spat recently where Gavin Baker was like, "Dario believes that there's only going to be one company in the world," and then Sholto and Dario came out and were like, "No, no, no, we didn't say that." But ultimately, if you believe in RSI, you believe that the labs are the most effective user of compute and can generate the most value from the compute, then the only thing that's going to happen is centralization of compute. And if you believe in AI researchers, RSI, AGI, then all of this exists, all of this is the base.
This is even true if there's no RSI. The current effective population of the frontier is currently increasing 10x year-over-year for a given level of capabilities, right? So if you get to the level of capabilities which is a very competent remote worker, or like a very competent software engineer or very competent researcher, that population of those would 10x year-over-year at the current rate of capabilities.
Without RSI. And then once you have RSI, it's going to be crazy.
Maybe it'll be growing like 100x a year or 1,000x a year, or intelligence is increasing but the population isn't increasing, or some mixture of the two, right?
Yeah. I mean, what world do you see, Dwarkesh, where everything is not centralized? Because it seems to me that every force is screeching towards centralization, and that's scary as hell.
I would love for it not to be centralized completely, but maybe that's the whole point of a machine of loving grace, right? It is everything and it makes our lives great.
Yeah, it's so hard to think about the future. But I agree with you that I think the fundamental problem is that AI training has huge economies of scale because any effort you spend into training an AI for a specific skill or specific set of knowledge gets amortized across billions of sessions or billions of users. Furthermore, so that's like one effect. The other effect is if you're slightly ahead in the AI race and compute is in shortage, you can charge a much higher mark-up because you can better economize the scarce resource. So there's the two effects which are giving more and more to the person who's ahead in the AI race. There may be more, right? So if models are learning from deployment and one model is deployed much more widely than another one, it's getting much more real-world data.
Yeah, your point is taken that whether it's user deployment and continual learning, whether it's training having these economies of scale, whether it's incremental progress that the best AI model helps you to make the next AI model—
RSI, all of these things. Oh sorry, I didn't even mention RSI.
All of these things point to centralization.
So I think one of the big intellectual projects, honestly, that we should spend some time thinking about async—or at least I will spend some time thinking about—is what is the vision of a decentralized, broadly empowered future after AGI that takes these economies of scale seriously? The other vision is that the government controls it. And maybe you think you can trust the government more because it's not a private corporation.
I don't trust the government, and I don't trust Dario, and I don't trust Sam.
Yeah, yeah. And that's a problem, right? Obviously it's very easy to be wrong about the future and you don't anticipate a key effect or something that changes everything. But ex ante, it's very hard to see a reason why there, or how we avoid a scenario where we have to choose one person—
Like capitalism worked, right? It's the decentralized decision-making and decentralized power, and why super centralized capitalistic economies actually grew slower than super decentralized capitalist economies to some extent. You have to have rule of law and all this. But then AI flips all this on its head.
