Well, if you want to maintain hegemony, it turns out like what did we do during World War II? We were like, Henry Ford, here you go, you're going to be absolutely freaking rich because you're going to make all the tanks, right? This is sort of the same thing they're doing with AI. It's like, here you go, Satya Nadella, you get to do this in AI, right?
Dylan Patel has become one of the go-to thinkers on all things hardware and AI with his writing over at SemiAnalysis. Today on Unsupervised Learning, I had an awesome time chatting with him on a host of different things. It was particularly fun because we talked right after the AI diffusion rule came out, so I got his live reactions to what it means for hyperscalers, what it means for China, the UAE, and others, and what the future of geopolitics looks like given these regulations. We also talked a lot about some of the key questions in AI: what will happen with open source models, what are Nvidia's long-term competitive moats, and we also talked about what's required to build really large clusters. He had some amazing stories from xAI's 100,000 GPU cluster. xAI kind of came in and was like, well, we can't find a data center anywhere. What they did is like the most Elon awesome thing ever, right? Which is like, I think folks will really enjoy this conversation. Without further ado, here's Dylan.
Dylan, thanks so much for coming on.
Yeah, thank you for having me. It's a very cozy, comfortable place. I can't believe we got you like during AI diffusion rule week. I feel like this must just be madness right now.
Yeah, yeah. I mean, the fun story was it was supposed to be originally Monday. They dropped on Sunday night slash Monday, right? And so I was like, I was up all night working on it and had to cancel, and we shifted it back a few days. So thanks for being flexible. And you shipped a pretty long piece in the intro.
Yeah, yeah, it was great. I'm curious if you just give our listeners some like quick context on what's going on on the regulatory side with the AI diffusion rule, and then maybe given the, you know, goals of what they're trying to do, what kind of grade would you give it?
Yeah, so I think originally, right, the October 2022 regulations were primarily on the semiconductor industry, but the wording there was like not minced at all, right? It was, we want to regulate AI, we believe it's going to advance rapidly. This October '22, which is pre-scaling, but they were scaling-pilled, right? Exactly. And like, you know, if you get a chance to talk to some of the people like, you know, Ben Buchanan, who's like the special adviser in the White House, like he's like, no, no, this is exactly why we did it. We knew it was coming. And it's like, oh, so the government is actually quite competent. And it's quite interesting because, you know, they hit really hard. They have the goal of like, the US must be ahead of China on AI, right? Because the next five years of progress, next three years, next two years of progress are going to shape the next century of like hegemony for the world, right? And so this is like taking an axe to like try and stop Chinese progress. As far as the grading, right, October '22 was really well-intentioned from if you have this worldview, right, that AI is, you know, in the next five years going to transform the world. You know, if you have the view that it's going to take 20 years, then it's a very different story. And actually, these regulations are actually pretty bad because they're going to limit US competitiveness long-term. But at least over the next five years, they definitely will like keep the US further ahead. So October '22 is really well-intentioned, but there was like quite a few loopholes. So they did another round in '23, then they did another round in December, and all of these like have slowly patched over loopholes. There's still some major ones, right? Namely, Chinese companies can get GPUs from, you know, foreign firms, right? So if you've seen the fuss that Oracle's been throwing, they've just been like ranting about it. Well, it's like, look at who their largest, one of their largest cloud customers is, it's ByteDance, right? Or, you know, hey, you can build a data center in Malaysia now, all of a sudden it's fine, right? And so Malaysia from 2024 to 2027, not the country itself, but companies operating there, mostly Chinese companies, many of them claiming they're now Singaporean companies, right? Like the largest operator in China of data centers, GDS, moved to Singapore and says they're Singaporean now. It's like, but now these companies are building three gigs of data center capacity. To put that in context, at the beginning of 2024, that was roughly Meta's global footprint, right? So it's like, oh wow, like the entire world's like social media is running off of what Malaysia is putting up in three years.
So what happens to all these like, you know, Malaysian data center builds given like, you know, obviously a big part of this regulation is both during countries and then also applying much more regulatory oversight of like who's actually training models within your data centers? Obviously all this planning has gone into building these things, but now it seems like the rug's kind of been pulled out in some way.
Yeah, I mean, this is definitely the most far-reaching regulation that I've ever seen, right? Probably there were bigger things during the Cold War or World War II, I'm not sure. But like, this is very far-reaching in that like it's regulating clouds overseas, right? It's regulating foreign companies, it's limiting what they can buy to a huge extent. And so Malaysia has a huge amount of data center capacity that's being built. We'll see if they continue building it, right? You know, I think most people will still continue building it. Companies like Microsoft and Oracle have significant assets being built in Malaysia, they probably won't stop. But other companies, you know, what happens now? You know, at the end of the day, the US is still very limited on data center footprint as well. We're building as fast as we can. There's a lot of regulatory bottlenecks, there's a number of different things in that nature. But you know, what happens as you deploy, you know, American companies deploy AI globally, right? It's like, is OpenAI going to serve GPT in Japan and Taiwan and South Korea, or like do they just run it out of Microsoft data centers in Malaysia, maybe, right? Like this could be a reasonable way to like subsume this. But like the really important thing here, and the like sort of, you know, even if you're like very pro-America, anti-China, there's still huge negatives to this regulation in that like a ton of buyers can't buy, you know, that data center capacity in Malaysia and actually deploy, right? Like American companies can't do that, right? And so, you know, Oracle is a good example, right? They're one of the parts of the rule is you can only have 7% of your data center capacity in any country that's not like a US ally, right? These tier one countries. And Oracle was to have like 20% of their capacity in Malaysia. So now it's like, what do I do with these data centers that I was going to build? I'm obviously an American company, fine, I won't rent it to Chinese companies, but like what do I do now? So there is a bit of like, you know, the companies that have so much capacity in the US are the only ones that can now take over these data centers in Malaysia and build, 7% rule, right? Because of the 7% rule. So like, it's like Microsoft, Meta, Amazon, Google, right? These four companies have, you know, 70 plus percent of their data center, AI data center capacity in the US, and so they can deal. And they're building super fast elsewhere, they can deal with like, hey, I'll take 500 megawatts, I'll take a gigawatt in Malaysia, and that won't break me out of the ruling, right? Everyone else is kind of like, so now you've reduced competition massively in the market, right? And you've created sort of a monopoly. And so this is sort of like why these regulations, while they are like, yes, if your belief is like this whole AI acceleration and like, you know, you need to combat China, this is very important, US must maintain hegemony, that's one side of the fence. The other side is like, well, if you want to maintain hegemony, it turns out like what did we do during World War II? Well, we were like, Henry Ford, here you go, you're going to be absolutely freaking rich because you're going to make all the tanks, right? Or whatever, trucks. This is sort of the same thing they're doing with AI, is like, here you go, Satya Nadella, you get to do this in AI, right? And so there's a bit of like antitrust, like, you know, I like decentralized power, you like decentralized power, you like innovation, and does this like stamp out innovation in some ways? You know, probably not for American startups, but for like infrastructure hardware, it definitely is like pretty bad.
Obviously there's been, you know, like proliferation of players like CoreWeave and these folks. Like how do these regulations like, I mean ultimately, you know, I've heard you say it obviously favors, you know, the giant tech companies, but presumably like they can have footprints in these tier one countries as well. Like what happens to the kind of like, you know, the mini cloud ecosystem?
So I think CoreWeave has rapidly like become quite close to the hyperscale level, right? Like, and so their next set of build outs, like large build outs, are, you know, continued in the US of course, but then also Europe. I think they could expand overseas for a while and still be fine. They're going to have to hit certain regulatory like, you know, levels for, you know, being able to monitor their customers' workloads and all these like, you know, just when you throw up the more requirements you add, less people who can do it. Yeah, exactly. The cloud ecosystem though, right, excluding CoreWeave, like is very much hit hard, right? There's a lot of clouds that are in foreign countries that are trying to build up. There's sovereign AI, which is like, you know, a lot of startups in the Bay Area are serving sovereign AI firms in Malaysia and Singapore, in India, in the Middle East, whatever, that's their biggest customer. And it's like, these guys are heavily, heavily impacted by these regulations. There's a lot of cloud companies that are, you know, I guess like tier one is like the hyperscalers, tier two is like CoreWeave and Oracle, and then like below that is like everyone is just kind of like really hit really badly. And so, you know, I think the startup ecosystem is going to have to learn like what do I do if potentially, you know, my big Middle East customer or my big Singaporean customer or my big Korean customer, whatever, maybe not Korea, Korea is like allowed, tier one, but you know, one of these countries, sovereign AI countries, am I now regulated? Like this is actually like potentially bad for startups in some way. It also is diversity on the foundation model side, right?
I guess like on the sovereign AI point, like contextualize before this regulation what was happening both in kind of the data center buildout world, you know, that's government fueled, as well as like the kind of training core models like for, you know, for a specific country or geography. It was basically the wild west, right? The US had an executive order a while back, it was basically just like if you're building a model that's 26 flops, which is like very large, that's like twice as large in terms of flops as Llama 405B, right? So presumably like very few people have done that yet, but it's like a notification requirement, that's it. But that's in the US, right? So if you're in China and you're doing this, or if you're anywhere else, it's fine. Of course, there were not really data center buildouts of the scale to hit that quite yet, but they were coming next year and the year after, right? Or 2025, 2026, these buildouts were coming. And so, you know, there was not much regulating foundation models, regulating other countries from building foundation models. Like obviously like OpenAI and other companies were like told like, hey, I know the UAE is telling you they're going to build you a massive cluster, you're not allowed to do that, right? Like there's obviously things like that tacitly under the table happening, but there were no like, here are the regulations. So, you know, if you were a company in China, you would, you know, do whatever you want, and if you're a company elsewhere, you do whatever you want, and you could rent GPUs from wherever you wanted, and you could build whatever you want. And this like smacks it down, right? You can't like export model weights right outside the US slash outside of like trusted clouds, which is again the hyperscalers, right? Of foundation models that are really the like biggest big ones, right? You know, like think Llama 4 and GPT-5 and that kind of stuff. Or you can't like, there's like various things around like protecting against synthetic data generation, which all the Chinese companies are doing a ton of synthetic data generation of GPT-4 so their models can be good, right? Like this is part of the like dirty secret about why like yes, Alibaba and DeepSeek are really good, but they're also just generating a bunch of data from GPT-4 and using that to like post-train, right? Partially. There's a lot of regulations around, yeah, accessing clouds. And so a lot of clouds, you know, they didn't have a core, you know, like when you look at like the big tech companies, they've sort of all paired off, right? You know, Meta themselves, Google's themselves, but then Microsoft, OpenAI, Amazon, Anthropic, Elon's world is xAI, Tesla, you know, the lines are blurry, right? X, the lines are very blurry between these companies. But like these are sort of the hyperscalers in the US. But in China, Alibaba was doing good in AI, and they were building out, but besides that, like there's a bunch of new players. There's Moonshot, there's DeepSeek, and there's all these other players that are popping up, but they don't have a hyperscale partner. ByteDance, of course, right? And ByteDance, they do run a lot of their infrastructure, but a lot of it they rent. You know, what do they do? Well, they decided, well, let me just like not, not DeepSeek, but like ByteDance is just like, oh, I'll just rent from any GPU cloud out there, right? So there's random clouds in Europe, there's random clouds all over Asia, right? And it's like their business case is like, you're going to make a decent profit, right? And you're going to build GPUs and rent them to me, right? And you're not going to have any observability to my workload, I'm just going to do whatever I want. Most of it is just serving TikTok, right? Let's be clear, it's not actually like illicit, but you know, the potential is there, and they're obviously working on language models and generative video models, and there's always the like fear that they're going to manipulate our teenagers' minds and, you know, destroy after Sunday, I guess. I mean, they're still going to be able to do the latter thing, right? But they're heavily, heavily, heavily limited on, you know, the size of clusters they can get now, and who and what, and notification requirements, and like the companies have to be able to observe their workload, which is like bad security. So it's a really, really big blow to, of course, ByteDance and many Chinese AI players, but it's also a huge blow to all these random cloud companies whose business case was like, I'm going to sell to ByteDance, right? Or I'm going to sell to this Chinese company.
So where does Chinese AI go from here? And like, what, do you see any kind of like, if they're looking around for loopholes, like what, you know, what is the path forward there?
There's one obvious loophole, which is while there's like strict caps, right? Like each country can only buy 50,000 GPUs for the next four years, and it's like that's kind of nothing when Nvidia is making, you know, 6 million plus this year total, right? It's like, huh. So each country can only buy 50,000 GPUs, but then there's still loopholes, which is if you buy 1,700 or less, that doesn't count to the 50k. So the obvious thing is you just spin up a bajillion, you know, shell companies, and you just buy 1,600 GPUs and then you route them to China or whatever, right? So there are still some loopholes. But generally, you know, this is much harder to do. I think more so is just like, you know, China has a ton of amazing engineers and they will have to innovate, right? Like DeepSeek is like amazing at engineering, right? So it's like, you know, they're crushing companies with similar levels of compute, right? Because they're just engineering better. But they're not quite like OpenAI, Anthropic level. Now they need to take this like same, like compute for labs continues to scale like crazy, and theirs cannot scale nearly as fast. So it's like, we not only have to be better at engineering, we have to be like way, way, way, way better at engineering. With this new push to like test-time compute, like is that feasible with a huge compute deficiency?
Yeah, so I think there's some interesting aspects of test-time compute, right? It is way more compute intensive than people think. Like people are like, oh, training is dead, and it's like, no, building a test-time compute capable model requires tons of training. You wrote some really interesting data about this. It's not just like pre-training, it's post-training, right? So you have to generate a ton of data, you have to throw most of it away, you have to verify the data to make sure it's like actually accurate, right? The chain of thought or the reasoning chain, and then you have to train the model, right? And you have all these reward models, it's like very complicated post-training stuff. So, you know, and it's very compute intensive, actually. So you can make reasoning models today, right? o1 and like, you know, DeepSeek's R1, I think it is, and Alibaba has a reasoning model, I can't remember the name off top of my head, but these models like are like, the way you should think of it is like, you know, scaling laws are log-log, right? So you can scale data, you can scale parameters, you can scale, and it's like, well, both of those are sort of petering out, right? Because data is not scaling fast enough, and you know, just infinitely growing parameters doesn't give you anything, right? Or it gives you diminishing returns. But we've scaled up this rung many times, right? The training runs that these frontier labs are doing are billions of dollars, right? And next year probably tens of billions of dollars, right? It's like scaling very fast, you can only get like a logarithmic improvement on that front every so often, right? Whereas test-time compute, we're like at the bottom rung of the ladder, right? And it's like, oh, I can like climb up this ladder really rapidly because, oh, right now we're spending hundreds of thousands, millions, you know, tens of millions, hundreds of millions, billions, right? And the compute deficiency doesn't like matter until you get to the billions plus scale, right? And so there are many rungs that they can, you know, out-engineer with the limited compute resources and sort of catch up. But there's also the flip side of like, well, Anthropic, OpenAI, DeepMind, etc., right? xAI, these guys are, and many startups in fact, right? Like Mira startup and so on and so forth, are all scaling up this ladder really quickly as well, as fast as they can. And so it's interesting to see what's going to happen. I think that, you know, that's on the training side, and on the inference side, you know, it's like if you actually like do a query of like o1 versus GPT-4o, it's like, yeah, it cost like 20 cents for a query and it costs like $6 for a query, right? It's like a dramatic difference. Now obviously the quality is better, and so that like, and that $6 is still way cheaper than like paying, you know, someone who makes way more than $6 for that work. But it is like, you know, and it's scalable, right? Whereas like human resources are less scalable. You know, it's kind of like this is a huge difference in cost. $6 is all inference, and it's like, well, you're limited on GPUs. You get into China, you're limited to 50,000 in all these countries that are friendly to China or even neutral to China, right? So it's like, it's going to be very hard to get that scaling of compute on inference, right? And, you know, you can throw as much money as you want on training, but to actually like change the world, right? Again, like if you believe AI is going to change the world, then you have to spend a ton on inference, right? Even if the margins are low, right? You know, even though like Anthropic's and OpenAI's gross margins on inference are like 75%, a lot of these other companies' gross margins on inference are like 10, 20%. Even at 10, 20% gross margins, you know, to amortize the cost of like a $5 billion model, right? Like, okay, you need to do like $10 billion of inference revenue, and it's like, okay, well then that's like $8 billion of hardware cost, which, you know, and that hardware cost of $8 billion is like actually depreciated over multiple years, so in that year you bought like $30 billion of hardware, congrats, right? It's like this, and that's like way too much, right? That's like, that exceeds the caps dramatically. So I think, you know, test-time compute, they'll be able to like deliver impressive results probably. The question is if they'll be able to scale it and sell it. And so that's like the issue. And like on the flip side, it's also like China, the cost of developing software in China is like way cheaper than in the US, right? They just have way more talented engineers, software engineers, and so, you know, their development practices and all these things are different as well. But like every company there, you know, there's a lot of like platform SaaS doesn't work in China, right? This is like pretty well understood. And like in the VC world, it's been like amazing for VC is that platform SaaS type stuff has been awesome, because you can sell to all these companies because you're developing it in one place and you're selling it to everyone, right? Whereas in China, it's like, okay, you're developing in one place, but everyone can just develop it internally, and it won't be maybe as good, it won't be best in class, but whatever, devs are cheap, right? Which kind of seems like the future we're headed to with all this AI coding progress. This is what I'd be curious about your take.
Yeah, no, I mean, I definitely think it's a huge risk for these SaaS providers, right? You know, the ability, I mean, obviously Clara and folks have talked about it very publicly, but I think the kind of moat that was spinning up these applications in the past is certainly a lot weaker now. And I think like the ability also, you know, that combined with the ability to more easily migrate data out of whatever system of record you're in, and basically all the like painful stuff that probably stopped someone from ever really upgrading their software in the past is now just like a relatively simple thing to do with a bunch of like AI agents you throw at it, go away for like a few days and there it is. Yeah, it's a real threat to SaaS going forward.
But a lot of countries have, you know, I think clearly from the beginning, have realized the importance of having this, you know, this expertise and models in their own country. But if anything, this, I feel like this regulation makes it incredibly clear, you know.
Yeah, yeah. I mean, any point they can be cut off. I think like if you look at UAE, this is exactly what's happened, and I think it's been pretty well reported in the media. Like at one point, G42, which is the UAE sort of champion of sovereign AI, funded by, you know, all sorts of like UAE money, and it is based in UAE, and they were at one point like threatened of being cut off, and they were like, the CEO was like a Chinese guy from China, and it was like CCP links potentially, blah, blah, blah, all this stuff. And then like, you know, all the media reporting on this goes silent for like six months, and then the next thing to come out from like whether it was Bloomberg or Reuters or Wall Street Journal, whatever, was like, yeah, G42 and Microsoft are going to do a partnership. And it's like, huh, I wonder like what happened here? Like is it purely organic or was there like some sort of tacit like, well, if you're just going to run all your own infrastructure, you know, maybe there's something here, but you know, now that we know Microsoft is partnered with you, you know, it's good. And like, you know, now when you look at it, it's like G42 and the US government are actually like talking all the time, and they're like, G42 will be a big cloud player, and they're deploying GPUs in the US, even not just in the UAE, but in Europe and the US. So it's through these hyperscalers, I guess you can enforce the American sphere of influence of like what, you know, both obviously, you know, oversight of what models are out there, but also like, you know, ultimately like who gets, you know, I think right now there's what, some human rights terms involved in like the tiering of the countries, and you could imagine like all sorts of ways that administrations use this stuff going forward.
Yeah, I mean, like the Biden, the regulation itself does mention human rights. Obviously the new administration is coming in. If you had me guess, I'd say like 95% probability the Trump administration just keeps these rules and maybe like obviously tweaks them to some extent, but the human rights thing they're probably going to throw out and like, you know, and switch for like, you know, American economic influence or like industry or like whatever, whatever the priority of like, you know, purchasing energy and weapons from the US government, right? Like these are the sort of things that might matter more for the Trump administration versus like human rights policies might matter more for the Biden administration. But at the end of the day, yeah, it's like a weapon for sort of the arsenal of democracy, you know, this is like a regulation enough. It's like, but I mean, taking a step back, I feel like you've talked before about, and I think it was maybe in the context of the chip ban, about this like Goldilocks that you're trying to strike of like preventing China from training their own cutting edge models, but not making it like so barren that like everything gets built up within the country itself. And I wonder like, you know, back to the I guess the grade question, like does it feel like this, you know, hard-handed approach, like is that the right way to further these policy objectives?
Jacob here, taking a quick break to say that if you're enjoying this conversation with Dylan, we'd love if you could share it with a friend. We're always trying to get the word out about Unsupervised Learning, and we have an absolutely fantastic set of guests coming up these next months. Now back to Dylan. Does it feel like this, you know, hard-handed approach, like is that the right way to further these policy objectives?
So I think October '22 and then the 2023 regulations, they were more like, you know, we're going to try and limit you, but they didn't like fully limit like China building their own domestic supply chain and all these things, right? Because the intention is you can't build your own domestic supply chains and you can't access it, but you can access it at like some tier. Now like with these most recent regulations, it's like no access at all to this stuff, right? Like you have an order of magnitude less compute than like any of the US labs, right? That's what we're trying to enforce on you. In which case, like obviously there's going to be like some leakage, and so maybe it's not a whole order of magnitude, maybe it's like, you know, 75% less compute, whatever, right? It's still a humongous amount. So now it's like China only has two options, which is like build your own supply chain, or, you know, and if we limit that effectively, right? So like Ben Thompson has this take, and he's like, yeah, if the regulations are so effective on preventing China from building their own semiconductor industry, or, you know, sort of like, or they build their own semiconductor industry and it's like competitive, then like action on Taiwan is much more likely, right? Whereas like if they feel that they're still benefiting from Taiwan, right? They feel that they still can build it up and get access through Western sphere, right? This is like the sort of geopolitical stuff I have no idea about, by the way, right? Talking out my butt. But like there's sort of the like, you know, the Goldilocks zone of like, yeah, you don't want to ban them too hard, you don't want to let them just have everything because then that induces risk, right? But if there's like that Goldilocks zone of like, you just can't, right? Like you're kind of like, you're behind, but not so far behind there's hope of catching up, then it's maybe fine.
I guess switching gears to, you know, the cluster buildouts that we're seeing today. Maybe just give some context to listeners on, you know, obviously you're the guru of tracking all this, like the size of cluster buildouts we have today, those being planned, and maybe just talk a little bit about kind of the biggest blockers that we have to larger clusters today in these tier one countries.
Yeah, so I mean, when you take a step through sort of like GPT-4, 2022, roughly, you know, few thousand GPUs, A100s, right? And they train this model, change the world, right? Now what's the next sort of step up, right? You know, you got to look at like from two to three to four, right? There's a couple orders of magnitude of compute increases. Now, you know, with GPT sort of five, quote unquote, or Orion, or whatever is coming out, they're, you know, working on it. This is a scale of, you know, hey, well we can't necessarily go for, you know, order of magnitude more GPUs because it's just too expensive and there's a lot of difficulties with building that out. So we go up, you know, we go from 20,000 GPUs to 100,000, right? And so xAI has built 100,000, Meta's built 100,000, both of them have actually building more than that, right, in one spot to do training. OpenAI plans are also, and Anthropic, you know, so on and so forth. Anthropic has 400,000 Trainium server GPUs coming up this year, or Trainium chips, not GPUs, but like their people are in the hundreds of thousands range for this year. But you know, sort of what's built today and what's training models today is on that order of magnitude of 100,000, right? And the GPU that's being used is the H100. So you have 5x the GPUs and you have 3x the performance per GPU roughly, so then you're at like 15x more compute. And we'll see what comes of this, right? Like, you know, the thing is like you don't build a cluster or run a model and immediately get like, you know, something great out of it, right? You have to like do tons of experimentation and, you know, training takes months and post-training takes months and safety takes months, right? It's not like snap of the finger you get the model right now that you've built the cluster. And so there is a bit of a lag, right? And so, you know, this year we'll see models come out that are from that sort of quote unquote last generation cluster, but the next generation one is being built now. What this next generation one is like hundreds of thousands, right? And for context, right, like, you know, each GPU, right, once you include networking, building all this sort of stuff, maybe is, you know, the GPU itself of H100 is like 24,000, but once you add everything else up, it's like 44, 45,000 per GPU all in of everything, right? And so the cost of like 100,000 is like a lot, right? It's like 5 billion for 100,000 GPUs, right? And so Elon's round was 6 billion, and, you know, he built 100,000 GPU cluster, right? Like, you know, obviously there's like other costs, right? They had to buy data, they had to do like, you know, employees, and they had to do like, they had to convert a factory into a data center, right? And like set up all this crazy stuff that they did, right? Which is like, you know, I'll talk about that in a second, but like there's a lot of stuff that needs to be done to get, you know, a model trained. And now that Elon's built the cluster doesn't mean a model's going to come out. You know, I'd be surprised if they have a leading model even this quarter, right? It's going to take some time. And so like what is, you know, that's on the training front. And these next generation clusters being built this year, not 5 billion, they're, you know, 15, right? You know, and obviously you can't scale the money up, you know, like insane amounts, but 3x, right? And obviously in the context of that, Anthropic raising this like 2 billion that, you know, in this current VC round seems perplexing. Well, so what's nice is, you know, Anthropic, OpenAI, Anthropic especially, can, you know, OpenAI and some of these companies that are the startups, right, can raise a round and that money can only be enough for maybe a year, year and a half of GPU rental, right? And that's rental, right? They're not the one buying the GPUs, right? So Amazon might be the one spending, you know, 5, 10 billion dollars, but they might for one year rent it for 3, 4, right? And, you know, when you look at the revenue, the credits they get from Amazon plus the round they raise, right?
That's enough to pay for something like that right now. The Trainium server is not $10 billion because it's like a cheaper internal chip for Amazon, but you kind of get this benefit if your cloud partner really loves you. In the case of OpenAI and Microsoft, they were able to strike a deal with Oracle. Oracle is spending like $10 billion plus for them this year to build out data centers and GPUs for OpenAI, not Microsoft. Interestingly enough, Microsoft is doing stuff as well, but they're not charging $10 billion this year. They're charging a few billion dollars this year. So there is that benefit that you don't need to necessarily... you might have signed the dotted line and Oracle may have taken the credit risk of whether OpenAI will exist on this contract in three years. I mean, you and I believe yes, but this is something you have to consider as a possibility. And so there is that risk. The numbers that companies are raising are not enough to pay for the whole cluster, but that just means Anthropic is going to raise even more nine months from now or a year from now whenever they release their reasoning model. It's a constant fundraising thing. Sam's strength is that he's a god-tier fundraiser. So that's one aspect of it.
The other aspect is when you look at the hyperscalers—Meta, Google, Amazon, xAI—all these guys are building their own data centers. They're the ones delivering the capex. It's really hard to get the electrical infrastructure, the substation build it all out, get it to work, get all the chips to be networked properly, deal with all the failed chips. Because when you're talking about this number of chips, actually quite a few are failed. A lot of them are silent failures. There's a lot of difficulties in getting a cluster to work. The big data center guys have known this. The hyperscalers, xAI kind of came in and Elon was like, 'Well, we're just going to do it.' And it's like, 'Well, we can't find a data center anywhere because all of them are already taken for the time frames we need.' So what do they do? They did the most Elon awesome thing ever. They found a random closed appliance factory in Memphis, Tennessee, located next to a giant power plant, water treatment facility, a garbage dump. Literally, what most people in SF would be like, 'This is the worst place on earth.' Not really, but you talk to people at X that actually go to the data center, like, 'This place kind of sucks.' Memphis has really good wings, by the way. Great food in Memphis, wings, barbecue, etc. But it's like, okay, we bought the appliance factory. Why did we buy it? One, there's a gigawatt natural gas plant right next door. Two, there's a natural gas line, a main that they tapped, and they set up their own generation capacity on site. Three, they're upgrading the substation to take more power from the grid. So they set up their own mobile generators to create power, setting up Tesla battery packs. They've now filed permits to build their own power plant on site, not just mobile generators but a massive natural gas combined cycle plant so they can do a million GPUs, or what he claims he's going to do. We'll see. I won't bet against Elon. He's doing all these things to get the data center now because there's no power. The substation wasn't ready. The power is right there, but the substation was too small and it was being upgraded too slowly because US power grid upgrades take a long time, at least six months to a year. They had already thought about it, but it was only like six months to a year. So they're like, 'Okay, we'll just generate power on the site.' They tapped the natural gas line next door and set up all these mobile generators. The power from these generators is pretty dirty, and GPU training is like, the moment I stop training, power just goes to zero. Or maybe if I'm doing a gradient update, all my GPUs go idle and they're exchanging weights, and it's like, power went to zero, then power went to 100 megawatts. This sort of stuff can blow up grids. So Elon's like, 'Okay, we'll just throw a bunch of Tesla battery packs there.' They're doing all sorts of crazy things. How do we cool this facility because it wasn't ready to cool this much stuff? We'll water cool everything and then rent a bunch of chillers, water chillers, many of them restaurant-grade container things, and just place them outside and cool. They're doing all this crazy stuff, and it's way more complicated than this, and it works. It's very impressive how they deal with all the problems.
One more funny story is this power problem I mentioned. Meta accidentally open-sourced this code. It's literally called 'Power Plant No Blow Up.' It's a flag. When you're doing the gradient update, when you're exchanging weights, instead of having the GPUs go idle, basically when you're sending all the weights to the network, instead they just have them do fake matrix multiplications so the power stays stable. Because otherwise the power plant might blow up if you go like this. It's like, what the hell? So this is the crazy stuff that's happening. There's so many complicated things going on in these $10 billion build-outs that people are doing.
Fascinating. If you were running the government right now, obviously it seems like the energy side is a massive policy and bureaucratic blocker to more of these data centers happening. What's the best thing that government can be doing to unblock this stuff? Is every future data center build-out going to have to be pieced together Elon-style?
To be clear, the number of conventionally built data centers is like this, but what we want is a straight line up. For example, gas generators are sold out for two to four years from GE. Substation equipment is the same thing, sold out for four years. So now you have to get creative. Even though it's growing really rapidly, some of the stuff is held up. I think in one part of the Midwest, it might be Ohio or Indiana, the cost of building a new power plant to transport that power on the grid to a customer costs more than the cost of actually generating the power. It's like, what the flip? The grid needs huge investments. There needs to be a removal of a lot of the environmental regulations, but also to an extent there needs to be more of a vibe shift on ESG-type stuff. This is the attitude of people at AI labs—I don't know if it's actually true or the correct one—but maybe you just say, 'Screw it, to build AGI faster, we do it with natural gas, and then because we're doing it with natural gas, AGI will create enough economic wealth and prosperity that we can just do carbon sequestration and it will all be good.' This is a pretty wild take, but this is what some people believe. To some extent, Meta has kind of... they laid off their whole DEI thing, they uncensored politics, there's been a massive vibe shift. One of the things they're doing is setting up two gigawatts in Louisiana alone, basically all powered by natural gas. Now they can get it in two years. They've figured out how to get the substations and all the supply chain stuff. If we just throw ESG out the window, we can go faster. You're doing it in states that don't have as much environmental stuff—Texas and Louisiana and so on. That's part of it. Now, to this moment, Google and Amazon are still quite committed to their green pledges. Microsoft is sort of in the middle. Meta and xAI obviously don't give a... not that they don't give a... obviously xAI, Tesla does way more for green than what they're destroying. It's complicated. Is there a way to change the vibe on those two and therefore they move faster? Or is there a way to make it so you don't have to do crazy stuff like grids are upgraded, power generation, on-site power generation, all these different things? Help build solar and wind and batteries faster and cheaper in addition to doing all the gas. There's a lot of complicated things to do on the energy side. There are ways to still make it green and fast. You could just throw up a lot of solar, you can throw up a lot of wind. The power generation of renewables is like this, but wind and solar don't actually correlate that well. So most of the times you have power, obviously you have excess power sometimes, but most of the times you have power. Then you throw on a small amount of battery and now it's like almost all the time you have power. But there's still times where the sun isn't shining and the wind isn't blowing for a long enough period that your batteries drain. So now you have some backup gas. These are ways to make it greener and still do it fast. But yeah, I think grid and industrial policy around that is the biggest thing that needs to be done to accelerate this, the domestic version of it. Now that we've done the international regulations, it's the next area of focus.
Yeah, you would think. I myself have more of an attitude of we should win harder rather than we should try to make them lose. That's sort of like I think most people think that's a better ideal and what America stands for more. So we have these 100,000 GPU clusters. Obviously, we're going to see the output of these models at some point in 2025, and I assume that will obviously impact a ton of the future investments based on what we see there. But obviously, people have to invest way ahead of that. We already kind of know what we're getting in for build-outs in this next year. As you kind of think, putting your betting hat on for two years from now, five years from now, how big clusters do you think we'll have two years from now?
I would imagine it would be... there's a couple things. You could say it by number of chips, but it's not fair. A100s were like 400 watts, H100s are 700 watts, Blackwell is 1200 watts. The power per chip is scaling rapidly, so saying the number of chips is not fair. Energy is a good metric because it's like how much useful work you get per... energy is growing rapidly. The amount of energy you're devoting to this—150 megawatts is these 100K clusters. We're seeing 400, 500 megawatt clusters sort of being built out now. In 2026, I'd imagine it'll be gigawatt-scale clusters. That's what it looks like based on... one gigawatt in 2026. Meta is trying to do two gigawatts by early to mid-2027. Others are all doing similar stuff in different parts of the nation. These build-outs are rapidly escalating to earth-shattering amounts of power in one spot. Then obviously the dollars per chip and the power, all these are variables that change over time. But yeah, the build-outs are scaling. When you go from 20,000 GPUs, 400 watts each, that was on the order of 20 megawatts, to 2 gigawatts from 2022 to 2026-2027, it's like, holy crap, you just scaled that much power, two orders of magnitude in like five years.
So if we're sitting here at the end of 2027, how much money has OpenAI raised?
I think it would be really difficult to say. I really do think it would be north of $100 billion. It's pretty clear that they've been trying to convince all these sovereigns to invest and they've been passive on getting them. But I think you're going to see o3 release, you'll probably see another reasoning model release, you might see GP5 release. It's like, how are they not going to raise a ton more money at a ton higher valuation? Revenue skyrockets. I think that's what's going to happen. I think similar stuff—obviously OpenAI will lead the pack, but others will also get similar trajectories happening, like Anthropic, xAI. The amount of capital that will be deployed is going to be insane.
On the model side, obviously we've seen this massive scaling of test-time compute. It seems to work really well with easily verified data—coding, math. I think there's this big question of what other tasks will fall into these test-time compute models. What's your thinking on that?
I really liked when OpenAI released the five levels thing. I thought a lot of people were puzzled, like, what the... it's like reasoning and then agents, because everyone's been... the startup ecosystem is like chatbots, agents. I think it makes a lot of sense because test-time compute, you're verifying your outputs and you're making sure they're accurate. These test-time compute methods of synthetic data generation and then throwing away a lot of it and verifying it and using it—this is where other modalities can come in. Computer use is kind of predicated on all these synthetic data generation pipelines and test-time compute working. Agents are all predicated on the reliability of a model getting high enough to where you can actually get the answer reliably and chain multiple together. These different paradigms will stack together. You can't do computer use without test-time compute. You can't do agents without test-time compute. When you combine test-time compute, computer use, agents, it's like, oh wow, now it's autonomously working on the computer without even having to manually code in all the APIs it's accessing. I think what tasks it can do will branch out drastically once you get the core of reasoning not solved but at a really strong point, and the pipelines for doing all this stuff. What kind of use cases? I think it's like whatever SaaS business model. Of course, coding is great, but it's like, what about software engineering? Because coding is very different than software engineering. Customer service—great chatbots do that kind of, but not great. Searching through documentation—RAG is okay, but actually RAG is really bad because there's no way to verify that you properly pulled in the right stuff, that your retrieval actually worked. People can be like, 'Oh, our retrieval is really accurate,' but it's still not accurate enough for an agent. When you stack these things together, I think it's really any information task. I don't see why it wouldn't work. Now, obviously, what gets done in 2025-2026 is very different than what might get done in 2030. I don't know if I'm the best to say what specific use case, though. I think I can just see the writing in the wall is clear.
What about the open-source side? Obviously, open-source models have been relatively quick to catch up to GPT-4o type level models. I think there is a big question with these test-time compute models, even in the regulations themselves. The way they framed it is open-source models are all fair game, and whatever the open-source models' capabilities are up till then is fine, and then basically it kind of assumes there's going to be some massive gap between closed and open models that just increases over the next few years. Is that how you're seeing things too?
I think so, I agree. I think Meta will release Llama 4 and it'll be better than GPT-4. Obviously, you can say open-source models have closed the gap, but does anyone have a voice mode? Does anyone have quite the level of inference cost that Anthropic and OpenAI do? No, not really, because their models are really efficient. GPT-4o is much smaller than Llama 405B in terms of active parameters, way smaller. So there is still... open models are behind. Maybe not in capabilities entirely, but in some ways. I think what actually ends up happening is that reasoning models are not necessarily going to be fully open-sourced. You see the DeepSeek model, you see OpenAI's model. Meta will release better models as long as they think it makes sense to. But will they only release chat and completion models? Internally, they have their own code model that is trained on their own code database in addition to all the stuff in Llama, and that's not being released. That one's far more capable, at least at coding. There are things like this that point to Meta is not going to just open-source everything. I don't think that necessarily means you're putting together the reasoning system or the agent system. It might mean here's the base weights, congrats, we didn't do any of the reasoning training, or here's the instruct model, or maybe we even do some reasoning. But I guarantee you, if Meta has the best model in the world, they will not open-source it. That's what I would guess. Can it feel like, hey, they're six to nine months behind with a reasoning model? When will there be an open-source equivalent to o3? In six to nine months? Yeah, that's a great question. I'm not exactly sure. I don't think so, actually. I think o3, especially depending on what level of compute you throw at it—if you looked at the ARC challenge—seems unlikely to me. But at the same time, Google's already got a reasoning model. Anthropic allegedly has one internally that's really good, better than o3 even, but we'll see when they eventually release it. What actually ends up happening is going to be very... the whole question is how much capex is Meta willing to spend to keep up with these. I think they'll keep up on compute. The science is hard too. It's not just some level of compute. The numbers you're counting—if OpenAI by the end of 2027 has raised $100 billion, that's a lot of money to put toward free open-source models. But to be clear, most of Meta's GPU purchases are for recommendation systems on Instagram. You can say Zuck bought this many GPUs on some stream or whatever, but when you dig into it, more than half of those are for recommendation systems. You're not spending $30 billion on LLM infra or Transformers. You're doing this on all your products. I just think Meta won't open-source everything they do. Llama 4 will be open-sourced, but the moment they have something at such a capability level, I don't see why they would open-source it because they get benefit from it. They get to attract a lot of talent. They get to have the whole community tell them what they're doing wrong with their models. You look at Llama 2, Llama 3, people are like, 'Oh yeah, you're doing your RL wrong these ways, you're lobotomizing the model in these ways,' and Meta is learning from that. They're attracting all this talent. If they have the best model, they're not going to do that. If they're six to nine months behind, maybe they're six to nine months behind on capabilities, but then the cost is way higher for inference. Or maybe cost is the same on inference but their capabilities are worse. If you look at any specific benchmark, it's like, oh, five percentage points, great. But those benchmarks that are in the 70s and 80s are irrelevant. It's about, oh, you're 10% on this benchmark versus 50%. When you look at things like SWE-bench and ARC, the deltas are getting really big. The deltas are really large. There's only like two models that even score any reasonable amount. These are the sort of things that I think matter. It's like, can you do a task or not, versus, oh yeah, it's 5% better at this task.
