These are still standardized data centers for GPU scale. Right now, we step forward to sort of what is the scale of clusters that people have built last year, right? And it ranges widely. It ranges from like, hey, these are standard data centers and we're just using multiple of them and connecting them together really with a ton of fiber between them, a lot of networking, etc. That's what OpenAI and Microsoft did in Arizona, right? And so they have, you know, 100,000 GPUs. Meta, similar thing. They took their standard existing data center design, and it looks like an H, and they connected multiple of them together. And you know, they got to—they first did 16,000 GPUs, 24,000 GPUs total, only 16,000 of them were running on the training run because GPUs are very unreliable, so they need to have spares to like swap in and out, all the way to like now 100,000 GPUs that they're training on Llama 4 on currently, right? Like 128,000 or so. This is, you know, think about 100,000 GPUs with roughly 1,400 watts apiece. That's 140 megawatts, 150 megawatts for 128, right? So you're talking about you've jumped from 15 to 20 megawatts to 10x, you know, almost 10x that number, 9x that number to 150 megawatts in two years, right? From 2022 to 2024.
And some people like Elon, he admittedly, and he says it himself, got into the game a little bit late for pre-training large language models. xAI was started later, but then he bent heaven and hell to get his data center up and get the largest cluster in the world, which is 200,000 GPUs. And he did that. He bought a factory in Memphis. He's upgrading the substation at the same time. He's got a bunch of mobile power generation, a bunch of single-cycle combined. He tapped the natural gas line that's right next to the factory and is just pulling a ton of gas, burning gas. He's generating all this power. He's in a factory, an old appliance factory that shut down and moved to China long ago, right? And he's got 200,000 GPUs in it. And now what's the next scale? All the hyperscalers have done this. Now the next scale is something that's even bigger. And so, you know, Elon, just to stick on the topic, he's building his own natural gas plant, like a proper one, right next door. He's deploying tons of Tesla Megapack batteries to make the power more smooth and all sorts of other things. He's got like industrial chillers to cool the water down because he's water cooling the chips. So all these crazy things to get the clusters bigger and bigger.
But when you look at like say what OpenAI did with Stargate, that's in Arizona, in Abilene, Texas, right? What they've announced at least, it's not built. Elon says they don't have the money, you know, there's some debates about this. But at full scale, at least the first section is like definitely money's accounted for, but there's multiple sections. But full scale, that data center is going to be 2.2 gigawatts, right? 2,200 megawatts of power in, and roughly like 1.8 gigawatts or 1,800 megawatts of power delivered to chips. Now this is an absurd scale. 2.2 gigawatts is like more than most cities, right? To be clear, delivered to a single cluster that's connected to do training, to train these models to do both the pre-training, the post-training, all of this stuff. This is insane. It is what is a nuclear power plant. Everyone is doing this. Meta in Louisiana, they're building two natural gas plants, massive ones, and then they're building this massive data center. Amazon has plans for this scale. Google has plans for this scale. xAI has plans for this scale. All of these, the companies that are racing are racing hard and they're doing multi-gigawatt data centers to build this out because they think that, yeah, obviously pre-training scaling is going to continue to some extent, but then also all this post-training stuff where you have RL, sandbox for computer use or whatever. This is where they're going to, all these verifiable domains where they just keep learning and learning and learning, self-play, whatever it is, makes the AI so much more capable because the line does go up. As you throw more compute, you get more performance. The chart is about scaling laws. To some extent, it is diminishing returns. You 10x the compute, you don't get 10x better model, right? You get diminishing returns, but also you get efficiency improvements, so you bend the curve.
And these scale of data centers are wreaking a lot of havoc on the grid. Nathan was mentioning Amazon has tried to buy this nuclear power plant, Talen. And if you look at Talen stock, it's just skyrocketing. They're building a massive multi-gigawatt data center there. And you just go down the list, there's so many ramifications. Interesting thing is like certain regions of the US, transmitting power costs more than actually generating it, right? Because the grid is so slow to build and the demand for power and the ability to build power and like repamping on a natural gas plant or even a coal plant is like easy enough to do, but transmitting the power is really hard. So in some parts of the US, like in Virginia, it costs more to transmit power than it costs to generate it. There's all sorts of second-order effects that are insane here. Can the power grid support this kind of growth? Trump's executive orders, there was a Biden executive order before the end of the year, but then Trump had some more executive orders which hopefully reduce the regulations to where yes, things can be built. But yeah, this is a big, big challenge, right? Is building enough power fast enough? Are you going to basically have a nuclear power plant next to a data center for each one of these? So the fun thing here is this is too slow to build the power plant. To build a power plant or to reconfigure an existing power plant is too slow. And so therefore you must use natural gas. Data center power consumption is flat, right? Which is why nuclear is also good for it long-term. Nuclear is a very natural fit. But in the short term, you can't do solar or anything in the short term like that because data center power is like this. You're telling me I'm going to buy tens of billions of dollars of GPUs and I them because the power is not being generated? Power is cheap. If you look at the cost of a cluster, less than 20% of it is power. Most of it is the capital cost and depreciation of the GPUs. And so it's like, well, screw it, I'll just build a natural gas plant. This is what Meta is doing in Louisiana. This is what OpenAI is doing in Texas. And like all these different places, they may not be doing it directly, but they are partnered with someone.
So there is a couple hopes. One is, and Elon, what he's doing in Memphis is to the extreme. They're not just using dual combined cycle gas, which is super efficient. He's also just using single cycle and like mobile generators and stuff, which is less efficient. But there's also the flip side, which is solar power generation is like this and wind is another, different correlation. So if you stack both of those plus you get a big chunk of batteries plus you have a little bit of gas, it is possible to run it more green. It's just the time scales for that is slow. So people are trying, but Meta basically said, whatever, don't care about my sustainability pledge. Or they'll buy a PPA, power purchasing agreement, where there'll be a massive wind farm or solar farm like wherever, and then they'll just pretend like those electrons are being consumed by the data center. But in reality, they're paying for the power here and selling it to the grid, and they're buying power here. And then another thing is Microsoft quit on some of their sustainability pledges. Elon, what he did with Memphis is objectively somewhat dirty, but he's also doing it in an area where there's a bigger natural gas plant right next door and like a wastewater treatment and a garbage dump nearby. And he's obviously made the world a lot more clean than that one data center is going to do. So I think it's fine to some extent, and maybe AGI solves global warming and stuff, right? Whatever it is, this is sort of the attitude that people at the labs have, which is like, yeah, screw it, we'll just use gas because the race is that important and if we lose, that's way worse.
I should say that I got a chance to visit the Memphis data center, and it's kind of incredible. I mean, I visited with Elon, just the teams, and the rate of innovation there is insane. My sense is that nobody's ever done anything of this scale, and nobody has certainly ever done anything of this scale at the rate that xAI is doing. So they're like figuring out—I mean, so I was sitting in on all these meetings with them brainstorming. It's insane. It's exciting because they're trying to figure out what the bottlenecks are, how to remove the bottlenecks, how to make sure that there's just so many really cool things about putting together a data center because everything has to work. The people that do like the sysadmin, you know, the machine learning, all that is the exciting thing, but really the people that run everything are the folks that know the low-level software and hardware that runs everything, the networking, all of that. And so you have to make sure you have procedures that test everything. I think they're using Ethernet. I don't know how they're doing the networking, but they're using NVIDIA Spectrum X Ethernet. There's actually, I think, yeah, the unsung heroes are the cooling and electrical systems which are just glossed over. But I think one story that maybe exemplifies how insane this stuff is, is when you're training, you're always running through the model a bunch. In the most simplistic terms, running through the model a bunch and then you're going to exchange everything and synchronize the weights. So you'll do a step, this is a step in model training, and every step your loss goes down hopefully, and it doesn't always. But in the simplest terms, you'll be computing a lot and then you'll exchange. The interesting thing is GPUs, most of it is networking power, some, but it's a lot less. So while you're computing, your power for your GPUs is here, but then when you're exchanging weights, if you're not able to overlap communications and compute perfectly, there may be a time period where your GPUs are just idle and you're exchanging weights and you're like, hey, the model's updating. So you're exchanging the gradients, you do the model update, and then you start training again. So the power goes mhm, and it's super spiky.
