Critical Capital: Varun Sivaram -- The trillion-dollar collision between AI and the grid

AI data centers can be extremely volatile and they're extremely sensitive. You could have a cascading blackout if too many of these super sensitive AI data centers join the grid. All of this is to say, this is not your grandmother's load growth story. The grid, which has been stable over more than a century, needs to adapt at light speed to these AI factories.

We've been building large data centers for 20 years. Over that period, the electricity system could handle these warehouse scale computers, often hundreds of megawatts, for a couple of reasons. We had spare grid capacity, and the load profiles of the data centers were predictable. So, utilities and grid operators could plan for them. But, that logic has changed in the AI era. The rise of multi-gigawatt campuses is testing the limits of our grid capacity. The computing inside those campuses is very different from cloud workloads. They're volatile. A training run can spike and drop in seconds, and it can cause big fluctuations in demand for power, which means the old playbook no longer works. And in the race to scale AI, how you operate a data center matters as much as where you build it.

There's a hundred gigawatts or more of latent power capacity just sitting on our existing grids that we could use to power data centers. But, if they can be flexible, just modestly flexible a small percentage of the year, we can fit more data centers immediately under power grids.

Thanks, Alfred, for having me on, and congratulations on the new podcast. This is the heart of why there is both this technological and cultural collision. You asked about the history of the grid. You know, our grid is what I consider, and many consider, to be the technological marvel of the 20th century. It enabled the US economy to expand at breakneck pace, and across the country, populations expanded, and economies dynamically expanded. Then for the last couple decades, load growth paused. Loads kind of stopped growing, it was flat. And very recently, loads have grown at a breakneck pace yet again because of electrification, population growth, economic growth, and most importantly, as we're talking about today, data centers. Regulations can take a decade or more to mature. But in the AI space, it's we want to deploy gigawatts, tens of gigawatts tomorrow. And yet the state needs 50 gigawatts wants to come online over the next 3 years, from now and 2028, but only 25 gigawatts can actually get built. So, that's the collision we're talking about. And you used the words multi-trillion. It is a multi-trillion-dollar collision because there's trillions of dollars of demand for AI data centers. And if we can't meet that demand from a policy perspective, we lose American national competitiveness. And if we do meet that demand, but in the wrong way, then we just raise rates for everybody, and that's an affordability crisis.

So, AI data centers are a new kind of load growth compared to historically. So, you talk about air conditioners. In the second half of the 20th century, air conditioners were a massive, multi-10-gigawatt load growth phenomenon, but they were distributed across the country. And each of these air conditioners typically was a small load. Data centers are different. Data centers are large and heavily concentrated loads and so they create this geographic heterogeneity. They're more like chocolate chips in the cookie rather than a very equally spread out cookie better. That's a terrible analogy. I made it up right now. It's probably not very good.

Well, well, so let's talk about these chocolate chips, right? Because each data center historically, you know, back when data centers represented less than 5% of American electricity consumption, each of these data centers comprised these racks of servers and the racks would pull 5 10 kilowatts. Now, each rack of servers with NVIDIA GPUs, the latest and greatest, are pulling hundreds of kilowatts and headed toward a megawatt for every rack. So, you see multiple orders of magnitude increased energy density. You're talking about gigawatt scale campuses on a single node of your power system. So, that extreme density is one difference in terms of AI's load growth compared to previous more homogeneous load growth. And the second is the profile. AI data centers can be extremely volatile and they're extremely sensitive. You've seen examples, for example, in Northern Virginia where multiple data centers will all just trip off at the same time. They've caused NERC, the North American reliability watchdog, to say, 'Look, this is a potentially a reliability catastrophe. You could have a cascading blackout if too many of these super sensitive AI data centers join the grid and each of them not only is ramping up and down in the span of seconds or even milliseconds because training can be a highly volatile use case, but in addition, they can all induce each other to just drop off the grid if they detect a mild harmonic or spike in power quality or fluctuation.' All of this is to say, this is not your grandmother's load growth story, right? These chocolate chips are very idiosyncratic and the grid, which has been stable over more than a century, needs to adapt at light speed to these AI factories, as Jensen Huang calls them. We believe that AI factories can also be adaptive, responsive, flexible assets on the power grid to help the power grid out instead of the relationship only running one way, where the power grid itself has to accommodate these extraordinarily different and demanding loads.

So, you mentioned the Phoenix demonstration that we did. We've actually done five demonstrations at commercial data centers around the world, starting with Phoenix with Oracle, Chicago, Virginia, London in the United Kingdom with National Grid, Oregon, Hillsboro, Oregon. Across these demonstrations, we've showcased that flexibility is possible, where these AI factories act as real assets. And you asked, Alfred, what's the potential here? The potential here is in the United States, as Tyler Norris at Duke University wrote, there's a 100 gigawatts or more of latent power capacity just sitting on our existing grids that we can use to power data centers. And around the world, that number more than doubles. There's plenty of existing stranded grid capacity that we just can't use today because AI factories or data centers are not able to be flexible yet or haven't been given the right incentives to do so. But if they can be flexible, just modestly flexible a small percentage of the year, we can fit more data centers immediately on the power grid, so that enhances America's AI competitiveness. We can keep rates low because we better utilize our existing power system instead of triggering these exorbitant upgrades that cost everybody in the community additional rates, and we can keep the grid more stable and reliable because the AI factories can respond to fluctuations and needs of the power grid. That's like three birds with one stone. It's just Alfred, you and I served in policy. You never get a three for one. You barely even get a two for one. Typically, there's a trade-off. There's almost no trade-off here, which is a very rare thing and the reason that I wanted to jump on this idea and found a company is you can get three amazing things for very little cost or trade-off and that's why this technology works so well.

So, you remember those chocolate chips? And I told you how scary the chocolate chips are?

They're really different from previous load growth episodes. AI factories act in a different way than any other power load acts. An AI factory is electronically controllable, which means literally from a laptop, we are able to modulate the consumption of these GPUs. They don't have these massive operational drawbacks of other loads. If you wanted to stop a factory from using energy, you might stop producing cars, there might be some ramp down, ramp up time, etc. Machinery gets harmed. That's not necessarily true if you do it gracefully with software for an AI factory. And most interestingly of all, AI factories can move their workloads from one location to another at the speed of light, which you can't do with any other user of energy. You asked about temporal and spatial flexibility. Temporal flexibility means that you might have some jobs running in an AI factory. Maybe it's fine-tuning some models that you can slow down or pause because the customer finds that acceptable. They're like, I really don't need this model fine-tuned right this very minute. I can wait an hour or a few hours. Spatial flexibility is where you can move AI workloads from one location to another. Serving inference, you may be speaking with a chatbot and the milliseconds that it takes to move from Virginia to Chicago as we demonstrated with the Oracle data The milliseconds that it takes to move are tolerable for a user who expects responses on the scale of seconds if that, right? I just got a ChatGPT query back right before this podcast. It took 3 minutes of thinking. I would be okay if it took 500 milliseconds more. My point here is there is inherent flexibility that we can take advantage of in these AI factories and the network of which they're a part. And taking advantage of that flexibility and putting it together with other kinds of flexibility such as what I call resource flexibility. If you have an on-site battery that you can discharge if it's an appropriate state of charge. You put all these together and these AI factories can become living, breathing, responsive entities.

Alfred, you and I are both Council on Foreign Relations members and so in some sense, we're both committed to finding innovative ways for countries to collaborate and cooperate on the global stage. But make no mistake, I was an international relations double major along with physics in my undergrad. I am a realist through and through. That means I come from the standpoint of nations exist in a state of anarchy competing against one another and it is brutal geopolitical competition. On the margins, it would be great to find ways for them to cooperate, but I am under no illusions that there is a governance superstructure that constrains nations least of all States. Under that backdrop, under this backdrop of brutal anarchic competition, I think that winning the AI race is absolutely critical for the United States in this generational competition with China first and foremost and with a series of other adversaries and competitors. China has 400 gigawatts of spare power capacity by 2030 that it can deploy toward AI data centers and AI factories. Energy is not its bottleneck. Chips are China's bottleneck and that position is reversed United States. Chips are not our bottleneck. We are not going to be fab limited in 2030, but we will be energy limited on current course and speed. That's why unlocking the energy capacity for AI deployment in the United States is the most critical question for geopolitical competition. There's so many other issues, Alfred, that you and I could talk about. We could talk about the Strait of Hormuz. We could talk about drone warfare in Europe, etc. etc. The number one determinant of America's geopolitical success in the 21st century, in my opinion, is going to be how many data centers we can build in America. And that's a provocative claim, right? Others say, 'Do we really need them in this country?' Like might it not be possible to have a network of AI data centers in allied country, which I actually think is a great idea. But the ability to build AI infrastructure in the United States, I think is a crucial determinant of our ability to stay ahead in the frontier model race, our ability to make our economy a low-latency AI-driven economy, and our ability to develop the AI services and goods that we can then use to make our economy the most technologically advanced in the world and export those services around the world in order to build lasting economic prosperity under the backdrop of a brutal anarchic military and geopolitical competition. There's no question. We got to build them at home, and unlocking power is the key to do so.