Data centers use HOW much water? Microsoft's Scott Guthrie explains

Doing well, Corey. Thanks for having me.

Yeah, 28 and a half years. So it's been a while. It feels like just yesterday, but it's been a while.

Yeah. You know, that's still early consumer internet even at that point. It was before, you know, Netscape had just come out.

The internet was just starting to get big. I think, you know, IE3 I think came out that summer. I was an intern at Microsoft and so it was all still pretty new. And yeah, it's amazing how the world's changed since then.

Yeah. I think that's, you know, the fun thing of technology is there's always something new, and at the same time, the new thing builds on the things of the past. And so with agents, obviously they have similar dynamics of the early days of the internet. We're all still trying to figure out what you can do with them and how they work and what the patterns are and how do you run them reliably at scale. And a lot of what we're doing at Ignite and talking about is just that, kind of how do you build them with things like GitHub and Copilot Studio, and how do you integrate them into employee workflows with things like Microsoft 365 Copilot. How do you run them on infrastructure with things like Azure? How do you connect them to custom data with things like Fabric, and then introduce your own custom models with Foundry? And then, in particular, one of the things we talked about at this event and announced today is how do you kind of run them and operate them and observe them as they're doing things inside your company.

And with our new Agent 365 offering, we now have a way that you can sort of even register the agents, have them have an identity, watch what they're doing, and then ultimately be able to govern and control and orchestrate them at scale so that you can really focus on delivering business value. And there's still a lot of things to figure out across all those things. I think it will be for years. But if you just look even at this event versus our one last year, it's similar to the early days of the internet — it's amazing how much progress has happened.

And the other big announcement we made today was obviously we do all that with OpenAI, but we also announced our expanded partnership with Anthropic. And so we're also using Anthropic's cloud models inside GitHub — we have been for a while — but then also inside M365, and then now it's available on Azure as a model you can use with our AI Foundry, natively right there, natively running inside Azure and being able to leverage and work across everything I just mentioned.

I think there's, you know, different users have preferences. I mean, I think we saw that pretty clearly with GitHub Copilot where, you know, a lot of people just really prefer Claude for developer scenarios, other people have a preference for Codex, and other people have a preference for other mechanisms. And so I think we're very much about how do we provide choice and flexibility. And I think especially given just how dynamic and how fast the AI space is moving, you know, not forcing you to use something you don't want to use and having that flexibility where you can bring an existing toolchain or existing model and integrate it with everything else.

I think that is the name of the game for people to get the most value. And we're doing a similar thing on the infrastructure side when you think about we support NVIDIA GPUs, we support AMD GPUs, we support our own first-party Azure Maya AI accelerators as well. That diversity of choice and diversity of options, it's good for customers and it also ensures that we have the most competitive offering because we have a complete offering that allows you to also pick and choose the pieces that you most want.

Yeah. And I think in a given quarter, a given month, a given year, I think you're also going to see this dynamic, which is healthy for the market, healthy for consumers of AI: this model pulls ahead on this scenario, this model pulls ahead on that scenario. And to the extent that you can also have common tools that let you do evaluations, safe rollouts of new models, in some ways it's like the DevOps, if you will, of AI development, where you want to run your tests and your evals and know that you're not regressing on quality in one scenario while you're also making another scenario better, watching out for model drift and all of the things, and then having kind of safety and security that works sort of independent of a specific model means that you can try out new things and become either convinced that it's going to be better or realize, well, it's better here but it's worse here, maybe I won't use it.

And that's a lot of what we're trying to build is sort of the toolchain that lets you do that and do it repeatably and do it reliably, because as more and more organizations, whether they're startups or enterprises, put these models into production and use them for mission-critical workloads, you need that level of hygiene and that level of control to avoid having a problem that impacts your customer experience or impacts your business.

Yeah, I mean, it's definitely — we work a lot with businesses in particular, and I think for all of the businesses that we talk to, there's sort of two big conversations that people want to have with us. The first is how do I demonstrate business value so I can get my CFO to continue to invest in AI. And so people are like, I need to move from prototyping, from demos, from experimentation to I can measure customer support costs went down 12%, or sales increased by X%, or business operations improved by X, Y%. And so I think people are increasingly focused on not just sort of the potential of AI, but really like, okay, let's demonstrate how we use AI in business to really drive tangible results. That's one aspect.

Yes. And then there's the, okay, now that we're seeing the big results and now that we're using it for mainline tasks that are really driving business value, how does it not break? And I think a lot of people go through an experience where something goes wrong in their AI workflow. Maybe someone rolled out a code fix without it being fully tested, or someone rolled a new model and it improved on eight dimensions but regressed two. And that's when people suddenly realize also, wow, I've really taken a dependency on AI. And I now need the maturity level that I would have when I'm going to touch one of my core systems, and that's where you need observability as well. And that's why I think observability just pops up over and over again in our conversations.

And you know, I think people are also realizing as models are smart and as employees can ask the model to do things, how do you know that the model doesn't give an answer because it's been given access to maybe more information that the employee shouldn't know — things like financials, or internal HR or legal things that are going on inside an organization. Or, you know, if a customer asks in a support bot like, hey, what's the maximum discount that my salesperson is allowed to give me, you might not want that support model to give you the answer. And that's where also I think things like governance and things like the right compliance levels really come into play. And again, the more you have a really good story for that, the more organizations can bet on AI and get real value and are willing to invest to do even more.

And so it is a great unlock, and I think it's also a great need, and we're excited with Agent 365. I'm sure there's more we need to add to it, but I think it's the most complete solution out there today. It is kind of multi-model, it is multi-vendor, and it's also integrated inside the number one enterprise AI solution out there, which is Microsoft 365 Copilot. And I think we're really excited. We're using it even internally, and when we turned it on, you know, step one is even just figuring out how many agents do you have in your organization.

We were having a discussion even on our leadership team at the company level on Friday, and the number of autonomous agents it came back with was about an order of magnitude more than we thought we had. And a lot of it's individuals and departments that had set up little tasks, and once it sort of showed us how many, we were like, whoa, that's a lot. And I think that's also going to be a similar dynamic when people start using it, is it will help people realize they're using AI a lot more than they thought. And again, it's a good way to make sure that you're using it responsibly and appropriately and safely and securely.

Totally.

Oh, thanks. I think the other thing that we're seeing with AI adoption and agents is, if you have a suboptimal business process today, like you're doing everything in spreadsheets and email and handoffs across eight different teams to make a decision, you can automate that with AI. But you often are automating a suboptimal process with AI. And I think a lot of what you want to do is re-engineer that process.

To your point of enabling more people to use these tools as opposed to just developers or just the security office or CISO office — if you could get the people that actually own the process to be able to use the tools and be successful, it really enables people to step back and say, okay, why don't we change our process? And the people that can most change the process are often the people doing the process. And so to the extent that we can kind of democratize those tools, hopefully it really enables organizations to kind of do deeper business process reinvention and use technology to solve it. And I think that also is going to be a great unlock for AI.

Yep. Yeah, that's the part I love the most as well because that's the part I work on, and it's definitely fun. And especially the last couple years as the AI buildout has happened, it's not easy either in terms of just the scale that we need to kind of do it at. But you and we talked, it's been fun at the event and then the previous couple weeks leading up to it, we've also disclosed a bunch of details about some of our new Fairwater AI data center designs that we're particularly proud of, and they're in production today. And so if you're using an AI app, there's a decent chance tokens are running through it. And it's fun to be able to kind of talk about some of those cool innovations.

Yeah, the new Fairwater data centers that we're building — we're building some in Atlanta and Wisconsin, which are two areas that we've both kind of disclosed, and there's lots of other sites around the US and the world that we're doing stuff in, but those are two that we've done now. You know, aerial photography — we showed a video in my keynote this morning for the first time of the Atlanta facility. They are the densest concentration of GPU power in the world. And by densest I mean the most GPUs, and then the most GB200 Grace Blackwell GPUs, the latest, all liquid-cooled, which means that just from a pure token perspective, a Grace Blackwell GB200 is roughly 12x the throughput of AWS Trainium 2.

Exactly, exactly. And so if you pack them all into one data center, and the data center is a flat network — so it's basically you think of it as a two-tier flat network as opposed to a traditional cloud data center, which would usually be three tiers. And so that means that when you want a GPU to talk to another GPU, you're going through more switches, which means lower throughput, more latency. We've tried to keep a very flat network so that each of these GPUs can interconnect with the others at incredibly high volumes.

And then part of what we've done to kind of optimize speed even is these data centers in the video you'll see are double-decker, meaning they're two floors. And the reason we do that is we put the network core of the data center in the center of the building on the second floor. And that means that no cable in the data center is more than about 230 meters — no more than a quarter of the building. And that minimizes latency because you're only as fast as the speed of light. But it also means that from a retransmission perspective, you're also really minimizing the number of retransmissions literally at the wire level. There's no oversubscription of the network, so it's line speed. And that combination, it's just a screamingly fast data center.

And it's fun. It's running. It's real production now. You know, OpenAI is using it for a bunch of workloads already. And so it's been fun to kind of see all of that come together. And put in perspective, of that network I mentioned, there's 5 million individual cables in that data center. It's just an insane amount of connectivity and GPUs.

And then it's all liquid facility water. So in other words, the data center itself, all the GPUs are cooled by water as opposed to traditional air chillers. And so outside the data center in our facility in Wisconsin, we have a chiller plant and it's the second largest chiller plant in the world. There's only one more that's bigger — it's Qatar in the Middle East — but it's the largest chiller plant in the United States and in North America or anywhere else around the world outside of Qatar in the Middle East.

What's cool about the liquid cooling, because people often ask me like, what about water consumption? It's about 20 houses' worth of water to fill the data center. So 20 American homes. So in the grand scheme of things, it's not a lot of water. And then once we fill the data center, we will not replace the water for at least six years. So there's no evaporation, there's no water loss. It's continuously recirculated, working like a radiator system in a car.

Closed loop system. It's about five feet per second of water flow into a GPU and out. And so we flow the water in, the water temperature goes up by about 20 degrees Celsius, runs out and cools somewhere, shoots out the building, we cool it, shoot it back in the building when it's chilled again. So it's incredibly efficient from a sustainability perspective. It's also one of the commitments we talked about in the keynote this morning is all of our data centers in Azure will be 100% renewable energy by the end of the calendar year.

You know, we made that commitment. It was before the AI push. So it became a lot harder as we started building.