Microsoft's head of cloud and AI joins us as we ask, is this AI buildout going too far? That's coming up right after this. Welcome to Big Technology Podcast, a show for coolheaded and nuanced conversation of the tech world and beyond. We're joined today by Scott Guthrie. He's the head of cloud and AI at Microsoft. And he is the perfect guest to give us some context on the massive and some would say insane buildout of AI data centers taking place today. What does it mean? Is it going too far? What will it lead to? Scott, I'm so thrilled to have you on the show. Welcome.
It's great to be here, Alex. Thanks for having me.
All right, let me take you through the headlines over the past couple weeks. It's crazy that this has just been over the past few weeks, but here we go. Nvidia agreed or announced that it would invest up to a hundred billion in OpenAI, starting out with 10 billion. Oracle announced it would invest 30 billion in OpenAI or a $30 billion buildout with the company. Anthropic raised 13 billion. So, you know, we're just talking about a cool 143 billion. No big deal. Is this crazy? Is this overinvestment?
Well, I think it's a great question. I'm sure that's top of mind for everyone. I think, you know, stepping back for a moment, I would say if you look at AI and the impact I think it's going to have in the economy, it's going to be, I think, the most profound technology shift in our lifetimes. And so I think if you look at the long-term trend, I don't worry about overinvesting. I think that, you know, there will be a question on the horizon of different companies making different strategies in terms of their investment and how they get their return in the one, two, three year horizon. So, you know, am I going to say that every company is perfectly timed? I'm not going to make that assertion. But at the same time, I do think the long-term secular trend of AI is going to be that we're going to need more infrastructure. There's going to be more ROI from it and it's going to be more widely used. And so, you know, I think directionally from an industry perspective, the investments do make sense and will ultimately yield pretty profound results.
You think so? You think that this level of buildout is healthy?
I think we definitely are not nearly at the point at which there is too much AI infrastructure given, I think, the number of AI workloads that are coming for the world. And, you know, I think we're seeing over the last couple years as people use AI, they get value, they use it more, the models get better, and people then use it even more for new use cases. And I think at this point across the industry with AI, we're still more supply constrained than we are demand constrained. And I think, you know, I expect that to continue over the next couple years as the technology continues to evolve and as people start to integrate AI into more and more workflows.
Okay. So, let me put it bluntly then. Microsoft has a partnership with OpenAI, has invested 13 billion thereabouts, has the capacity to build big data centers. Why did Microsoft make the decision not to do the hundred billion level buildout with OpenAI or even the $30 billion that the company is doing with Oracle and leave it to other partners to do that?
Well, we have a great partnership with OpenAI and it's gone back many, many years and continues going forward. And we are building out and doing a lot of projects with OpenAI and across the Microsoft cloud we're building out AI data centers all over the world. And, you know, at the same time, you know, we are balancing our investment to make sure that it maximizes the AI infrastructure for both our first party Microsoft offerings that we're investing in, our customers' AI offerings that we're investing in, and obviously OpenAI's offerings that we're deeply enabling. So, you know, we are very invested. I don't think it's a binary. Are we building out for OpenAI or not? We definitely are building out for OpenAI. And at the same time, the way our partnership works is we're supportive of others participating in that as well.
Okay. Okay. But I just want to put a fine point on it because, again, like if you believe that this technology is going to be massively transformative, which you stated, and that we're not at the sort of optimal point of the AI buildout yet, that there's room to continue to do more. And again, it's the partnership with what is the consensus leader in the space. They needed more infrastructure. There must have been some calculation within your group or your company to say, is it worth it for us to be the one that goes out and builds this massive, massive footprint, you know, in partnership with them or somebody else. So I definitely understand there's multiple stakeholders, but what made Microsoft pause on that front?
Well, we have a balanced view and so we take both a long-term view in terms of making sure that we're building out in all the locations that we want to build out, that we're being thoughtful in terms of kind of the investment spend and the infrastructure that we're building, and also recognize that we don't have to do it all. And so I think we're always trying to kind of take a continually balanced view of that. And as you've seen from our capex and as you've seen from our earnings calls, we are investing a lot in infrastructure and building out like crazy. But again, at the same time, you know, we're always constantly re-evaluating and watching closely, you know, which data centers in which markets to what specifications and making sure that we keep, you know, good discipline as we're doing it that optimizes for both the long-term, near-term, and midterm horizons.
Okay. I'll just ask one more follow-up and then we can move on. Good discipline. What about, this would have been undisciplined to have gone to this level.
Well, I don't think it's so much the volume level. I think it's, one of the things that we do when we add new data center capacity or AI infrastructure is, you know, making sure that we can use this infrastructure for a variety of different AI use cases. I think one of the things that's really going to differentiate AI infrastructure companies in the future is that ability to kind of maximize yield on the infrastructure, like how are you driving down the cost of, you know, tokens per watt per dollar. And, you know, part of what makes the Microsoft portfolio so unique is the fact that we have a lot of our own AI products, Microsoft 365 Copilot, GitHub Copilot, the work that we're doing with Nuance and Dragon in healthcare. We've got the world's largest consumer application with ChatGPT that runs on top of Azure, and we have thousands, hundreds of thousands, and millions of businesses that are also building their own AI applications on top of us. And so as we think about, like, what market are we going to build a new data center? Is it for training? Is it for inferencing? And, you know, how do we make sure that that infrastructure is going to be maximally used? You know, we feed in kind of each of these different customer scenarios into our calculus, and there are certain tranches of capacity that we're happy to build out because we can see very clear line of sight in terms of how we're going to maximize the usage and the revenue from it. And there's others that we're maybe less likely to see the immediate ROI that we'd like. And so we try to be disciplined about it. As we've kind of shared in our blog post, you know, we do kind of look at every request first, and we do have an opportunity on that. And as you've seen from our capex, we are swinging at a lot of opportunities. But that doesn't, you know, we're not going to be undisciplined and say blanket we're going to do everything. You know, we know that, you know, some opportunities will have more certain returns than others, and we're trying to make sure that we maximize our focus around those.
You know, as we're talking, I'm kind of laughing at myself because, you know, my question is basically boiling down to you've, like, talked about your capex. Isn't Microsoft expected to spend like 80 billion on infrastructure this year or in the neighborhood?
I think that's what we shared in our last call.
And I'm like, well, why aren't you doing another hundred billion? But the fact that you're not is actually very interesting. And it goes to a point that you just made. And I'm trying to read between the lines, and you tell me if I'm getting this right. You think you're talking about where you invest and where you're pretty sure you're going to get an ROI. And to me, if I'm sitting in your shoes, the question I would be asking is, is it worth spending, you know, all that money on training where there's been a lot of noise about diminishing returns of training larger models with these unbelievably massive data centers? Now you have the startups like OpenAI and Anthropic, you know, their belief in the scaling laws seems unabated. And so the numbers get bigger and bigger, and they seem to believe that they'll continue to get an exponential return from training these bigger models. But is your decision in terms of being disciplined based on a belief that you're not sure if scaling up will continue to work and therefore it's too big of a risk to make such a large bet on training an even bigger model in a bigger data center?
Well, I think there's a couple different elements of that. I think one is recognizing that you want to have the best models. So, you know, training is super important because if you don't have the best models, then, you know, your actual ability to monetize AI goes down. You know, then, and, you know, part of what makes our partnership with OpenAI unique is the fact that we do have access to the best models, frankly, whether they're trained on our infrastructure or anywhere else. You know, that's part of our partnership that's really important. And I also think when you think about training, training is evolving from maybe where simplistically we'd think of training a couple years ago of you do training in one place and then you do inferencing where you are executing the models and building applications. You know, there's now multiple types of training. There's pre-training, there's post-training, there's reinforcement learning, there's fine-tuning. There's a lot of new techniques that both sometimes require lots of contiguous infrastructure and sometimes requires lots of infrastructure, but sometimes it's smaller sizes that can be used for very specific tasks. And so when we think about the investments of our infrastructure, we're trying to think about all of this and compose it all end to end. You know, for us that means, for example, we want to make sure that we have lots of inferencing capacity because that ultimately is how customers pay us and how ultimately you make money from any AI product that you build. And I think increasingly on the inferencing side, you know, one important element is the geopolitics of the world have gotten complicated over the last many years. And, you know, customers in Europe want to make sure that their AI is in Europe, and the customers in Asia are going to care about their AI in Asia. Obviously, the customers in North America and the United States are going to care about their AI being delivered in North America. And so, you know, even as we build out our infrastructure, we want to think about it not just narrowly as we want to have one giant pool all in the US. We need to kind of be distributed around the world to kind of meet those geopolitical needs and to make sure that our AI is as close to the customers that are going to be using the AI as possible and can meet all of the data residency and data sovereignty needs. And so even if you look at our infrastructure builds around the world, we have regions in more countries in more locations than any other infrastructure provider. And again, as we balance out the investments we're making on AI infra, you know, we're trying to keep that in mind versus narrowly put it all in one location.
I totally understand that, but I have to go back to the diminishing returns of training question. Where do you stand on that?
Well, I think if you look at training broadly, I think you're going to continue to see more value from the models by doing more training. But kind of going back to my answer earlier, I don't know if that's always going to be pre-training. I think increasingly lots of post-training activities are going to significantly change the value of the model. And so by post-training I mean take the base model and how do you add financial data or healthcare data or something that's very specific to an application or a use case. What's nice about post-training is that you don't have to do it in one large data center in one location. And so part of the technique that we've been focused on is how do we take this inferencing capacity around the world, and a lot of it is idle at night as people go to sleep. You know, how are we doing increasingly post-training in a distributed fashion across many, many different sites? And then when employees come to work in the morning, we serve the applications. And so having that kind of flexibility and being able to dynamically schedule your AI infrastructure so that you're maximizing revenue generation and training ideally in a very swappable dynamic way, I think is one of the things we're investing in heavily and I think is one of the differentiators for Microsoft.
Okay. Okay, but you'll forgive me for going back to this scaling pre-training question. I'm just trying to see what you believe here, and you haven't said it outright, but from your answers, it does seem to me like you believe that spending wildly on scaling pre-training is a bad bet.
I wouldn't necessarily say that. I think we've definitely seen as the scale infrastructure for pre-training has gotten bigger, we are seeing the models continually improve and we're investing in those types of pre-training sites and infrastructure. You know, we recently, for example, announced our Fairwater data center regions around the US. We have multiple Fairwaters and, you know, we did a blog post recently of one of our new sites in Wisconsin. And these are, you know, hundreds of megawatts, hundreds of thousands of the latest GB200s and GB300 GPUs and are, yeah, we think the largest contiguous block of GPUs anywhere in the world in one giant training infrastructure that can be used for pre-training. And so we're investing heavily in that as you could see kind of from the photos from the sky in terms of massive infrastructure. And, you know, we do continue to see the scaling laws improve. Now, will the scaling laws improve linearly? Will they improve at the rate that they have? I think that is a question that everyone right now in the AI space is still trying to calculate. But do I think they'll improve? Yes. And the question really is around what's the rate of improvement on pre-training. And I do think with post-training, we're going to continue to see dramatic improvements. And that's again why we're trying to make sure we have a balanced investment both on pre-training and post-training infrastructure.
And yeah, and just to parse your words here, it's, you can see improvement by doubling the data center, but that's why I use the word bet because are you going to get the same return if it doesn't improve exponentially and just improves on the margins? And that I think is the big question right now, right?
It's a big question. And, you know, the thing that also makes it the big question is it's not like a law of nature that's immovable. And so there could be one breakthrough that actually changes the scaling laws for better, and there could be a lack of breakthroughs that means, again, things will still improve, but do they improve at the same rate that they historically did from a raw size and scale perspective? And that is the trillion-dollar question.
Okay, great. I do want to get to the ROI of AI spend in a moment. You know, it's always great to have a chance to speak with someone who's in a position like you are within Microsoft because we get a chance to, like, take some headlines which might paint a portion of a story and then ask you what the truth is. There were some stories over the past year talking about Microsoft had, like, cancelled options to build data centers in certain locations, and people took those headlines and they read into it that there was no demand for AI or that it wasn't going as well as Microsoft's telling us. But what is the reason for why those data centers that there were the options and they were canceled? What happened there?
Well, we're constantly... Well, I think in general the headlines were focused on things that we canceled as opposed to all the things we signed. And so if you look at a given...
It's amazing how news works that way, right? If it bleeds, it leads. So...
If you look at kind of the overall investments and certainly if you look at the overall capex spend, it has been going up and up and up. And so has, again, the revenue that comes from it. And so I think I would kind of focus on the overall picture as opposed to individual tranches or individual projects that we potentially made decisions on. Now, you know, the thing that we did do and we continually do is look hard at every single investment decision we make. And we don't take this level of investment and this level of project and infrastructure lightly. It's critical that we invest wisely. It's critical that we make it successful and that we bring it to market on time with the right quality and the right security. And it's critical that we have the right go-to-market to monetize it. And so, you know, part of our calculus that we do as a leadership team is constantly looking at the variables for all of those. And there are places and times when we slow down or pause projects, and there are times when we accelerate projects somewhere else. And kind of going back to my comment around the world, you know, the regulation and geopolitics of how AI is going to be used going forward has changed quite a bit. And what Europe thinks about where GPUs can be based has evolved quite a bit, I'd say, in the last 12 to 18 months. And I think it's going to continue to evolve around the world. And so even as we think about the investments we're making, we're also being very, very thoughtful in terms of where geographically are we investing so that we can, again, maximize the AI tokens we can serve in real production applications. And then ultimately use that maximization to ensure that we're delivering a good return on investment for every capital dollar we spend.
Okay. And I have some technology questions for you, but just to keep on speaking about the financing of this stuff because it's so important. So there has been some interesting reporting about how the AI infrastructure buildout has begun to be funded by debt, not just profits. Great story in the Wall Street Journal this week. It says, 'Debt is fueling the next wave of the AI boom.' I'll read the beginning. In the initial years of the AI boom, companies' comparisons to the dot-com bubble didn't make sense. Three years in, growing levels of debt are making them ring truer. Early on, wealthy tech companies were opening their wallets to out-joust each other for leadership in AI. They were spending cash generated largely from advertising and cloud computing businesses. There was no debt-fueled splurge on computing and networking infrastructure like the one that inflated the bubble two and a half decades ago. However, that is starting to happen. Now, OpenAI's deal with Oracle has pushed Oracle to start taking on debt. They say, this is according to the story, analysts at KeyBanc Capital Markets estimated in a recent note that Oracle would have to borrow 25 billion a year over the next four years. Obviously, you guys are not Oracle, but, you know, you're watching this happen as it plays out and see the parallels to the dot-com boom, I'm sure, is not fun. You've been at Microsoft for, I think, 27 years.
28, sorry, I don't want to miss that last year there. So you've seen it. Scott, this seems to be an issue at least from the outside. What do you think about it being on the inside?
Well, I think obviously there's a tremendous amount of spend from lots of different companies. And I would say, yeah, the thing I can speak most to is what we're doing. And kind of, you know, per my comments earlier, I think we're trying to make sure that we have a smart investment play and a long-term strategic play that allows us to ride the AI revolution that we think is going to transform the world and do it in a way that leverages some of the strengths that we have at Microsoft, which is we have a very good cash flow. We have a very diverse portfolio of businesses, in particular in the commercial enterprise space. Whether it's cloud infrastructure, productivity applications, business applications, security, etc., all of them are going to be transformed by AI. And, you know, if you look at, say, to your comment earlier on the Wall Street Journal post, I think if you even read further in the post, you know, it does show the ratios for different companies. And there are some companies that are 400% debt-to-equity ratios and...
And then there are other companies that are much smaller, and that would be Microsoft. And I think, you know, we want to make sure that we're not... And I think, again, based on our capex spend and the rate at which our capex spend is going up, you know, we're not going to sit on the sidelines and not be bold as we invest. And at the same time, you know, I think the thing that our investors expect and ultimately I think every investor of every company will expect is to see that revenue growing in terms of AI services and products that are being delivered in terms of net revenue recognized in a quarter and making sure that the proportionality of that to the spend and in particular to the obligations that maybe are being undertaken with debt are balanced. And, yeah, that's the thing that we've been focused on. I think, you know, if you look at our last quarterly earnings, I think people were pretty pleased with getting the balance right there. And, you know, every quarter going forward, people are obviously going to be looking at making sure that that balance is right so that they see us investing for the long term and going to win and at the same time doing it in a way that is sustainable and allows us to kind of ride through, you know, the ups and downs that inevitably will happen over the next many years as this technology, you know, transforms the world.
What are the consequences if this goes wrong with the debt? Well, you're obviously not taking on the same amount of debt, so there's a rationale behind it. What happens if it breaks?
Well, I mean, we have the ability, we're not constrained. I mean, our borrowing costs, ironically, right now are low.
