Hey everyone. So, Microsoft is back at $4 trillion. It first got there in July of last year and then it lost more than a third of its value. As investors, they kept asking when all of that AI spending was going to pay off. Well, today Microsoft gave part of its answer: its own MAI chips will become the main system running AI inside of its products at multi-gigawatt scale starting in 2027.
Um, Microsoft says the new MAI 300 gets 40% more AI out of every megawatt than... Excuse me guys, I'm just gonna fix the audio on here. We do this.
Okay, we are here guys. I am back. I'm sorry. Okay, we are jumping on now. Let's continue. Microsoft says that the new MAI 300 gets 40% more AI out of every megawatt than the leading GPU systems. Microsoft isn't alone here. Google, of course, has TPUs. Amazon has Trainium. Meta is building its own chips. OpenAI is even designing its own custom silicon with Broadcom. Now the biggest buyers of Nvidia, they're all trying to rely on Nvidia a little bit less. Look at Microsoft's stack. At the bottom, its own chips. Now at the top, the products that people are actually using. So that's Copilot, GitHub, Foundry. In the middle though, Microsoft is staying neutral. And this is really interesting. That means OpenAI models, its own models from MAI, Anthropic, open source, essentially whatever works here. It is a bet that models end up interchangeable and the money goes to whoever can serve them the cheapest and owns the customer. Now this is also a bet on margins. Microsoft's cloud gross margin has fallen for four quarters in a row and the new chip does not ship in volume until 2027. So, a lot of this, keep in mind, is still a promise. Scott Guthrie runs Microsoft's cloud and AI business. Scott, uh, welcome. It's great to have you on here live.
No, thanks Dre. It's great to be here.
Well, Microsoft has been working on MAI now for a few years and today you're saying that it will become the main system running AI inside of Microsoft products starting in 2027. So, what made this the moment to commit to that?
Yeah, I mean, I think ultimately at Microsoft, we believe that really the future of AI is going to be defined by who can deliver AI that's scalable, affordable, and accessible, and ultimately AI that's trusted by customers in the world that use it. And you know, as Microsoft, we're really focused with our Azure business in being the cloud platform for, you know, millions of customers around the world, serving billions of users. And you know, core to that has been having a data center footprint that's everywhere around the world that meets our customers where they need that technology to be. And then a relentless focus on how do we continue to optimize the cost to serve or the efficiency, if you will, of those data centers and all the software and the hardware that runs in it. And part of that has been working with third-party silicon providers. We have phenomenal partnerships with Nvidia, AMD, Intel and many others. And then also we're building our own silicon in-house as well and we see that as a complement and we'll always have a combination of both what we call first-party, meaning we built our own and designed our own silicon, as well as third-party silicon as well. And we have as part of that our Cobalt program, which is a CPU technology that allows us to serve ARM processors or ARM64 processors in the cloud. We use that for VMs and for standard compute. And then our MAI product line is our silicon that's specifically designed for AI inferencing and allows us to serve AI models, whether those are OpenAI models or a variety of other models including our own, in an incredibly cost-effective way and allows us to kind of generate more AI tokens for lower cost and lower power than other technology.
Right. And so you're talking about the cost to serve and efficiency. Um, talk a little bit about how, you know, serving compute has moved from training to inference. How much now is Microsoft serving in terms of inference? How has that mix shift changed and what do you expect that to be a year from now?
Well, I think you're going to continue to see inference capacity around the world, you know, for both Microsoft and others, continue to grow dramatically. You know, I think obviously everyone is now seeing some of the amazing things you can do with these new frontier models. And as customers look to take advantage of those frontier models, they're going to do much more inferencing, which is basically using those models for business cases. And I think you're going to see the percentage of capacity in the world that's dedicated to inferencing really skyrocket in the years ahead. It's already skyrocketing now, but I think if you fast forward where we'll be in five or 10 years from now, it'll certainly be the dominant amount of capacity in the world. And that's partly why the cost makes such a big difference. And so if you can lower the cost curve dramatically and if you can ultimately get more AI out of every dollar of capex spent and every, you know, watt of power that you're drawing inside your data centers, you know, you can really differentiate yourself in the market and that's what we're looking to do with MAI. MAI 200 is already in production today. That's the program that we launched about a year ago. We're seeing some great results with that and we're already running Microsoft Copilot and our Copilot and Teams on top of it, so we're using our own first-party services. And MAI 300, which is what we announced today, is the next evolution of that silicon and it delivers even better performance and better scale at much, much better cost. And you're going to see us start to deploy that in a multi-gigawatt size deployment starting in 2027, so only in a few more months. And, you know, that'll continue in the years ahead and ultimately probably become the dominant technology that we use as part of our inferencing fleet.
Right. So talk a little bit more about that. You sort of, you've talked about how Microsoft has always used a mix of third-party. You guys have been developing your own silicon. Um, but this is a big move because you're saying essentially that it will be the main system, MAI 300 will be the main system in the future running AI inside of your products. Um, so if I'm a Copilot user, like how quickly does this scale up at the end of 2027? What are the odds that if I'm a Copilot user, my query is running on your chip versus an Nvidia GPU?
Yeah, it's the nice thing if you're using Copilot or our Copilot or Teams offerings, it's sort of transparent as to what silicon you're actually using. That's not something that a user has to opt into. There's no features that they lose if they're using a particular silicon or the other. Instead, that's sort of a decision that we make based on the particular location of that data center and based on where it is in the overall evolution of the silicon deployment wave. And so the nice thing is because it's transparent to users, we don't need them to take any action in order to opt into a different silicon. It gives us sort of flexibility to be able to use that silicon for the use cases where it most benefits. And so, you know, we're using it already in cases where we know we can get a very big cost benefit today. And you'll see over time more and more of those use cases go live and, you know, start to be used by our customers in more and more of their interactions with us. And so you'll just sort of see it become a greater percentage of our overall deployments. And, you know, we'll scale it as we roll it out and it'll be fairly transparent to customers other than they'll get better performance and ultimately allow us to continue to bend the cost curve, which they obviously care about as well. You know, having said that, we have a fantastic partnership with Nvidia. We also have a great partnership with AMD also in the silicon side for AI and we're going to continue to be great partners with them. And this is not going to be a winner-take-all. I don't think whether that's true around silicon or whether that's true around models. I think the future of AI is not going to be defined by a single model or a single silicon technology. And part of what Microsoft's always focused on has been being a great partner and making sure that we support choice and that we actually use the best technology for the specific scenario involved and we'll continue to do that as well.
Right. In a supply-constrained world, you're using everything, but is the goal to ultimately serve 100% of inference on MAI 300?
Um, I don't know if we'll ever get to 100%. But our goal is certainly to ultimately serve probably the majority of our first-party scenarios. And, you know, for a variety of use cases, there might be reasons why it's never quite 100%. And we want to always maintain the flexibility where we can use multiple different technologies. Sometimes if there's a different cost curve for different scenarios, I think you'll see differences between inference and training as an example in terms of the technologies that we use. But you'll definitely see us continue to ramp the MAI program in terms of volume quite significantly. And we're obviously working closely with our supply chain partners right now as we kind of plan not just for next year but really for the next multiple years as we continue to ramp and as we continue to build out the cloud around the world.
Will you ultimately serve Microsoft cloud customers inference on MAI 300 on your architecture?
Yes. Yeah. So for our Microsoft customers that use both Copilot, Copilot, Microsoft Foundry, which is our API serving system that's part of Azure, you know, that will all take advantage of MAI.
I mean, sorry, I mean other, not just that are on Copilot, but will you be able to eventually offer it to Azure customers?
Yes. Yeah. Through our Microsoft Foundry offering, that's a way that any customer can build their own applications using AI models that are hosted on Azure and that will include models that are hosted on top of MAI. So yeah, any customer, not just ones that are using Copilot or Copilot, will be able to use MAI 300 processors and be able to get the same kind of performance and cost benefits that they provide.
Do you have any of those customers lined up yet?
Well, we have a large number of Microsoft Foundry customers. I think we've got an enormous amount of enterprise customers as well as startups and ISVs around the world that are using Microsoft Foundry today. And again, the good news is none of them have to make any changes to their applications in order to leverage MAI when MAI is running underneath their Foundry endpoint or the API that they're hitting from their application. They'll just benefit from it. So it's nice about sort of the way we're designing and architecting our systems. You know, we're kind of trying to make sure we don't force customers to have to take explicit steps in order to benefit. Instead, we want to make it much more transparent and have them just be able to benefit in terms of both the performance and the cost savings without having to do anything as part of it.
Will any of the labs, I mean, you guys are invested in OpenAI and Anthropic, do you see a future in which they're able to serve inference on MAI 300 chips or any MAI chips?
