AI Inference Is a Data Movement Problem: Microsoft's Saurabh Dighe on the Maia 300

Oh, thank you. Thank you, Austin, for having me. Um, I've really been enjoying your videos on YouTube, so I'm glad we're doing this as well.

Oh, yes, absolutely. We'd love to do that. It was a great conversation in January, and here we are 10 months later. Um, before I dive into Maya itself, which I'm sure we'll do over the next hour, I think it's important for us to explain why Microsoft builds custom chips and systems, right? We have the privilege of running planet-scale infrastructure. And we run a lot of workloads, whether it's SaaS, PaaS, or IaaS, where we get to understand our customers' needs and requirements. And understanding the infrastructure requirements they really need. So, we're not here to build a chip just to build a chip. We're here to really differentiate and deliver value to our customers. And this philosophy for us is not just about a chip story. It's about how the entire system works together to deliver that value to our customers. So, um, we've been on a journey for years in silicon and systems, right? Inside Microsoft, you may have heard about our compute silicon, which is Cobalt. Um, we're developing Maya, of course, for our AI. And we're developing Azure Boost for our front-end networks. And we're developing Azure DPU for storage processing and data processing. And we also have a lot of custom security silicon chips. So, we've adopted this philosophy of building our infrastructure that delivers value to our customers and gives us good freedom to innovate. So that the infrastructure becomes something customers can rely on at scale. So, our real Maya journey started for us with Maya 100, as I mentioned. But this was before the era of large language models (LLMs), right? We were starting to look at: what does it really take to accelerate speech, and sometimes convolutional neural networks? Because that was the growing trend at the time. Then we quickly moved into the world of large language models, where we decided to build Maya 200 to answer the question: how do I build the best AI infrastructure for inference in these models? And this was a bet we took, because a lot of people at that time were only talking about training large models. But we looked to the future and said: although training will be extremely important, distributing this intelligence will be just as important, if not more. And that's where we took an 'inference-first' approach with our Maya system. Again, I want to emphasize that it's not just a chip story, it's about how to build the best AI infrastructure system, and integrating the software, the orchestration processes around it, the model families that need to be moved, the custom silicon itself, the memory subsystem, the networks, the datacenter, and bringing all that together into the question: how do we maximize token throughput for the intelligence we deliver per watt or dollar we invest? So, this is our real philosophy behind the Maya infrastructure. We unveiled Maya 200 earlier this year. And today, we're unveiling the Maya 300 architecture. And in the Maya 300 architecture, we'll talk about the foundations we built our work on. And how we delivered a generational leap in performance per dollar and performance per watt with this new system and architecture.

Absolutely. Absolutely. And yes, I think, you know, since we talked until now, the shape of the workload and the compute profile has definitely changed, right? We're moving from a primarily chat-based experience to long-running agents. And that has definitely pushed us to rethink how we build the infrastructure, and how to provision it for this new agent-based compute profile. And we started to see that the number of tokens each agent generates is much more than humans can generate, right? We started to see more persistent KV memory because the context got longer. We're able to see that the agent plans, then makes a tool call, then ingests new documents, then runs inference, and repeats that loop. So, KV memory became more persistent. We also started to see how important it is for memory bandwidth and memory capacity to be design priorities, and how important it is to pack more efficient compute within given power and capital constraints. So, all these trends and changing factors, we're able to see them firsthand through workloads deployed at scale and how customers use them, and the ability to get telemetry from the systems remotely. And at the same time, designing the entire system to maximize the intelligence you can deliver within those constraints. So, yes, workloads are definitely changing, but I think with Maya 300, we feel very confident that some of the decisions we made now allow us to hit the right cost point, the right latency, the right performance, and treat it as the right service category for that workload. And as we discussed, we tend to talk about workloads, but they're not homogeneous, right? It's more like what we talked about last time, there's a Pareto efficiency curve. And we truly believe that one size doesn't fit all. So I think when we talk about Maya and some of the architectural decisions we made, we made sure the system was balanced, but it also leaned towards what we believe is longer context, i.e., memory-bound and long-learning workloads. This is the area where I believe we can extract the most value from the system. And there are other systems we'll be deploying; we'll deploy a diverse fleet to make sure we cover every point on that Pareto efficiency curve.

Okay, that's definitely what we'll do. I think one of the things we believe as we look at the core principles of how to design an AI inference system, is that AI inference for me is not just a compute problem. It's actually a data movement problem.

And so, we decided that instead of trying to solve the compute problem, let's see how we can create a new class of architecture that really focuses on the energy consumed in moving data. The industry has done an excellent job over the past few years through new process nodes, new processing techniques, new nodes, and new data types, to reduce the energy per arithmetic operation. But what hasn't kept up is how to reduce data movement. Because that makes a big difference, sometimes by two orders of magnitude, in terms of the energy that can be consumed to move data and keep the processing engine running. So, one of the core principles of our architecture is what we call the software-defined dataflow architecture, which we introduced in the 200 series and continue to improve. And the core idea here is that with AI inference, we can actually plan data movement ahead of time. And because we can plan data movement ahead of time, and because we know what these data movement patterns look like, we can have distributed memories, pin data in the right place and time, hide communication and computation, and continue to improve data movement efficiency in this architecture. Important. And now with AI-assisted kernel development, it's become more efficient. We don't need armies of software engineers to program the system, right? We can rely on AI agents to optimize the kernel. So, and to reinforce your point of view, if the work is fundamentally based on large language model inference, we believe that dataflow architecture provides the best efficiency in energy consumption per inferred token. That's where we started, right? At the same time, we said: okay, that's great, but we now need to improve the processing engine and data engines together, right? In a single system. Then we took an approach of distributing memories across the chip. So, we use a lot of static random-access memory (SRAM), and we use it as cache. And so, we make programmability and explicit data movement instructions available to the software layer compiler. We're also able to create much better compute and communication patterns on the chip. We have SRAM with our tensor unit and vector unit, and we also have a universal interconnect network that distributes data within the chips in a single package. And this really helps us again reduce data movement. We went towards an architecture that relies more on non-uniform memory access (NUMA), which means that once data is placed near the processing unit, you won't have to fetch it from another memory controller or a far location on the other side of the chip. So, a lot of thought went into how to keep data movement, storage, and processing at the lowest possible energy. And when we move outside the SoC, we look at it and say: oh, okay. How do we connect these chips together? And often people think it's simple: I'll just put a network and connect these chips together. But we looked at the network as an extension of the dataflow architecture. And that means that the specialized data movement engines (DMAs) and the communication protocol running over the chips in nodes or servers or other racks, can communicate using the same communication model used inside the chip as well. So, it's just an extension of that dataflow architecture, right? And finally, we also made sure to achieve high efficiency at the rack level. Because in the end, what matters is the deployment process. People don't care about the chip alone. They care about: how can the entire deployment process deliver more tokens at a lower operating cost? And that's where our rack architecture comes in, it really helps us build a more efficient deployment scenario in our datacenters. So, yes, there are many layers of optimization starting from the silicon itself all the way to how it's deployed and power is allocated in datacenters.

Of course, absolutely. Absolutely. What we announced today is really our conviction in the Maya inference system architecture, right? We have Maya 200 in production, and it outperforms some of the GPUs in our fleet today. And with that evidence and proof of its efficiency in production, we're saying we'll scale Maya to multi-gigawatt deployments starting in 2027. So, this is basically a very important strategic positioning for us at Microsoft, and it will be a core component in all Microsoft products and services in the future. Now, specifically for Maya 300, what we're saying is that we expect, compared to Maya 200, that this rack-level architecture will deliver a 7x increase in token throughput per rack over what was available in Maya 200. And this will be achieved with 2.5x better performance per watt. So, the interesting story now is how do we get there? I think that's what we're excited to talk about. And when you look at it, we talked a bit about the software-defined dataflow architecture, which is really a core component. We also moved to multi-die, so we adopted TSMC's 3NP process. And with the multi-die architecture, we were really able to balance how much floating-point operations (flops) you want, how much HBM memory you need in the system, and how much I/O or scale-out bandwidth you want in the system. So, we tuned the system further to deliver performance, again, without wasting flops or resources in the system, but delivering that useful intelligence through every watt. So, the multi-die architecture helped us a lot with that. In addition, we brought much better math, i.e., low-precision math, and we can talk about how formats like MXFP4 help us in agentic use cases and with growing and persistent KV memory, in improving our memory, like HBM, right? So, as I said, our HBM memory is a constraint in our environment today, and you use it to store weights as you would with growing KV memory. And with low-precision data formats, we improved weight storage, model weight storage, thanks to its low precision. And at the same time, we can improve KV memory using some of these low precisions. On the compute side, we can do math in FP4 and FP8, so we can continue to maintain the accuracy and quality of model output. And at the same time, pack more and more concurrency and higher KV memory into the same HBM memory. So, a lot of thought went into how to use low-precision math. And also a lot of thought went into how to actually design the memory subsystem. So, as we said, we have SRAM memory on the chip, but at the same time, we also moved to HBM 4. And we have eight stacks of HBM 4 in Maya 300, providing an overall memory bandwidth of 18 terabytes per second for HBM memory. This is about 2.5x what we delivered on the H200 chip. And if you look at the rack level, this is equivalent to 5x what we could do on H200. So, that was definitely our focus. Going back again to how to manage these longer contexts, more persistent KV memory, and memory-bound or instruction-bound workloads.

Right? Um, our scale-up network is the same network we use from the server all the way to the Superpod, which now has 12,288 accelerators in the Superpod configuration and 96 accelerators in the Pod, which we call the rack. Right? So, in a single rack, we'll now have 96 accelerators, which is double what we delivered on H200. And even though we doubled the number of accelerators, we can deliver this at 167 kilowatts per rack, which is extremely power-efficient compared to anything else you'll get from other vendors next year. So, 96 accelerators at 167 kilowatts with the ability to scale using a single scale-up fabric, right? From the rack to the Superpod and up to over 12,000 accelerators, this is what we spent our time on, creating a truly unique scale-up fabric.

Yes, I think that's an excellent question, and it really comes back to the problem we're trying to solve and why, right? And as you said, I think we're still focused on inference as a top priority for us. So, for inference-based workloads, we wanted to make sure we had a balanced system that allows us to scale memory bandwidth and capacity, as we discussed, while continuing to increase the number of floating-point operations. So, we moved to the P process, because we believe it's the most efficient compute node for arithmetic operations. We went with a full-height compute die, but we felt at the same time that moving to two compute dies was not the right architectural choice for the workloads we're seeing. So, instead, we stuck with our desire to have eight HBM stacks with HBM4 technology. And this is where our chiplet-based architecture came in, where we were able to create a memory interface die to extend the edges on both sides of the chip. And so we were able to extend the edges using chiplet architecture, and place four HBM stacks on each side, while keeping a single, very efficient compute die. And now, moving from HBM3E to HBM4, you get a doubling effect. On one hand, because the power consumption per bit is extremely important for moving HBM data. And moving from HBM3...

With HBM4, the industry moved toward a wider interface while keeping pin speeds relatively constant. The result is a much better data transfer rate, along with an improved power consumption profile for that rate. So that is one of the multiplicative effects you get. And of course, the second multiplicative effect is that in the MI300 device, we went from six HBM3E stacks, and now we've arrived at eight stacks of HBM4. So if you combine those two things with our multi-chiplet architecture, we go from performance delivering 7 terabytes per second for a given amount of compute to roughly two and a half times that, 18 terabytes per second. And with an additional increase in compute of about 35 to 37%. So I believe the bandwidth per byte, or the bandwidth per watt, improved noticeably compared to MI300. And with power consumption ranging from 750 watts to 950 watts, with just a 200-watt increase, we were fully aware of the fact that when we provide power for these integrated systems (SOCs) and our systems, that has to translate into much better AI results. And we didn't want to waste any of those resources. And that was the thinking behind the real balancing of the device to fit these workloads.

Yes, I think there's a quick correction, which is that we're moving from HBM3 to HBM4, not 4E. But everything else you said is completely correct. I think it comes down to two things, actually. First, we're optimizing for power efficiency and cost at the same time. When you look at power efficiency, you get a better level of picojoules per bit, or an overall better bandwidth-per-watt profile — when we move to a wider interface, you get a lot more data. At the same time, we also wanted to make sure we were using the right process technology for the right job. So for compute, if you want to reach the performance level you're targeting, you'll run into very bad process technology. For some of the HBM and HBM5 units, we decided to use a different process technology so we could manage the cost and power profile when moving to HBM4 as well. So I believe this is what drives our philosophy again around how we ensure higher efficiency in cost and energy with each new generation. And that's why the decisions to move to chiplet technology, along with the design architecture, helped us a great deal in this regard.

Absolutely. We've done a lot of modeling to understand prefill-to-decode ratios within our technology stack. And we are, in fact, very confident about it. As you know, even with what we currently use in our chips and across the racks, we can certainly deliver the required prefill and decode ratios. I think you raised a good point about what the prefill/decode disaggregation strategy is. And here I think our strength in networking shows, because we have a unified scale-up fabric, don't we? So we can definitely take prefill cases that contain multiple GPUs and put them in one rack or distribute them across racks, with decode cases in the same scale-up fabric. And that really helps us because we don't have to go across an external scale-out network to move KV data from the prefill site to the decode site. We can do all of that across the same scale-up fabric. So we run our fleet in a disaggregated manner, and we can allocate some of our MI300 units as pre-fill instances and some for decode, and when we match their rates, we're trying to align that with the decode side based on the number of prefill devices and the token outputs from those devices.

Yes. Indeed. Okay, I think that's a fair question. Let me build it from the ground up and then we'll talk about it. At the lowest level of our network topology, we have our integrated NICs. We introduced them in MI200. We continue to develop the capabilities of those cards. They are on-die integrated cards, and they are very highly efficient in power and area. And they were really designed for scale-up, weren't they? We're not trying to solve the problem of millions of endpoints communicating across a public cloud network. We're simply trying to look at it this way: what does the scale-up domain look like? It has to be very high performance, doesn't it? It has to have low latency, and we will continue to increase bandwidth. That's what tensor parallelism or any form of distributed workload across nodes in the rack needs. And looking at connectivity patterns, we optimized the network cards. From there, we used Ethernet standards for switches and cables because we again believe in the philosophy of keeping costs down and not resorting to more exotic implementations. We stick with Ethernet to actually take advantage of the economies of scale that sector provides. Then we built a node architecture that keeps some of those links within the server itself, because we again believe that adjacent MI300 units will have much higher connectivity, especially if you're using tensor parallelism. So we take those tensors and distribute them across a few MI300 units, or two units within the node. I think that's where much of the bandwidth inside the node is concentrated. And after that, half of that bandwidth goes to the switch links, right? So that's the first layer of switches. And also, all 96 accelerators have connectivity to the first layer of switches. Then we also decided, what if we wanted to, as models grow or when we started thinking about better and higher partitioning schemes, or providing flexibility in task allocation so that the software allocation task becomes much easier? We felt that a second network layer would really help us scale into those use cases, and we have our own transport protocol over the Ethernet network. So we were able to use some of those transport features to be able to say: "Yes, I can now actually scale out and get a second layer." With that, we can make accelerators in one rack communicate with accelerators in another rack using the same communication stack that was developed for both within a single rack and across racks. This really helps us now reach a stage where we can start scaling the cluster to about 128 racks within the same scale-up domain. You know that copper has a limited reach, right? So we kept using copper within the rack. But once we moved from the in-rack switch to the next level of switches, we started using optical fiber. And you'll start seeing some of those systems next year as well, where you'll see increased use of optical fiber in the scale-up domain. And that's with the goal of reaching that distance and building ever-larger clusters or superpods.

Using optics.