One, two. One, two. Okay, much better. Much better. All right, thank you. I think I'm the first speaker, at least amongst the first speakers. Happy to open up OCP. Thanks for being here, everyone. Just all right, my slides are up.
So, I'm going to be talking about d-Matrix and what we're doing at d-Matrix. We are an inference compute company. We started the company in 2019 with a singular mission to build the world's most efficient inference computing solution to be deployed at scale, right? And what, again, keep in mind, 2019 was very early. Not many people were even talking about inference. You know, I used to spend a lot of time just educating people about what inference really meant, why would you need a dedicated computing solution for inference. But that gave us the time to build something fundamentally different. We built a solution that is focused on memory-centric compute, and what that allowed us to do is put memory and compute into a fully integrated substrate and, you know, essentially punch through what everyone is calling the memory wall today and focus on some of the most critical bottlenecks of AI inference around latency, energy efficiency, etc. Right? And then that's just one piece of it. Right? Now you got to take that, you got to put systems and software and racks around that solution and bring it to market, and that is what I want to focus on in today's talk, you know, how did we bring the solution to market. So there's, you know, quite a bit to cover, so let me jump right into it.
So, you know, most of you are familiar with the ultra-low latency segment of the market. This segment of the market has really taken off in a very big way in the last 12 months, right? And this is something that naturally comes to d-Matrix because we built a solution based on memory-centric compute which really attacks the latency problem. Now, you know, we truly entered the agentic era in the last, you know, 6 to 8 months, right? We had coding agents, we have personal agents. Now in the last few months we have cybersecurity agents, right? In all of these cases, latency matters a lot. And why is that? Because, you know, you have users trying to interact with AI in real time, they want quick responses. The underlying models that run all of this, you know, these are reasoning models that think and reflect before they respond, and you want them to do that quickly. So, you know, obviously latency matters. Rogue agents, you know, running through an organization, you want to shut them down quickly. You know, latency matters. So, in general, latency has become like an underlying theme for pretty much everything that we're doing in AI right now, especially after inference has taken off in a very big way.
And so, that's what you're seeing on the chart there. I mean, this is a chart of token consumption. And it is not just a chart of token consumption. It is really a chart of the consumption of inference. It is a chart for the consumption of intelligence. And while it looks like, you know, we need lots of it, it's almost like we need infinite amounts of this. We live in a world where resources are finite, right? So we have a limited amount of energy. You know, there's only so many data centers that we can build and capital is scarce, right? So how do we solve this problem? By not brute forcing it, right? How do we build a solution that can do more with less? And that is what we've been focusing on at d-Matrix. We call it the world of infinite inference with no latency. We want to serve it, you know, in real time and we want to do it in a way where we do more with less.
So let's take a step back. You know, lots has happened in the last 12 months. But, you know, if you go back to the last 10 years, you know, really the predominant computing solution was GPUs, right? And most of it had to do with the fact that we spent a lot of time training models. Then we had ChatGPT that happened in 2022, and that was the first time people started worrying about inference and what it takes to deploy inference at scale, the economics, the energy needed to deploy inference. But it was still predominantly done on GPUs, right? 2025 was really a big watershed year for inference because you had the DeepSeek moment in the early part of 2025 that unleashed, you know, open-source reasoning models became available to everyone, led to a lot of innovation at the application layer, and that eventually led to a step function improvement in coding tools in Q4 of 2025. So you had GPT Codex, you had Claude Code, all showing a step function improvement in capability. And right around that time you also had the OpenClaw moment, right? That was another watershed moment. So you had an agentic harness that showed up which could essentially connect all these things together and really make it easy for someone to use these tools. Right? Since then, in 2026, we have entered the agentic era. We have had, you know, lots of progress on coding tools. We have had personal agents like Dots and Instinct and Muse show up, and these are really, really capable with fully harnessed infrastructure, right? So users don't need to even try to figure out how to put this all together, right? And that has truly allowed us to enter the world of heterogeneous compute because it became very clear to everyone that GPUs alone won't cut it anymore. You need different types of compute. CPUs made a comeback, other forms of XPUs like TPUs, etc., made a comeback, and we are now living in a world where all these forms of compute need to coexist to enable the world that we are about to enter.
Now, if you look at the inferencing workload, it's really kind of broken into two fundamental parts. There's prefill, there is decode. Prefill is what we call the thinking part. Decode is what you call the speaking part. If I were to ask you a question right now, you know, where are you going to, you know, where do you plan to have dinner tonight? Right? You're going to be thinking a little bit about, you know, where are you going to be tonight? What kind of food do you feel like eating tonight? That is the thinking part. That is the prefill part. You're contextualizing the question and then storing all of that in memory, and at some point you're ready to respond. You start speaking the response where latency begins to matter, right? You want to speak the response at a certain pace. So latency matters, and you're kind of pulling from memory when you speak that response. So the first part, prefill, compute-bound, you're thinking. Second part, speaking, latency-bound, latency-sensitive, but memory-bound, you're pulling from memory, right? And at d-Matrix, given that we have built a memory-centric computing solution, we do really well at the second part, the decode part. But in some cases we also do really well with the first part, which is the prefill. So if you have kind of low batch throughput that you need, you can pair that up with our solution and you can do whole-model inferencing with our solution, or you can pair up with a GPU. If you need a lot of throughput and you need low latency, then you can pair up with a GPU. So this is what we're showing here. We can do full-model inferencing in certain applications on the d-Matrix solution, but we can also pair up with a GPU where you need a lot of throughput and low latency.
So this is a chart that most of you know, you are probably now quite familiar with. This is the Pareto chart for AI inference. Right? Once you understand this chart, you pretty much understand all the different trade-offs. On the y-axis you see throughput per kilowatt. Throughput is the amount of work you need to get done. On the x-axis you see interactivity, which is speed. And the two of them, you know, typically pull in the opposite direction, right? So, you know, if you have more work you need to get done, you need a lot more time to do it, right? But if you want to do something really, really fast, you really can't do too much of it, right? So, you know, naturally both of them pull in opposite directions, and it's very difficult to have a solution that can do both equally well. Now, GPUs are very good at throughput. They're workhorses. So they do very well on the y-axis. Solutions like d-Matrix, which are memory-centric, do extremely well on the x-axis. Right? So in that sense the two solutions are complementary, and you can pair them up to address specific challenges that a workload or an AI model might try to resolve where you need a lot of throughput and you need low latency. So that's where the two solutions can be put together in multiple different ways, and there's really three primary ways in which GPUs can be paired up with fast XPUs like d-Matrix, and let me just quickly touch upon the three techniques. So from the very left you have something called speculative decode, where this is work that we have done at d-Matrix. We have, you know, published work that you can find and you can take a look at. Speculative decode is where the entire model, the large model, the parent model is running on the GPU, but you have the small student or draft model that is running on the fast XPU, and all it is doing is speculating on outcomes, right? And the GPU can look at those outcomes and quickly pick the ones it likes. Right? So that is one way of doing disaggregation. There's something called AFD, or attention feed-forward disaggregation, which is also becoming quite popular. And what that does is it takes the prefill part, which is the thinking part, that runs on the GPU, but you take the decode part and also break it up. Okay? So you take the attention piece, which is, you know, typically tends to be compute, you put that on the GPU, and all the experts in a mixture-of-experts architecture would run on the fast XPU, right? So that is another way of doing disaggregation. The third one is a very traditional PDD, or prefill-decode, where you run all prefill on the GPU and you run all decode on a fast XPU. Right? So again, you know, there's different trade-offs involved depending on whether you're optimizing for accuracy, optimizing for speed. You can pick any one of these techniques, and all three are becoming quite popular.
So let's quickly understand why the GPU is really not optimized for latency, right? And that is because of the memory architecture. You have a GPU, you have HBM memory, right? You have data sitting in HBM. You need to pull that data from HBM, you're typically restricted by the beachfront of the HBM and the GPU. So there's only so much bandwidth you can really incorporate in that connection between the GPU and memory. So at d-Matrix, you know, we were like, why don't we find a different way of putting memory and compute together? What if we put the two together onto a common substrate? What would that look like? Wouldn't that get us a lot of efficiency? And that is exactly what we did. Our first product, Corsair, was built with a fully integrated memory and compute, you know, substrate, right? So essentially we took the world's most ubiquitous memory, SRAM memory, and we paired it up with compute, and we built this, you know, meshed fabric of compute and memory, and that became our first product. And this is now in production, and today we are announcing general availability of the product in a rack-scale form factor in partnership with Supermicro. So we have our Corsair compute cards which have that silicon I just showed you. We have our Jetstream scale-out networking fabric, right? We put these cards into a Supermicro server, and we can rack and stack these servers and connect them through the Jetstream networking solution and create a solution that has 128 GB of SRAM in a single rack, all air-cooled. So this is in production today, and, you know, fun fact for you guys: at d-Matrix we have labeled or named all our products after fighter jets, and so Corsair is a fighter jet, and we are ready for takeoff. We like to say, you know, we are offtaking our customers' applications at this point, and we are ready for takeoff with our first product. So I thought I'd play a fun video for you guys.
Unfortunately, we don't have the sound effects. So, I'll have to probably do them myself. [laughter] [panting] But anyway, so the product is in production. We are generally available. Compute is scarce and in short supply. So today we are announcing Squadron, which is our first compute cluster. This is deployed in Charleston, Arizona. And it is a multi-megawatt cluster of compute, 1,024 XPUs, 16 racks, and really being used to offtake customers who want to bring their applications and take them to production. Right? So very exciting time for d-Matrix as we start deploying our compute at scale.
We are also announcing one of the world's first AFD demos. So this is an attention feed-forward disaggregation demo with an NVIDIA Hopper GPU. So what we're doing is we have — I talked a little bit about AFD earlier in the presentation. So a portion of the model — this is the GPT-OSS 20B model, and, you know, the next one that we'll be doing later in the quarter will be a GPT-OSS 120B model where we'll be partitioning this model using AFD techniques across the Hopper GPU and a d-Matrix XPU. And we are able to get to, you know, north of 800 tokens per second per user with a concurrency of 64 users simultaneously using this setup. So this is something you can check out at our booth. I know the trade show is not open today, but when it opens up tomorrow, please stop by our booth and you'll be able to take a look at this demo.
All right, so let me touch upon what's coming next. We talked about our first-generation product, Corsair. We are not stopping there. Our next product is a product called Raptor, which is going to be the world's first 3D-stacked DRAM XPU. And why are we doing that? Because clearly the models are getting bigger. Context lengths are getting much larger. We need more memory capacity. And SRAM, which is what our first product started with, does not scale very well with process technology. We don't get the amount of capacity that we need. So we decided we're going to introduce a new layer of memory with DRAM between the SRAM XPU, which is at the top, and the substrate. It's like a sandwich. The red tile is the DRAM, the compute tile is at the top, and then the substrate at the bottom. And this gives us a lot more capacity because now you essentially have still 10 times the memory bandwidth with more capacity compared to HBM, but you're doing it all at one-tenth the energy. Right? So it's a very compelling solution. It will be coming to market next year. But we want to be able to speed up the deployment of the solution as quickly as we can. And that's why a few weeks back we announced a landmark partnership with NVIDIA where we can take that XPU and build a compute tray that can plug into NVIDIA's rack infrastructure. So what I'm showing here is the Raptor compute tray, which is being co-designed with NVIDIA, and that plugs into an MGX Oberon rack that NVIDIA already builds along with the NVLink NVSwitch trays. So no changes there. We reuse a lot of the other stuff in the rack. We reuse the CX-9. We reuse the DPUs. We reuse the Vera CPUs. Essentially, this is a Vera Rubin rack where the Rubin trays come out and the Raptor trays go in. So, it's essentially a Vera Raptor rack with a 144 scale-up domain with 144 Raptor XPUs in that rack, right? And obviously that makes it very easy for us to take the solution to market because NVIDIA's AI factory ecosystem is in every cloud, every data center, every country. And what we want to do is not have to reinvent that portion of the infrastructure, but be able to plug in and rapidly ramp our product. Right? So this is some more specs on what that rack would look like. Very, very similar to an existing Vera Rubin rack except that it is now with Raptor XPUs.
Now, some of our customers, you know, also deploy Ethernet-based scale-up, right? So this week we announced a partnership with Arista where our Raptor XPU will go into a rack that will support UALink-based scale-up. So we will have two options. We have the NVLink scale-up option with NVIDIA's rack, but we'll also be building a rack solution that will be doing UALink-based scale-up. And this is something, you know, will be available later. We'll be starting to talk about this later this year. So stay tuned. We'll have more details about what this rack is going to look like as we start rolling out the details.
And we're not stopping there either, right? So, we're talking about what comes after Raptor, and that is a product called Lightning, and this is going to continue the journey in terms of stacking more DRAM on top of XPU. You know, essentially look at it this way. We're building out a skyscraper. We put the first layer with Corsair, that was SRAM-based compute. Raptor was adding in the first floor, and now with Lightning we are adding, you know, even more floors and more capacity, right? So this is a product that we will be rolling out more details in the near future, so stay tuned. But this is how it all comes together. So the roadmap looks like this. We have our Corsair XPU that can do fast inferencing with SRAM, 100 billion parameter models in a single rack, air-cooled. We also have an LPDDR tier in that same rack that can do 10 terabytes of memory for any kind of spillover for KV caches, long context, you know, concurrent users, etc. So that is our first product in full production. After that we go to Raptor, which increases the capacity by a factor of 30x. Right? So we go from 100 billion in a rack running out of fast memory to 3 trillion in a rack running out of fast memory with the DRAM tier. Right? And that option, the Raptor option, is going to be available in both NVLink and UALink. And then eventually we move to the Lightning product, which can do 20 trillion parameter models in a single rack running out of fast memory. Right? Again, we'll be available with both the NVLink option and the UALink option. Okay. So that's kind of how we'll be rolling out our roadmap and keep increasing memory bandwidth and memory capacity to support bigger models and longer contexts over time.
All right. So, you know, we've been planning for a world of infinite inference, and, you know, how do we serve all that inference with the fastest levels of interactivity and not do it in a brute-force way? And the way we did it was we essentially reimagined our computing substrate. We are using SRAM, we use, you know, 3D-stacked DRAM as I just showed you. We don't use any HBM in our solution, so we find a way to kind of navigate around the supply constraints on HBM. And we are complementary, right? So there is the HBM-based solutions, whether it's a GPU or a lot of the other XPUs that use HBM, are really focused on throughput. At d-Matrix, because of the architecture we use, we are really focused on the interactivity piece, which is very complementary, and so we can coexist alongside GPUs and other XPUs that use HBM and focus on the interactivity piece, right? And that is the reason we did these partnerships, whether it's with Supermicro, you know, partnering with NVIDIA, or it is our NVIDIA partnership inside the Oberon rack, or the Arista partnership which allows us to use existing rack infrastructure and not have to reinvent portions of the infrastructure that do not need reinvention, and allows us to take our product to market a lot faster. Okay? It's a very versatile product in that it can do whole-model inferencing for certain applications but can also do disaggregated inferencing for certain types of applications. So anyway, please do stop by our booth. Yeah, we are in booth C82. The trade show will open tomorrow. You know, my team will be there, I'll be there, and would love to chat with some of you who will visit and continue this conversation. But thanks for listening. Thank you. [applause]
