The New Era Of AI Inference Memory | Sid Sheth, CEO D-Matrix

We've raised about 500 million in capital along the way. So we have spent less than 100 million a year. And by the way, we still have 100 million in the bank.

We started in 2019 and you know back then inference was not even a word. We started d-Matrix, said we want to go build a clean slate inferencing solution. Now we are kind of center of this tsunami of compute demand.

Really comes down to the margin structure at OpenAI. You know they are running businesses, right? You know somebody like a Moonshot or Zhipu, you know they're not thinking that way, right? We started working on this DRAM-stacked approach back in 2022, 2023. Today I don't think it's truly really gone mainstream yet. I can guarantee you if you're doing this pod and you know call it two years from now, it is going to be absolutely mainstream.

Thank you, thank you, Sam. Thanks for having me.

It's wonderful. It's been a wonderful summer. You know, it's getting a little warm, but I've been spending, you know, just the right amount of time outdoors catching some soccer action. We had a wonderful World Cup that just wrapped up and my younger son, he's a big soccer fan and is a fan of Portugal and Ronaldo, and we have this healthy rivalry in the family because I'm a big Argentina fan and so we are constantly bantering on that. I'm glad my team made it to the final and we actually caught, you know, the final live and it was a lot of fun. So it's been fun. It's been fun.

Yeah.

Yeah. So you know, d-Matrix, we are an inference computing solution, right? And when I say solution, we are really building the silicon, we are building the subsystems, we are building the software obviously that goes along with the silicon and the subsystems. But then we partner with the right server vendors, the right rack-scale vendors to create a solution that can be deployed at rack scale in a data center, right? So that's kind of what we do. Our core engineering effort within the company is focused on building the silicon, the software and the subsystems, right? That's where we want to stay focused and then partner everywhere else to take the product to market and assembling a solution. We started in 2019 and you know back then inference was not even a word, right? I mean we went out and we would talk to investors and you know customers were the only ones who were looking forward, like some of the very large customers were the only ones looking far out and saying inference is going to be a very big workload and you know the amount of compute needed to run inference is going to be just, you know, it's going to dwarf everything that we're seeing today. So with that thesis in mind we started d-Matrix that said we want to go build a clean slate inferencing solution. What does that even look like? We don't want to take any legacy from the past. If we had to start clean slate today, what kind of decisions would we make? What kind of constraints would we go solve for? What kind of bottlenecks would we go solve for? And how would we optimize that solution? So that gave us like a really kind of the freedom to build a solution from the ground up, right? And along the way it's been a phenomenal journey because we haven't really changed anything we do here at d-Matrix by the way, right? We started with that simple vision and we are still doing exactly the same thing. We like to think the world came to us, right? I mean inference became more and more relevant as time went on and things have just happened. Every year almost there's been some kind of a watershed event that happened every year that made inference even more significant to the whole world. And now we are kind of, you know, in the center of this tsunami of compute demand around inference and token consumption and so the d-Matrix story is more relevant than ever before. And the good news is, you know, because our focus was so singular on the inference opportunity we made, you know, some decisions on the technology side, on the go-to-market side which we'll talk about that allowed for the solution to be highly optimized for the inference computing opportunity, right? So yeah, no, I think it's, we've, you know, we've raised about 500 million in capital along the way in three rounds of financing. You know folks like Playground Global, Microsoft M12, you know Temasek, Qatar Investment Authority, Industry Ventures, Samsung, SK, Ericsson, LAM Research, a combination of CVCs and financials and sovereigns that have funded this company. I'd like, you know, we've been very capital efficient. You know, seven years and 500 million, so we have spent less than 100 million a year and by the way we still have 100 million in the bank from that 500 million that I just referred to. So call it 400 million spent in seven years, so that's, you know, and one product in production and a second one on the way with a team of call it 200 to 300 people. I think we have been tremendously capital efficient, which you know you learn to when you spend 30 years in the semiconductor industry, that's the first thing you learn is it's all about, you know, droughts and famines and then suddenly there is a feast. And when you're in the feast, you are conserving for the drought and the famine that may come, you know. So yeah, so we've been quite capital efficient along the way and look forward to what's next. I mean, super excited about where this will go.

You know, really getting to the best cost per dollar per intelligence, you know, for intelligence, you know, for every dollar you're spending, getting the most intelligence possible.

Yeah. So I come from a family of entrepreneurs. I grew up in India. So I'm a third generation entrepreneur. Both my maternal and paternal grandfathers were both entrepreneurs. Actually every generation in my family has done something that was a first in the family for that generation, right? So it's getting harder by the time it got to me. [laughter] So because you know, it's, you know, the world is so small these days. It's very difficult to venture out into places that someone hasn't ventured out to. But for example, both my maternal and paternal grandfathers left their rural towns, went to the big cities in their teens. You know, learned the trade, right, by just working. In the case of my paternal grandfather, just being a paper delivery boy. He learned the intricacies of the paper business, right? And then eventually started his own business, essentially trading paper. And of course these days, you know, paper is, you know, we don't do all our business using paper. We still do a fair amount but in those days it was all done, you know, with paper. My maternal grandfather, same thing, left his rural town and went to the big city, learned what it takes to sell auto spare parts and eventually built his own auto spare part factory and vertically integrated. And my dad was the first to come to the United States in 1958. By the way, this will, you know, I know '58 was a very early time when people from India came here to study. Now it's very common, right? But 1958 was, you know, actually making your way here was an expedition, right? And you hardly had anybody, you know, you knew here. So you know, coming here, settling down here, you know, studying here, he spent about 14 years in this country and then went back eventually and set up his own manufacturing plant and became an entrepreneur and whatnot. So I grew up in an environment where I was constantly watching this, right? So you know, as a young kid, I mean, you just learn that entrepreneurship is a way of life. It is, there is no other way. And I don't think in my family the concept of working for anyone was common, right? It's like they just were not even aware of what that even felt like. And I used to get that a lot because when I started my career, you know, in the US, I started at Intel and I would, you know, visit my family back in India and the first question would be, so you know, have you started your own business? And I'm like, no, I'm working for Intel. What, you're working for? What does that even feel like? What is it working for somebody else? What does that even feel like, right? And so the pressure was always there and the push was always there to kind of go off and do your own thing and it was in my DNA. So it was a matter of time. I think I lasted, you know, a grand sum of four years at Intel and then I've never gone and worked for a big company after that. I've either incubated businesses in companies or have been at startups and this is my third or fourth time, you know, kind of incubating a business, right? And this time of course I'm doing it as CEO and founder, right? So essentially the buck stops at me now. The gun is to my head as I like to say. [laughter] But that's something I've been wanting to experience for a long time because, you know, that's something that has run in the family and I think I was just waiting for the right opportunity to come along. And I felt when AI happened or was happening, you know, in the late 2010s, that was my opportunity, right? That was when I felt like, you know, here is something that is going to become big enough. So I was so excited by what the potential was that I felt it was time for me to step out and go start my own company. And especially in semiconductors, which by the way you may remember, I mean, has been through a pretty lean patch from the early 2000s to the mid-2010s, right? I mean, so call it 2000 to 2015, there were years where there was not a single Series A company that even got funded, right? So we saw years where there was not a single semiconductor company getting funded. And so this is very different. I mean the period that we are living through right now is just diametrically opposite to what I've seen, right? So I think it was the right time for me to step out and yeah, here I am doing my own company and have never been more excited.

Yeah, yeah, yeah. So word on my co-founder, right, first. And I think he's a brilliant, brilliant guy, right? I mean, he sees things that most people do not, right? Because he has this uncanny ability to simplify things and distill them down to the bare essentials, right? Yeah, exactly. I think what I see in a lot of very smart people is they like to, and it's natural, like you like to take a problem and you like to believe that it is more complex than it really is and then you want to solve that very complex problem and then you want to tell people like, look, you know, wow, this problem is so complex and I was able to solve it, right? And smart people want to solve complex problems, that's a natural tendency, but he is different in the sense that he has this unique ability to take a very complex problem and break it down into something that's really, really simple, right? And then, you know, typically once you've done that, the solution is also equally simple, right? And so that is his immense ability. And so we actually started working together at my previous, you know, I was incubating this business at a company called Inphi which was focused on data center interconnects and he was at Broadcom working on similar stuff, you know, basically building physical layer products for enterprise networks. So you know, I kind of tried to convince him to come join me as I incubated this business, which he did. And so we essentially worked together at Inphi for about seven or eight years. And I've known him before that, you know, we went to grad school together, we overlapped there, you know, but then he went off in a different direction working for TI and Lucent and then eventually ended up at Broadcom. But you know, we stayed in touch and you know, obviously we were in the same industry so you know, there was reasons to talk about stuff together. But we really worked for the first time at Inphi and you know, he joined pretty early in the journey and we set that business up over a period of seven, eight years. That business, you know, eventually by the time it got acquired by Marvell in a $10 billion acquisition was running at about $3 billion in revenue annually, right? And currently is probably north of four plus billion, is probably 50 plus percent of Marvell's revenue, so it's a franchise business for Marvell. And the reason and the way it all happened is really, you know, goes back to, you know, what I just stated earlier is like you take a technology transition, you look forward to some kind of tech transition that is happening in the industry and you figure out a new way of riding that transition which gives you an advantage as the incomer, right? The incoming person always has an advantage over the incumbent. The incumbent is kind of tethered to the past, the incoming person is not tethered to anything and that is the opening. And that's exactly what we did. And you know, back then technology was transitioning away from NRZ signaling to four-level signaling and you know, what they call PAM4 signaling and we created these DSP chips that would be able to serve that opportunity. And Broadcom, which was the incumbent, was kind of tethered to the past and took longer to make that transition. We ran through the transition much faster and were able to build this franchise business, right? And we'll talk about the inferencing opportunity, very similar, right? I mean, it's a transition and you know, how you ride through that transition is going to hold the key, right? But yeah, I mean, so we've been working together now for what, 15 plus years, right? And you know, he focuses a lot on the tech and I focus a lot on the business and the go-to-market and you know, obviously, you know, fundraising etc. But when we started in 2019, I mean, as I told you, inference was not even a word, right? And so the way we looked at it was we were like, look, let's, you know, we have comfortable jobs at Inphi. Both of us were, you know, very well paid. The business was doing really well. It had, you know, started ramping. So there was really no reason for any logical person would not want to quit that, you know, having spent eight years building it to a point where it is just about to inflect and then you say, you know what, I'm done here. I'm going to go start again from scratch. Only a very stupid person would even do that, right? And we were told, you know, by multiple folks, guys, you're crazy. You shouldn't be doing this. You spent eight years building this thing up and now it's ready to inflect and you want to step out. But you know, that's, we just felt the opportunity in front of us was just so much bigger that we had to do it, right? So we stepped out and we ideated the company for a solid one year before we incorporated the company because, you know, we didn't want to just go out and do a company for the sake of doing a company. You know, if you're going to do this, it better be against an opportunity that's very large. We're really excited about it. We really believe in it. So we ideated for a year. We called up, you know, numerous investors, both corporate and financials. We always felt we wanted some kind of corporate backing from the early days because we did not want to build a product in so early. The market was just so early that we didn't want to build a product in isolation. We wanted to build something that was aligned with somebody's business plan, right? And in our case what happened was Inphi wrote the first check. So we were ready to step out. You know, my boss tried really hard to kind of hold on to us and you know, he's a phenomenal guy. He was obviously very disappointed to see us go so he decided to invest and wrote the first check into the company and you know, helped us get on our way. And you know, after that it's been, you know, making calls to the first. So the first check was really a corporate check, right? And then eventually we started reaching financials, right? And we started talking to various seed funds and we got another $3 million check about six months into the company, right, in the middle of the pandemic. Pandemic had just started. We'll talk about that, you know, if you have time here, but you know, what that did to our journey. But we managed to just get enough and with a, you know, scant crew of 10 people were able to essentially tape out our first chip in 12 months of starting this company which, you know, proved the thesis of in-memory compute, right? And what does it take to build a computing platform that is memory-centric where compute and memory are not two separate blocks and that allowed us to build that first initial conviction that we were on the right track and then of course we kept going and the rest is history, right? [clears throat]

Yeah. I think we had a very unique pathbreaking approach, right? I mean the whole concept of in-memory compute, people were doing near-memory compute and you know, there was, but this was, we were essentially taking SRAM memory and augmenting it with more transistors to do matrix math, right? So the idea was very novel. We didn't want to, we really felt it was a, you know, we needed to prove out the technology. And I think the feedback we got from pretty much every investor we spoke to saying, wow, this is pathbreaking, but question number one, who are the big customers here? And in our case we were targeting the data center market. So okay, great, do data center customers actually want to deploy this technology? Have you spoken to them? And so we spent a lot of time with customers and given that, you know, we had built a business targeting data centers coming from Inphi, we knew a lot of people at these target customers so we could have those conversations pretty early on in the process about, look, this is what we're building, what is right about it, what is not right about it, you know, where should we pivot etc. And so we got a lot of feedback pretty early on. So really the first tape out was all about proving to folks that this core idea of memory-centric compute is actually achievable and not only is it achievable, it is credible. It was something that is deployable in a data center and can be deployed at scale because you can flush out a lot of the technology risks in the underlying platform. So really that was the fundamental goal.

Yeah. So really, I mean, if you go to see the different types of memory that are in most prevalent use today is SRAM which is on-chip, right? There is DRAM which, you know, depending on how you package the DRAM can be co-packaged with the processing engine or can be sitting, you know, further away from the processing engine. And then there is flash, right, which is kind of the third tier, right? I mean, furthest away from the processing or the destination, right, of the data. So really depends on, and you know, what you're trading off is cost and capacity, right? And you know, if you're the closest to, and performance, of course, if you're closest, you know, to the processing engine, you have the best performance, the lowest amount of capacity, most expensive. As you go further away from the processing engine, you know, you have lots and lots of capacity. You know, very inexpensive, but then your performance is going to take a hit, right? So you're really trading off depending on the type of application you're optimizing for, which tier of memory do you want to use in your application, right? And we all know that AI is extremely data hungry so keeping the data too far away from the processing engine is not going to help you. So the question then becomes, how do you architect a memory hierarchy along with the processing engine or the compute engine that would allow the compute to access data as quickly as possible? And then of, you know, other things matter too, right? I mean, models are getting bigger so guess what, I mean, if it's going to come closer to the processing engine then how are you going to also account for the fact that you need to increase your scale, your capacity as the models get bigger and how do you keep cost down because inference is all about, you know, cost, right? And you know, dollars per token, we'll come to that, but you know, how do you keep your dollars per token attractive, right? So you're kind of optimizing along all those vectors and that's what makes this a hard problem, right? It's like, what kind of memory technology do you pick to kind of couple with your processing engine? And GPUs picked HBM. You know, SRAM is available to everyone, right? I mean, every chip today has SRAM on-chip. The question is how much capacity of SRAM do you want to dedicate to the chip, right? So that is something everyone decides depending on what they're optimizing for. And then of course there is LPDDR which is a type of DRAM too, right? More capacity focused, not as performance focused. HBM is more performance focused, doesn't have the same capacity as LPDDR. There is GDDR memory which is kind of a tweener, right? And then there is flash which is obviously, you know, kind of for cold storage and you know, anything that is not hot data can be kept in flash which is further away from the processing engine. So really these are the options you've got to play with, right? And you got to decide how you want to kind of mix and match and marry this to your computing engine. And in our case we started with SRAM today, right? And we optimized that chip for more SRAM so that we can kind of have the best-in-class performance. And where we are going next is essentially a very new direction for this industry and you know, we are one of the pioneers there, is taking DRAM and instead of co-packaging it with your processing engine, you stack it directly on top of the processing engine. So, you know, look at the HBM skyscraper. The HBM skyscraper and the GPU. You know, if I were to kind of draw an analogy here, the GPU is like a processing center and the HBM skyscraper is like the warehouse. The warehouse and the processing center are on two sides of the street, right? Opposite sides of the street. So every time you want to fetch a packet, you got to cross the street and that's latency and energy. In our case, we have taken the warehouse and placed it directly on top of the processing center. Now you got elevators, you know, going up and down fetching packets, right? And it's a lot less friction, a lot less energy, lot lower cost, a lot lower latency, right? All those good things. So that is I think the future of inference, at least for those who want to focus on speed. Anyone focusing on fast inferencing, that is the future direction. And so given that at d-Matrix we really always optimize for low latency, we optimize for energy efficiency, that direction seemed like the right direction for us, right? So that's the path we are down on now.

Yeah, that's right. Yeah.

That's right. That's right. Yeah.

So by the way, everything we do here at d-Matrix is a chiplet-based approach, right? So I think that's the other decision we made pretty early in the company's life, which is we don't want to build bigger chips, right? We want to build smaller chips and then fabric them up into a collection of these smaller chiplets that will allow us to, you know, modularize the inferencing platform. So if you have smaller workloads, you can scale it down. You can package fewer chiplets. If you have very large workloads, you can obviously, you need more compute. You have the option to package more of these chiplets together and scale up the amount of compute you need. And so that was a very deliberate decision. We wanted that elasticity in the platform, right? And so we can, and also we can take it into many different markets, right? You can take a single chiplet application into a laptop and you can take it into, you know, like maybe a four-chiplet platform can go into a car, but if you're doing like a data center then you can scale it up to 16 chiplets and you can build a board with 16 chiplets and so you can go across the spectrum, right? So that was also the thought that, you know, today the company's focused on data centers but down the road we could be focused on other markets, right? So how do we build a platform from day one that allows us to go into other markets? So that is something we do here. Now with the chiplet-based approach, I think the other advantage of the chiplet-based approach is obviously the development cost of a chiplet or a smaller chip is less than a larger chip, right? And so that is the other reason we went down that path. And what we do is we essentially put multiple chiplets into a package and because of this energy-efficient computing approach that we have, we can actually put a lot more compute in a given power envelope. So for example in our case, we have a PCIe card that has, you know, eight of these chiplets sitting on a PCIe card, right? And if you add up the area across those eight chiplets, that comes out to be about 3200 mm² of silicon, 3200, right? And reticle size today is 800, so the largest GPU today is an 800 mm² chip. This is 3200 mm² of silicon, almost four times the reticle size of the state-of-the-art silicon available today, right? And we have that available in a 600-watt PCIe card, right? If you go look on the market today, most PCIe card solutions, right, don't have that much silicon in that kind of power envelope, right? They probably have like, you know, anywhere between 500 to 800 mm² of silicon. We have four to five times that on a single card, right? And the reason we were able to do that is one is because of the chiplet approach, because of the energy-efficient computing approach, right? But that allows us to throw a lot of SRAM into a single card and we have about 2 GB of SRAM. So again, you know, the largest we have seen is about 200 to 300 megabytes on a single PCIe card. We are at 2 GB, right? So we are, you know, almost 10 times more than what's available in the market today. And then we take that and we essentially scale it up, right? So now that we have 2 GB on a card, we can put eight cards in a server and then we can put eight servers in a rack. So very soon you have like 8 times 8, call it 64 cards sitting in a rack and 64 times 2 GB per card is about 128 GB of SRAM sitting in a single rack and that's the densest SRAM solution on the market today, right? If you look at, you know, the only other folks who are doing SRAM-based solutions today are Cerebras and Groq. Groq is obviously part of Nvidia. Groq when they sold to Nvidia was at 10 GB, 10 to 12 GB of RAM in a rack and now Nvidia is helping them, you know, redo that whole platform and they'll get to 128 GB later this year, early next year. So essentially catch up with d-Matrix. Cerebras is roughly about 44 in a single wafer-scale engine and they can put up to two, I think, in a rack, right? So call it 88 GB of SRAM in a single rack. So we are 128 which is still the densest in the state-of-the-art and it was all possible because of this chiplet-based approach and we could sprinkle a lot more SRAM in a single rack, right? And so now we're not stopping there. That's Corsair. The next generation product, Raptor, will take the story further where we retain the SRAM-based computing tile but now we're adding a layer of DRAM on top of that computing tile, right? So now you suddenly add more capacity. You don't lose your memory bandwidth advantage because what happens is given that the DRAM is sitting directly on top of the compute, you are essentially exposing all the DRAM banks to the compute engines that are available below, right? So there is a direct connection between a DRAM bank and a computing engine right below and it is all microbumped. Essentially the bandwidth is extremely high because you have, you know, tens of thousands of these microbumps sitting between the DRAM and the compute tile and now suddenly you have like, you know, call it 100-plus terabytes per second of bandwidth on this computing platform which is very similar to where we are today with SRAM. So we really didn't lose our memory bandwidth advantage. We just added to the capacity of the computing platform, memory capacity of the computing platform.

Yeah. No, no, we are geniuses here, Sam. We saw it all.

No, nothing like that. Nothing like that. I think look, you know, I think we saw a trend. We saw a trend which was, you know, the models were getting bigger and smarter. That was it. Right. And...

Yeah. I mean,

That's right.

That's right.

That's exactly right. I mean, we're talking, this is the year 2020 right now, right? That I'm talking about. And you know, by then GPT-3 was just out, right? And GPT-3 175B was out. I don't think there was a model that big. I think PaLM had just come out which was I think also around the same size, 200B. And so we had gone from a world of, you know, when we started in 20, we started first ideating in 2017, 2018, the ResNet was all the rage, right? And that was like a 25 million parameter model which was all the rage. And then comes BERT in 2019 which is like a 300 million parameter model. And then suddenly we're looking at a 175 billion parameter model. I'm like, my goodness. And people had started playing with it and you know, the initial feedback at least on, you know, some of these tech forums and all that was, wow, you know, this model is exhibiting behaviors that we've never seen before. There's these autoregressive, you know, capabilities and generative capabilities where, you know, the model seems to be in certain rare cases, you ask, you know, the right question and it would respond back with stuff that most users felt was like, you know, my god, this was not even trained on this stuff, right? And so I think there was initial signs. But I think what we got excited by was BERT which was, you know, in 2020 was being deployed by Google in English language search. And Google was pretty open actually. When we were having our conversations with them, they were saying, look, you know, almost, we deployed this thing like six months ago and almost 10 to 20 percent of English language search is already running through BERT, right? People are actually using this thing. Nobody does keyword searches anymore. People are actually conversing with the search engine and BERT is actually pretty good at parsing those searches and all that. So we were getting these initial signals from Google, from, you know, what was happening at OpenAI that as these models are getting bigger, it's unleashing a new set of use cases and capabilities that we haven't seen before. And we were just looking around and saying, you know what, either we could go build a solution that is optimized for ResNet and we would be company number, you know, 101 essentially doing that, as 100 companies doing that and we can be the 101st company, and that's a me-too play, why would we even want to do that? Or we can just go all in on this new thing and just bet the company on, you know, transformer-like models and optimize the hardware and the software for transformers. And of course it was too early for us to do what some of the companies are doing today is like go all in and, you know, bake a transformer on the chip. So our platform is, you know, very programmable. We can go across workloads. But you know, we kind of essentially optimized some engines on the chip to be transformer friendly. We optimized a lot of the kernels and the software to be transformer friendly, right? But that was the focus. It's like, look, that's going to be the big thing and either we are successful or we are not, right? And we went down that path and you know, of course, you know, now here we are, transformers are everywhere, right? But you know, back to your question, I mean, really it was just that, it's like models are getting bigger, they're using more data, they're becoming more capable. And if they need more data and people are, you know, getting exposed to these new capabilities, they're going to use this a lot more. And as more humans start using it a lot more, the latency is going to matter. And so some of this was just common sense and what kind of architecture needs to come together here that would solve for latency, that would solve for memory capacity, memory bandwidth, and then, you know, it leads you to this answer around in-memory computing and, you know, kind of optimizing for that.

Yeah. No, I think look, you know, it's not one thing, it's a combination of a few things coming together, right? We started working on this DRAM-stacked approach back in 2022, 2023, right? So again, it was, you know, I know lots of people are talking about it today. I don't think it's truly really gone mainstream yet. You know, there's more and more people talking about it, but I can guarantee you, if you're doing this pod and, you know, call it two years from now, it is going to be absolutely mainstream, right? I mean, it's going to be like, I mean, how can anyone not even have thought that this would be the future of inference, right? I mean, so that's my prediction, that most inferencing solutions will kind of veer down that path, right, over time. Now, in...

Absolutely, yeah, yeah, absolutely. You know, the GPU is on the HBM treadmill, right? You can't, 100 percent of the supply chain, the cost going up, right? You can't really get off that thing. The programming model has been built for that, right? I mean, it's difficult to unwind and there's no reason to unwind because the GPU plus HBM solves for, you know, a class of problems on the Pareto curve that are very important and very critical and training needs it. And so I think that is done, right? I mean, there's going to be a roadmap for the next 10 years that will continue down that path. But then there are all these other points on the throughput versus interactivity curve where HBM really just cannot get to, right? And that's where you need these alternate technologies and you need these alternate approaches. And absolutely, you can't take, again, back to innovator's dilemma, right? You can't take what you already have and hope to completely, you know, change it out or gut it out, right? I mean, it doesn't work. So you got to go start from scratch somewhere else. And I think that is what Nvidia was trying to do with the Groq acquisition, essentially bring them in and have a, you know, totally different platform on which they can kind of build this high-interactivity, you know, memory-based computing solution, right? So I think that's kind of the longer-term plan there. Now in our case, you know, it all started with SRAM, right? Everybody's like, okay, you know, you need a fast memory and you need, you know, you need access to this memory, you know, memory bandwidth is the problem. So let's go solve for that. SRAM did it naturally because it's on-chip, but then it ran into capacity limitations. Plus SRAM is not really scaling as well from process node to process node, right? I mean, SRAM scaling stopped, you know, roughly call it, you know, 5 nanometers or whatever, right? It's not been scaling as well beyond that in terms of capacity also. So you're really not even as you go from one node to the next, you're essentially flat or worse, right? In some cases you might even be losing, you know, SRAM capacity.

I'm actually, I'm just talking raw capacity, right? Like this raw, the number of bits that you can pack on, say, a generation, you know, a generation of a product that is in 5 nanometers and then you go to 4 nanometers. The conventional thinking in semiconductors was as you go from one process node to the next, right? You can actually get more bits in the smaller process node, right? Because you're scaling down and you can pack more. But that's not happening with SRAM. With SRAM, it's actually staying flat. So as you go from say a 5 nanometer process node to a 4 nanometer...

process node, you're really not — you're not really getting a whole lot of benefit, or three nanometer, you're not getting a whole lot of benefit in terms of SRAM scaling. And your capacity, the number of bits you can pack on a single reticle-size chip, like a full reticle size, that's the maximum size chip you can build today, is capped. Now you can take a Cerebras-like approach, which is wafer-scale and all that, but even there, you will be capped. If you are building a chip of that same size and going from one node to the next, you probably won't get much in terms of capacity, right? So I think that is the problem with SRAM, is it's essentially not scaling, so you run into capacity constraints. If you have a big model that you want to run on the chip, the models are getting bigger, you need a memory technology that can keep growing in capacity to scale with those growing model sizes.

Right. So that is where we saw this back in '21, '22, because as I told you, GPT-3, you know, had grown to 175B. There were models that were coming out in '22 and '23 that were, you know, call it 400 billion, right? And we were already kind of getting to that, you know, part when we were like, wow, these models are getting bigger, sequence lengths were getting bigger, context lengths were called — you know, they called it sequence length back then — and you know, sequence lengths were getting, context windows were getting very big.

And we realized, you know, yes, again, same story, you extrapolate that out and we're like, wow, memory capacity is going to be a big thing, right? We have all — what we have solved for is memory bandwidth, we haven't solved for memory capacity. So what is our story on memory capacity, right? And so we started hunting again. So we looked at — the first thing we looked at was stacking SRAM on SRAM. We said, 'Why don't we just go 3D with SRAM?'

We were looking — yeah, we were looking to build a skyscraper. So let's just build a skyscraper with SRAM, you know, let's keep building more of these and just go vertical.

But then we realized that, again, same problem. You just can't get to the type of capacity you need to get to, and it's very costly. Stacking SRAM, memory in general is very costly.

So stacking SRAM on SRAM becomes very, very costly, fast. So we said, no, that may not work. And that's when we ran into this, you know, die-stacking approach. Interestingly, the crypto guys were the first who had gone down that path. And this was, you know, back in 2020, 2021. So when we ran into it, it was not like we were the first to see it or anything like that. We had seen other people do it. It's just that the crypto market, you know, went bust in '22, '23 and that technology never really quite got commercialized. So by the time we showed up in Taiwan in '22, '23, there was this thing sitting there that some of these crypto guys had played around with for a few years and never commercialized. And we said, you know what, we are going to take it and commercialize it, right? Because a lot of hard work had been done to build out a recipe, and we just said, this is perfect, this is exactly what inference needs, and all we have to do is commercialize it. So we went, we took that recipe, we worked with a lot of the suppliers in Taiwan, built up the supply chain, did a bunch of PC's over the last two, three years. We exercised that supply chain multiple times, and finally are ready to do a production tape-out on that Raptor chip this year, after three, three-plus years, almost three-plus years of working through that recipe. Right. So yeah, no.

Tape-outs? Yeah, there were a lot of prototypes. There were a lot of tape — in fact, many of them had exercised really the same chain that we exercise today. Yeah. So they had — it was just that those chips never went into production because they could never get to the point where they had to run big volume on it. But they had done a whole bunch of prototyping work on that technology and that approach. So, you know.

Bitcoin mining. Yeah. Yeah. Yeah. Exactly.

We didn't change the interconnect, so not for our platform. We retained the old interconnect. I mean, the only thing that had to change is because now you're stacking memory on top of — so your die-to-die, in our case it's a die-to-die interconnect, and we run it — now you've got to run it through a die interposer that is sitting under the chip, right? So you're going 3D, so the die-to-die interface is running 3D. Previously it was just planar, right? It didn't need it, it didn't have to run 3D. So now there's this additional thing that you have to kind of design for as you kind of route the die-to-die interface through, you know, an interposer. And we did some of that work on our current generation product, by the way. Right. We didn't entirely have to redo it on Raptor. We also have an interposer on our current product, which is strictly a capacitive interposer for, you know, things like — you run into things like ground bounce and power bounce when you bring up these AI chips, and so we have this capacitive interposer to kind of address that. So we do that today, so it's not like it was entirely new. But where the big changes will come at our end on the Raptor platform is the scale-up network, right? So what we have talked about so far is the die interconnect because of our chiplet approach, where we're connecting chiplets to each other. But how do we connect multiple cards and trays in a rack, right? And today we didn't talk about this on the Corsair generation, but we have a product called Jetstream.

Which is a, you know, a PCI card that sits alongside the Corsair card in the same server, and it's an Ethernet-based solution that allows for scale-out. And we call it pseudo scale-up because it is not a true host-initiated Ethernet transfer. This is like more device-initiated. So we essentially have been able to cut the latency down significantly compared to traditional, you know, RDMA over Ethernet, RoCE solutions, right, that are very common in the market today from, say, an Nvidia or a Broadcom, etc. We looked at those solutions, but they're very high latency. So we just built our own solution using FPGAs and we cut down the latency because of this new way of initiating packets over Ethernet, right? So that solution is available today, and that kind of pairs with the Corsair solution, right? And that's what gives us the ability to scale out the Corsair rack solution. So we can go across like 16 racks. You know, we're not limited to one rack, and we can, you know, kind of scale the solution out if you want to support bigger and bigger models. Right. Now that will change as we go to the Raptor generation. We are going to be looking at things like NVLink and we're going to be looking at things like UALink. These are the two most prevalent scale-up solutions. So there we are exploring, you know, which of these ecosystems do we want to become a part of, and how do we plug our Raptor solution into an NVLink ecosystem or an Ethernet-based ecosystem.

Yeah. So, I mean, first, you know, the first thing is obviously around, you know, how many XPUs do you want to pack in a given rack, right? I mean, the XPU density is the number one thing, and that directly drives the tokens per dollar per watt, right? So really that's the math we're looking for is, okay, if you have a given workload that you call it — you pick your target workload for a given rack, and you want to maximize XPU compute and XPU memory capacity in that rack, right? Keep in mind with the Raptor generation we'll be able to do two terabytes of that die-stacked memory in a single rack, right? So we went from 128 GB of SRAM in a single rack on this generation to 2 TB of stacked memory. So we really were able to solve — you know, if you get this right, we'll have solved the memory capacity problem in a given rack. But now comes the question, right? I mean, at the end of the day, it's about packing these XPUs and what kind of tokens per dollar do you get out of that, what kind of tokens per watt do you get out of that, what is the latency number, right? And that is where, you know, running the modeling exercise with an NVLink scale-up network versus a UALink scale-up network, you know, which switch are you talking to, what is the fabric interface, what is the connector — I mean, there's lots of complexities and details in how you build a compute tray for NVLink, how you build a compute tray for UALink, who will build it for you, when is it available — you know, so many decisions have to be kind of looked at before you decide which is the right option.

I think this is just, you know, volume economics at work, right? I mean, this is what it looks like, right, when you have a new market with infinite demand. I always say that the inferencing market has infinite demand, right, because there is no cap on the amount of intelligence that we as humans would like to get access to, right? So let's assume that the size of this market — there is no cap on it. And okay, if you are serving a market of that size, then what kind of — essentially prices should go to zero, right? If the market is infinite, then the prices should go to zero, right? That is the theory, right? So you're watching that in action. Essentially people are coming to the realization, wow, this market is of infinite size, and the volume economics are kicking in to drive the pricing, token pricing, down very rapidly. I mean, some of these charts on token pricing drops are just staggering, right? But then the consumption charts are equally staggering too, right? I mean, I was reading some numbers — what's like, we are at what, three quadrillion tokens per day or something like that? I mean, and 40, 50% of those are from the open-source models, right? And that has all happened in the last year, right? So if the demand curve looks like that, then what does the price erosion curve need to look like to support a demand curve like that, right? And so I think what you're seeing is essentially that — you're seeing that all play out in real time. Jensen didn't ask me to sign that letter, but had he asked me to, I would have signed it.

You know, I mean, I strongly believe that, you know, there's always going to be both proprietary models — they serve a purpose, right — and then the open-source models also serve a purpose. There's going to be lots of people who just don't have access to these proprietary models who want these open-source models that are distilling these, open-source, fine-tuning them, and essentially building on top of them. And I would say that is a vast majority of the US companies are actually doing that, right? I mean, the number of US companies that have access to their own pre-trained proprietary model is like what, you know, four or maybe three or four or five companies, right? So it's not like there's a whole lot. And that basically means that how do you guarantee access to the rest who want the ability to have their own model weights, fine-tuned model weights, and then kind of host it for their application, right? So I think this is great news for that. I think it's a must. I think it needs to happen. It's healthy. What does it do to, you know, token pricing at Anthropic and OpenAI, right? I mean, of course, it's going to put a lot of pressure on it, but it also means, you know, what you just said is intelligence per dollar, right? So that's going to force the OpenAIs and the Anthropics of the world to pack more intelligence into each token, right? So if you want to keep your token pricing flat, right, and you don't want to erode it very rapidly, then the only way to do it is to pack more intelligence into that token, right? And then you can keep charging the same amount for the token, but you're packing more intelligence per token. So yes, prices are dropping effectively — prices are dropping, but your intelligence is going up. And so you're kind of keeping your pricing flat, right? That is what is going to happen, right? Essentially the iteration on these — you know, the rate at which Anthropic and OpenAI will iterate on their models from here on out will be just a lot faster. There's going to be 5.6, 5.7, 5.8, 5.9. You know, you'll see these very quick iterations almost every two, three weeks, right? With each iteration, they're going to be packing more intelligence per token, right? And they can't wait six months before the next big frontier model comes out, right? So there is going to be these fast iterations. It's going to lead to a lot of innovation because you'll have a lot of these algorithmic techniques, training techniques, ML techniques to essentially release a model to kind of improve the velocity of, you know, model releases at these frontier labs. And with each of those approaches, you're going to be finding a way to pack more intelligence per token, right? And I think that's fundamentally what you're going to see at least at the big frontier labs. And the OSS models will not stop, right? I mean, they'll continue to kind of, you know, keep tapping into that capability and, you know, releasing open-source models that are not too far behind the frontier models at the frontier labs, right? All good news as far as I'm concerned. I think this is going to just explode the demand for inference. So for a company like d-Matrix, that is even more relevant, right? This is going to just drive token consumption even higher. People will have access to more intelligence. It'll lead to more applications, more wonderful things that we can do with all that available intelligence.