How d-Matrix is Disrupting AI Inference in 2025

We work with TSMC today. That is our primary fab partner, and a lot of the prototyping work we do is with TSMC. And you know, they have a lot of programs where you can try out your ideas on a smaller scale, where you can build smaller chips and test out your technology ideas, and we have done a lot of that. We actually, for every generation of product, we've typically done a few test chips before we productized it, and they've been very supportive, and it is very possible to do that. So that's kind of been our way of getting our products to market and testing out our technology before we actually productize it.

So, thanks for having me on the podcast. You know, I've been a systems and semiconductor guy for many, many, many years. Started my career at Intel in the mid '90s. Actually, I spent the first 10 years of my career being a chip designer. And I, you know, spent a lot of time doing different portions of the design work at Intel in the first five years that I was there. Worked on circuits, you know, bleeding-edge circuits for the Pentium processors, worked on, you know, system issues related to bringing up processors. After that, I went to a startup called Azaera which was focused on building interconnect chips for Ethernet applications. So that was the next five years of my career. So really the first 10 years of my career were spent in heavy R&D and really working on both processor chips and chips built for interconnects and networking.

After that, I want to say, you know, I transitioned. I was still at the startup and I was looking to transition into something that was more customer-facing and moved into a business role. Spent another five years working on different aspects of, you know, what it takes to take a product to market, right? Really focused on go-to-market and, you know, and I had an advantage because I was essentially taking the chips that I had helped build and working with customers to figure out how to deploy them and making them useful, so to speak, for those customers.

After which I transitioned to a bigger role at a company called Inphi. And Inphi was in the high-speed networking and high-speed interconnect space. They were looking to break into the data center market. They had a really good, you know, footprint in the telco market but were looking to break into the data center market. So I went there and incubated a business for them which was only focused on data centers and that was a huge success for the team. Spent about nine years building that business that eventually grew to two and a half billion in revenue from zero and was acquired by Marvell in 2021 in a $10 billion acquisition. I stepped out in, you know, 2019 to start d-Matrix because I was getting very excited by the inference computing opportunity.

And, you know, I know it sounds very obvious today, but it wasn't very obvious back then, right? A lot of the narrative was about training and training bigger models and doing inference with, you know, image classification, and image models were really, you know, all the rage back then. And we quickly realized that it didn't make much sense for us to go do something with training because that was a market that was well capitalized, you know, a lot of investment. Didn't make much sense for us to go after the inference market at the edge that was focused on images and image classification. So we kind of were looking for a new entry point and that is how we started focusing a lot on language and transformers. And I'll talk more about that as we talk about the d-Matrix journey, but that's been my journey.

I grew up in India in a family of entrepreneurs. Both my grandfathers were entrepreneurs back in the 1940s, left home to start their businesses. My father was an entrepreneur too, left his home, came to the US, worked here, went back and started a business. So, you know, entrepreneurship is probably in my DNA somewhere. It was a matter of time before I, you know, stepped out and did something of my own, and here I am taking my shot at this. We'll see where it goes.

Yeah. So I think, you know, I've lived through the process of building a semiconductor business before. As I was, you know, stating earlier in my introduction, you know, when I was at Inphi we essentially incubated a business to enter the data center market, you know, building interconnects for data centers. And what we saw was, and I'll come to your question in a bit, but this perspective is important because that answers the question that you just asked, which is in the semiconductor business, especially when you're building chips, you know, they're very capital intensive obviously and take a long time, you know, multiple generations of product to get to success. And the only way to really be successful is to really intercept some kind of a discontinuity in the market. There has to be some kind of transition that is underway and you need to intercept that transition at the right time.

So at Inphi, the transition that we intercepted, there's really two major transitions that happened at the same time, was that data centers were going optical. You know, most data centers used to be based on copper interconnect, but they were transitioning to optical interconnect because obviously the speeds were getting to a point, bandwidth was becoming very, very critical, and the higher the bandwidths, you know, the harder it gets to run signals over copper. So it was transitioning to optical. And the other thing that was also happening at the same time was the signaling standard, which used to be NRZ, which is the one-zero standard, was transitioning over to PAM-based signaling. It's a four-level signaling. So we were going from a two-level signaling standard to a four-level signaling standard, and so that required new electronics, new optics, and so you had this tech transition and a market transition happening at the same time.

And that is what we intercepted very successfully. And, you know, the incumbents back then were companies like Broadcom and Marvell, and they were giants, right? But Inphi was successfully able to break into the market and dislodge some of the incumbents and build a very successful business doing that, right? So same thing applies for where we are today. We see, you know, a major transition happening in the market again, obviously. There is the whole transition from general-purpose computing to accelerated computing, which is a mega, mega change that is happening across data centers. But within that accelerated computing, we are also seeing the market transitioning away from training to inference, and inference becoming a very relevant workload.

And that is also a tech transition because you need a fundamentally different architecture as you migrate away from training, which is predominantly a performance-oriented workload, to inference, which is an efficiency-oriented workload. So you can't really build a common platform that does both training and inference. You can, it's not like you can't, people have done it, but it's not going to be, it's like building a Swiss Army knife, right? You're trying to do too much with the same thing. Building a dedicated inferencing solution, given that inference is going to be a much larger portion of the opportunity for AI compute, which means we think a dedicated solution for inference is going to win, right? And that's what we are focused on at d-Matrix, and that's why we think we can find enough share in the market to build a very valuable company.

So we do all our work, you know, first of all, we are building a product that is pretty run-of-the-mill, you know, is built on standard semiconductor process technology. So we are not doing anything funky here with the process technology that requires us to go to boutique fabs and, you know, build partnerships with fabs that work on very exotic processes and process technology and whatnot, right? So standard CMOS logic process. So, you know, we can work with TSMC, we can work with Samsung, right? We work with TSMC today, that is our primary fab partner. And a lot of the prototyping work we do is with TSMC. And, you know, they have a lot of programs where you can try out your ideas on a smaller scale, and, you know, what they call the MPW program or a shuttle program, where you can build smaller chips and test out your technology ideas, and we have done a lot of that. We actually, for every generation of product, we've typically done a few test chips before we productized it. And they've been very supportive, and it is very possible to do that. So that's kind of been our way of getting our products to market and testing out our technology before we actually productize it.

Yeah. So, you know, first of all, the team at d-Matrix has, you know, successfully sold into the hyperscalers before, right? We've been through, you know, call it 10, 15 hyperscaler ramps with products, not at d-Matrix, but, you know, through our prior experience, right? So we know what it takes to ramp a product at a hyperscaler. We've been doing it for 20 years, right? When, you know, when Google first had a cloud offering, and not necessarily a cloud offering, but they were setting up infrastructure to power their own applications, they were, you know, setting up, you know, pretty enormous data center infrastructure back in 2007, 2008, we were building chips that would help them power that infrastructure, right? This was back, you know, many, many moons ago.

So my startup from, you know, the 2000s, when we were building chips to connect networking equipment, we were selling products into Google to help them power their infrastructure. So, and, you know, all the way from back then to the Inphi journey where we had our chips sitting in pretty much every hyperscaler on the planet, all four big hyperscalers here in the US, three in China. So we have a lot of experience when it comes to deploying product at hyperscalers. We know the anatomy of what a typical hyperscaler ramp looks like. We know what the hyperscalers are typically looking for and how they engage, right? So at d-Matrix, we are kind of modeling our business along similar lines. We are building product that would embed into the equipment and infrastructure that the hyperscalers are deploying. So we are not going to them saying, look, you need to rip and replace what you have to accommodate us, right? We are looking at their equipment and we try to understand what they deploy and we try to fit inside what they already have without them having to rip out stuff to accommodate us. So really what we are trying to do is a cloud augmentation play, right? So we can go and we build accelerator cards, and maybe I can show you the card while we're on this topic.

So yeah, this is our 600-watt Corsair accelerator card. It's a, you know, it's a PCIe card and, you know, it's got two chips here that you can see. Each chip has, you know, four chiplets. So there's a collection of eight chiplets sitting on a single card that are, you know, connected to each other in any configuration. And then if you turn this card around, there's a lot of DDR memory that is sitting at the back. This is a pretty dense card. It's like a 24-layer card. A lot of engineering that has gone into making this because we have a lot of, you know, interesting chiplet technology that goes onto this card that needs to be routed very efficiently. So that's, you know, the guts of the card, and then, you know, that gets packaged into this accelerator unit. You see two cards there put together. And then they are connected with a bridge on top which allows every chip on every card to talk to a chip on the other card directly, right? So from a software perspective, this is treated like a single unit from a software perspective. So just as Nvidia has NVLink, we have the ability to connect all these chips sitting across these two accelerators into a single config from a software perspective, right? So that allows us to unlock a lot of efficiencies when we are running inference. So coming back to your question, I mean, that's what we ship, that's a card we ship, that's a unit we ship, and that plugs into any AI server that is built by, you know, folks like Dell or HP or Supermicro or, you know, Lenovo or even if it's a custom server sitting at one of the hyperscalers, as long as it can house a PCIe card, we can plug into that socket and augment that server for AI inference.

So that is our go-to-market model where we go to the hyperscalers and we tell them, hey, you've got a fleet of servers that currently don't have our hardware in them. But what if you were to put our hardware into the fleet of servers? We could augment that entire fleet with a solution that will completely change the economics of doing generative AI inference. Now, wouldn't that be attractive to you? And they pretty much across the board love that, right? Because it doesn't require them to pull out and rip out stuff, but it allows us to integrate our solution into theirs, and then they can go augment that fleet with more capability.

Yeah, sure. So a few questions in that one question. So let me unpack some of that for the audience, right? So first of all, the chiplets themselves, right? So you see this is, you know, a chip here. You know, there's obviously a lid that if we kind of pull that lid off there, you'd see four chiplets inside a chip here and four here. So there's a total of eight chiplets sitting on this 600-watt PCIe card. If I remember correctly, the only other 600-watt PCIe card on the market is the Hopper H200. Blackwells are still not available in a PCIe form factor. Still, I'm sure they will make those, but they're currently not available. So the only PCIe accelerator from Nvidia that's the latest and greatest is the H200. Now, the H200 has about 800 mm² of silicon on a 600-watt card, right? We have across these two chips 3,200 mm² of silicon. So that's four times the amount of silicon that is sitting on this card at the same power envelope, in the same power envelope, 600 W, compared to the latest and greatest Nvidia offering. And the reason we are able to do that is because of the chiplet technology and the chiplet approach, and obviously a lot of design techniques that we have built into the chiplets that allow us to run the chiplets in a highly energy-efficient manner. So we have, you know, almost 4x the amount of silicon on the same 600-watt power envelope, right? And so what's in those chiplets, right? So obviously we've got a lot of compute.

But again, you know, the bottleneck is not compute, right? If you look at generative AI inference, the bottleneck is really memory bandwidth and then eventually memory capacity, and it's about finding the right balance between compute, memory bandwidth, and memory capacity. And that's what we have strived for on this solution, right? So we have obviously 10 petaflops of compute on that card that I just showed you, and 10 petaflops of, you know, 4-bit compute sitting on that one card, which is, you know, plenty for now. But the key thing is to optimize around memory bandwidth. Now, you talked about putting, you know, the weights in SRAM. We have a combination of our in-memory compute and SRAM on those chiplets. So there's a combination of, you know, call it across that one card, we have almost 2,000 in-memory computing cores, and we have a lot of SRAM also that is built into those chiplets. So we have almost 2 GB of what we call high-performance memory. So this is memory that is sitting on the chiplets directly but running at a very, very high bandwidth, you know, call it 150 terabytes per second of bandwidth. You compare that to HBM memory that is sitting at, you know, roughly about, you know, 4.8 terabytes or, you know, 5 terabytes. The Blackwells are closer to eight or 10, but the Hopper that I, you know, was comparing our product to, because that's the only other PCIe product, is sitting at about, you know, 5 terabytes, right? So there's almost like a 20x, 20 to 30x improvement on memory bandwidth. Now, that is where we kind of punch through the memory wall, right? The, you know, it's not about just having a lot of compute, but it is having a lot of compute and a lot of memory bandwidth to go along with it. So you can feed that compute very efficiently from memory, and the memory is, you know, is local.

Now there's about 2 GB of that which is, you know, typically not enough for a lot of the AI models. So we scale this up further. So we'll have, you know, eight PCIe cards sitting in a single server, and then we have eight servers sitting in a single rack. So we have a collection of 64 cards sitting in a single rack. So 64 times 2 GB is about 128 GB of that high-performance memory that is sitting in a single rack. With that 128 GB, you know, we can easily serve, comfortably serve 70 billion parameter models or 100 billion parameter models out of a single rack, which is for the most part plenty for anyone who, you know, most enterprises, most, you know, very high-end enthusiasts, many sovereign opportunities that we are going after, including some of the hyperscaler opportunities. 100 billion parameter model sizes are pretty attractive. They can cover a lot of ground with that type of model. Now there are frontier models that are much larger, and we can certainly run the frontier models across multiple racks or even out of the capacity memory that I showed you. There is, you know, DDR memory sitting at the back. You know, we can run trillions of, you know, trillion-parameter, multiple-trillion-parameter models running out of that capacity memory. They would run a lot slower, but you can run them out of that capacity memory. So we have the ability to do this, you know, we have this flexible memory architecture where we can run 100 billion parameter models in a single rack, or we can do more across multiple racks, but it's running at these extremely high speeds because of the memory bandwidth problem that we addressed.

And all this is possible while embedding our hardware in a customer's environment without them having to purchase the whole solution from us, right? We can go augment their infrastructure with our solutions and give them the benefits of running what we call, you know, ultra-low-latency batched inference, right? So extremely fast while allowing the user to batch multiple, or the customer to batch multiple users. So really that's what we bring to the table. It's a unique combination of this chiplet in-memory compute, you know, technology, highly energy-efficient design, you know, how we connect the chiplets to each other, the numeric, you know, we have 4-bit numeric and 8-bit numeric that are block floating point, which is now becoming a standard for inference, all of this goodness built into this one solution, one PCIe card form factor that can go augment our customers' fleets with, you know, and supercharge those fleets for generative AI inference, low-latency generative AI inference.

Yeah. I mean, it's both. I think we, you know, we are really quite agnostic, right? As I was stating earlier, our go-to-market is built on taking our solutions and augmenting our customer solutions, right? So we don't quite care whether the solutions are in the cloud or in the enterprise, but the journey for most AI deployments is starting in the cloud, right? So I think our journey is starting in the cloud today, whether it's sovereign clouds or, you know, what we call GPU clouds or neo-clouds, which is, you know, the smaller dedicated AI clouds, and then there is the hyperscaler clouds. So these are really the three focus areas for us today, and then over time this will migrate into on-prem enterprise clouds, and we'll migrate as the workloads migrate there. But right now, you know, the high-velocity deployments are happening in the clouds, public clouds, and then eventually they will go to the private clouds over time.

Yes. Yes. So typically, I think you need a person who really understands the application that they're trying to accelerate. And what is happening is there's plenty of data that suggests that applications which are highly interactive with the user tend to be more successful, meaning the users are more likely to stay with an application that really interacts with them, right? Anything that is more offline in the interaction, you know, doesn't tend to keep users around for too long, right? So anything that draws eyeballs is stuff that is highly interactive. So latency matters, right? Clearly. So, you know, you need to look at the application. I mean, there is obviously, you know, other constraints like cost and energy efficiency and whatnot, size of the model, how big is the model, and, you know, you got to kind of look at the whole picture to figure out, you know, what kind of underlying accelerator hardware could really allow for a better user experience and a, you know, better cost profile, right? So it's a combination of these things. So yes, the hyperscalers have the people, have the right expertise and the right, you know, folks with the right set of skills to make those calls. They can match the right hardware to the right application, right? And then they can go deploy, and they also have the ability to go deploy new accelerators in their fleet to enable that user experience.

So, you know, if a new accelerator solution can demonstrate that we can improve the user experience, which is latency, or we can improve cost, or we can improve things like energy efficiency by a factor of at least 2 to 4x, that is a big win typically for a customer, right? So if you have the latest and greatest Nvidia hardware and you can show a 2 to 4x improvement over that in that generation, that's a big win, that's a very big win, right? So again, for inference, I like to say it has been hard to beat Nvidia on training, and, you know, they've done a terrific job of, you know, anybody who was at GTC earlier this week and saw the keynote, I mean, they have done a terrific job of advancing their roadmap, and, you know, they have a roadmap that goes out, you know, four years and five years, and, you know, it's almost impossible to kind of play catch-up there, you know, with all the technology that they bring. But when it comes to inference, I think there is no one-size-fits-all. You can't really take a big rack of Blackwells and just throw it at every inferencing problem. It's not going to work. It's not like every single inference request looks the same. Inference requests may need to be latency-optimized, may need to be cost-optimized, may need to be throughput-optimized, right? So there is no one-size-fits-all. You can't say like, here is one piece of hardware that's going to serve all of inference, right? It's difficult to do that. So inferencing fleets are going to be heterogeneous. There's going to be GPUs, there's going to be CPUs, there's going to be accelerators. And that is what we are focused on. We are not saying that we're going to be the best at everything, but we are really good at low-latency batched inferencing. So anytime you need multiple users where each user needs a really interactive experience, that's where you need to think of d-Matrix.

Yes. Yes, we work. So by the way, we worked with Rescale, you know, we know them well. And, you know, but you're absolutely right. I think that kind of skill set, by the way, already exists at the hyperscalers. So, you know, they either partner or they have the skills inside the company to build that orchestration layer. In their case, you know, they are doing query profiling. They have a query coming from a user, they're trying to batch that query based on the profile of that query, right? Whether it's a query that is latency-optimized, or is a throughput-optimized query, or a cost-optimized query. So, you know, depending on the region it comes from, right, you may have to, you know, route the query to the right data center and so on and so forth, right? So I think there is just a lot of things you need to kind of take into consideration before you decide where you want to send that query. And so yes, that orchestration layer, I mean, Nvidia announced Dynamo at GTC this year, that is again an orchestration layer for inference queries, right? So yes, that is going to be a very important piece. Because again, going back to the fact that inference is not a one-size-fits-all. Not every query is the same. So you need this orchestration layer that is going to be able to efficiently route queries to the right hardware, right? And we are seeing a lot more of that, and that is going to be where inferencing hardware is more heterogeneous. It is not just all GPUs, right? And that is something that gradually, I think, will become clear to a lot of people, but it's pretty clear to us.