SBT C Suite Spotlight - Sid Sheth, CEO, d Matrix - Full conversation

Yeah. Well, I'd have to go back a ways to get to that answer, but it all started back in my childhood. You know, I come from a family of entrepreneurs, and this goes back even, you know, three generations. Both my grandfathers were entrepreneurs, both on the maternal side and on the paternal side. And then my dad was the first engineer in the family and was an entrepreneur himself. So he actually broke new ground in the sense that he was the first in the family to come to the United States in the 1950s and did a graduate engineering degree back then, and then worked in the US for some time, went back to India and became an entrepreneur. So I essentially grew up in a household where I was surrounded by entrepreneurs, and my dad was an engineering entrepreneur, which was a big influence on my life. So growing up in India and going to school, engineering was just a natural progression for me. I never really considered anything else. I was just gearing up, watching him run his business, being an engineer. That was just a natural, you know, footsteps that I followed in. And when it came time to which specific area of engineering I wanted to specialize in, in the '90s, you know, the PC was out. I was playing around with PCs back in high school, and I was just fascinated by electronics and what it could do. And I just kind of went down the path with electrical engineering because that was the most fascinating at the time. And here I am, you know, 25 years later, 30 years later, I'm an engineering entrepreneur myself.

Yeah. So Sudeep and I were working together before we started d-Matrix at Inphi. Inphi was acquired by Marvell in 2021 in a $10 billion deal, very successful. And there I was running the data center business, focused on building interconnects within the data center. So connecting computing clusters, AI computing clusters within the data center. So we used to build essentially the entire platform. We built the DSP chips, we built the drivers, the amplifiers, even in some cases the silicon photonics front end, all of it. So we built a collection of chips which we called a platform, and that was wildly successful. Those products were used by pretty much all the big hyperscalers, both here in the US and in China. And that business eventually grew to two and a half to three billion in revenue from zero, which is when I incubated it in 2010, 2011. So I think we were just discussing what would come next after this, and we spent a lot of time speaking with customers. This was when we were still at Inphi. And the feedback we got was, compute is the place where there would be a lot of innovation that would need to happen, not just on the training side, which is where Nvidia was already pretty dominant, but on the inferencing side, where it was still a pretty wide open field and there was no clear winning solution for inference at scale to serve AI workloads that were only becoming bigger by the day. So that seemed like an area where we felt we could add value, because a lot of the techniques that we had learned building these extremely high-speed interconnect chips could be applied to how you build a very efficient inferencing chip. And we started looking at that for about, you know, we ideated the company almost for a year before we stepped out and incorporated the company. So we spent a year looking at what the solutions in the market were, where the market was going, what kind of technology would succeed in a market like that, and then we convinced ourselves that there was really an opening here for somebody to come in and do something different. That's when we stepped out and incorporated d-Matrix.

Yeah. So again, it goes back to first principles. We looked at pretty much every inferencing architecture that was out in the market at the time, and we did not see anything that really serviced the inferencing opportunity. And when you look at inference, it is really about efficiency. It is about doing more with less. And when I say more with less, meaning you want to be able to do a lot more compute with less dollars, a lot more compute in less time, right, and a lot more compute with less energy, right. So I think you are trying to build something that is from the ground up focused on efficiency. Performance is important, but it is about being sufficiently performant, but at some point it is about being efficient with what you already got. So how do you build a platform like that, and what kind of decisions does one need to make going back to first principles again to build something like that that would serve the market broadly? And so if you first started with a core, which is the compute engine, and if you look at inference, it's actually in some ways a pretty simple workload. All you're really doing is, you have inputs coming in, you are essentially grabbing data, typically the model parameters from memory, you do a bunch of compute, and it's very repetitive, very parallel, and results are then fed outside, right? And you're just doing the same compute very repetitively and in a parallel fashion. And we felt, you know, you're making so many trips to memory. Wouldn't there be a smarter way to collapse that first tier of memory into the compute? And if you could do that, then you would just save a lot of time and energy making those trips to memory to a traditional SRAM and fetching all of that data just so that you could do the compute when the input came in. So why not just co-locate the first tier of model parameters directly into the compute so that you save all that time and energy? And that really became the core idea around the digital in-memory computing engine. And we had to make it digital because there were other techniques that used analog compute. In our case, if you want to deploy this at scale in a hyperscaler environment or a big cloud environment, analog computing tends to be unpredictable, you do have to sacrifice accuracy. So we had to go down the digital route to make the compute very predictable and very accurate.

Yeah, first, there's companies coming up. I just heard the other day there's inference companies being formed now, you know, every week. So I think the trend will continue. But I think in general, first of all, there's an acknowledgement that the digital in-memory computing architecture is the right one for inference, specifically inference at scale. And if you want to run inference in a highly diverse environment where you just don't have one workload, you have a multitude of workloads that you want to run on this inference engine, now the one advantage we have is obviously time to market. We are the first to commercialize. I'll talk a little bit about the product that I placed right here, which has all that wonderful digital in-memory computing technology, but we are the first to market to have commercialized that technology. And we've been working on it for close to five years now, multiple test chips along the way to validate that technology, and now it is in production. So that's one big advantage for the d-Matrix team. The other is, I think, just the way we have built the core technology, it has been handcrafted. We've essentially gone into an SRAM bit cell and augmented that bit cell with more transistors to make it a multiplying bit cell, and then we have embedded that into an array of memory fabric technology which has adder trees, and all of this has been placed together by hand. So it has taken us multiple years to kind of create that. And then we have made it sufficiently portable, because we have built all of that with logic rules cells, or basically we just take cells from the logic library and we've hand-placed those cells, and that's how we are able to build that fabric. So you can take that technology today, there's nothing binding it to any particular fab or any particular process. We can take that and move it to a different process, different fab, without too much of a lift. So that's the beauty of that technology. It's sufficiently customized while being sufficiently portable, and that is the beauty of how we've built it. And that takes time, by the way, because there's always trade-offs you have to make when you make a decision like that, like how much do you really want to hand-place and customize, how portable do you want to make it, right? You always have to make trade-off decisions, and making those right trade-off decisions takes time, and then making it work and commercializing it takes multiple years.

Yeah. So this is our first product that we announced a few months ago, and this is now shipping to customers. This is the PCIe accelerator card, which is a 600-watt card, and what you see on there is two chips. Each chip, if you pop this lid here, would have four chiplets inside. So this is probably the industry's first chiplet-based computing accelerator. A lot of world's firsts on this piece of engineering, and every time I hold it up, it is with a great sense of pride of what the team has accomplished. So much engineering, so much marvelous work, it's an engineering feat. So world's first, as I stated, computing accelerator with chiplets, world's first with digital in-memory computing technology, world's first with very high-speed chiplet interconnects to connect these chiplets in an all-to-all configuration, world's first with block floating-point numerics. So we use inference-friendly block floating-point numerics, and the world's first with a truly open software stack, with MLIR and PyTorch and things like Triton DSL and whatnot. And then what we do is we take this card and we package it up into this accelerator unit, which is what we ship to customers. And what you see there is two cards in there. And this thing is pretty heavy, by the way. You could use it for a workout if you wanted to. But you see this bridge on top here. This bridge connects these two cards together. Again, the beauty of this is a bridge that allows any chiplet. So there's a collection of eight on one card, eight on the other, so a total of 16. Any chiplet on any card can talk to the other chiplet directly through this bridge. So from a software perspective, this unit is essentially treated like a single accelerator. Very similar to what Nvidia does with NVLink, where they are able to essentially virtualize multiple GPUs into a single GPU through NVLink, and we can do the same with this bridge. So it unlocks a lot of efficiencies. I'm going to hold it up here for the camera. I always love to do that. And it's a shipping product.