d-Matrix CEO Sid Sheth Shares His Blueprint for Chip Success | Seed to Silicon

We incubated a business from scratch. That was another nine-year journey. So that's how it all got started. I started my career at Intel.

I would say companies that are funded at scale — this is not for the faint-hearted. It's a very capital-intensive problem to go solve, so you really need investors with deep pockets, and how that can continue to scale generation over generation, not just a one-generation thing. Nvidia and Jensen personally know each of these guys really well, right? So they go and deliver hardware to each of them long before any of the other companies even get access to hardware.

No, thanks. First of all, thank you everyone for coming. I know it's Monday, you know, is always a tough day, first day of the week, and for you to brave Bangalore traffic — I just braved through, you know, one and a half hours of Bangalore traffic and got here, and I know what it is like. So thank you. I really appreciate all of you for being here. Yeah, so you know, I'd like to say first of all, I'm a chip guy, being a chip and systems guy pretty much my entire career. You know, there was a time in my career where I was early on, when I was doing my master's at Purdue, I was trying to figure out should I go down the software track or should I go down, you know, do more core electrical engineering. And I just had that passion for building something that, you know, you could feel and touch, and that is what kind of led me down the path of wanting to build, you know, chips, right? And so that's how it all got started. I started my career at Intel building chips, and this was Intel in the late '90s. It was a very different company back then, and not like the Intel that you might read about in the news these days. They really dominated compute and learned a lot about what it takes to build and a methodology. They had a very, very strict methodology around how to kind of go about building silicon and building chips, and that was great training for me. But you know, as Sudipto said, I come from a Guju bania family, and entrepreneurship was somewhere in my DNA because my dad was an entrepreneur, both my paternal and maternal grandfathers were both entrepreneurs. So I knew at some point, you know, working in a big company was not going to be for me. And so right after Intel, I stayed there for about three to four years, and then I was ready to kind of leap out and do something of my own. And my first startup was a company called Aelos, where we were building 10-gig connectivity chips. This is, you know, 2000, circa 2001, and you know, this was literally right after the bust, and it wasn't very clear that bandwidth, which was so hyped up during the internet bubble, suddenly there was no demand for any kind of bandwidth, right? But you know, we started this company to go build the next step up in connectivity chips, and we did that for about nine years. Eventually that group got acquired by NetLogic and then eventually by Broadcom. And then I went to Inphi. This was in 2011, and it's interesting. Inphi was a company that had been around for a while, and they were just getting ready to go public, but they had not really been able to break out, and they wanted to break out into the data center market. And you know, I had just come off this journey with Aelos, you know, NetLogic, Broadcom. We had penetrated, you know, big data center players. So you know, the CEO of Inphi just said, 'Hey, I just want you to come and do exactly what you did there for the last 10 years, just one more time, and this time for me.' Right? And that's how the Inphi journey got started. And it was really — we incubated a business from scratch, and that was another nine-year journey, very successful. We broke into pretty much every single hyperscale customer on the planet. We worked with Google, Meta, AWS, everyone was using our chips by the end of that journey. That business today is a $2 billion business at Marvell, immensely successful. And I like to brag a little bit here, you know, about this is about the d-Matrix team, because a lot of the core members of that group formed d-Matrix. And I quiz people and say, like, is there a group of individuals that have built a $2 billion business in the semiconductor industry in the last 20 years? There's only one group, and that is the group at d-Matrix, right? So it's a great group of people that founded d-Matrix, and they know what it takes to build a chip, they know what it takes to deploy a chip, they know what it takes to architect a solution that is very practical, they know how to build something that can go to market within a certain market window, and then I get to do the easy job, is just to monetize all of that, you know.

And we get that question a lot, right? So I like to say, you know, hardware is hard, right? We all know that. Within that, building chips in hardware is even harder, right? And then if you are building a company that goes and builds chips, it's insanely hard, right? You have to be really stupid to go and do something like that. And you know, I guess we were amongst the few stupid people who wanted to leave a pretty comfortable job. I mean, we had a very, you know, comfortable job. We could have run that business. That business was doing about $300 million in revenue by the time we were thinking of doing this, and then it ramped very quickly to, you know, close to a billion in revenue, right? So who would want to kind of leave all of that? And a lot of people told us, 'You guys are crazy. I mean, you spent eight years building this to where it is, and now you want to step out and start all over again? And do you even understand the pain of starting a company? And are you sure you want to do this?' But you know, that was good. Those were the right questions to ask because that was a test, right? And you get tested when you're trying to start a company. You don't go and start a company for the sake of starting a company, right? Never do that. Be very sure about why and what you want to do, and spend a lot of time thinking about it. And we did that. We spent almost a year ideating d-Matrix before we kind of went out and said, okay, you know, we're going to incorporate and we're going to go on this journey, right? So that is something I would encourage entrepreneurs to do. But I think in our case, we were pretty clear that we wanted to focus on inference, which is not — you know, by 2019, training was, I think Nvidia was already the 800-pound gorilla in training by then, and there was kind of the first wave of chip companies that got founded in 2015, AI chip companies, that many of them went after the training problem because that was the big problem of the time. And you know, the insight later emerged that inference would really be the bigger problem, right? So it's one thing you train only so many times, but then you need to infer, you know, an infinite number of times, right? So a little bit of luck because, you know, we started in 2019, we had more insight, we were able to learn from the companies in front of us. But I think the real insight or bet we got right — a little bit of luck, I would say, a little bit of insight — was really focusing on transformer workloads in 2020. And the way that happened was we were looking for a product-market fit, and we were like, you know, we are, you know, company number 101 building AI chips, and most of them seem to be building chips doing computer vision acceleration because that was the hot thing to do at the time. And we don't want to be yet another company doing the same thing. What should we really be doing? Where is the technology-market fit? Where is the product-market fit? And so we kind of kept peeling the onion on that, and what it led us to was we said, okay, let's run a process of elimination. What do we not want to do? So we said, okay, we clearly don't want to do what 100 other companies are doing. That's obvious. That problem, you know, will be solved if it becomes a big problem. There was another area we looked at, which was recommendation engines and ranking and rating engines. Meta was using them a lot. You know, they were actually funding programs to go build chips in that space. And we looked at the market and we said, you know, there is really four companies on the planet who would really need this, and if you don't win one of them, you don't have a business. It's too risky. Not a place. And you know, the other thing was from a technology perspective, that problem could be run pretty well on CPUs. You don't really need an accelerator platform. I mean, Nvidia is doing it today. The reason they built the Grace Hopper platform or the Grace Blackwell platform, and they introduced Grace, is because they want to run recommendation engines on the Grace CPU, right? And they have a different memory hierarchy, totally different memory hierarchy for how they run recommendation engines with Grace. So it was more of a memory-bound problem that could be run on CPU. So we eliminated that. We said, not much value we can add there. And right around that time, we ran into data from Google and Microsoft, who were both telling us that, look, transformer workloads are coming. They were invented in 2017. Google had already introduced transformers into English language search, and 10% of English language search was already running through transformers in 2020. And they were saying the trajectory is up. They were also telling us they were going to introduce it into foreign language search the following year, and you know, things would just grow from there. That was Google data. Microsoft data was also telling us that, you know, there is pretty much — Microsoft is a text company, 70% of the data at Microsoft is in some form of text, and they were saying we're just using a lot of transformers to basically mine through all the text data. So we said, you know what, this is great. Seems like nobody is working on solving the transformer inference problem. Let's go bet on transformer inference. And that went on to become a really good bet for the company because that became the underlying engine of generative AI. And it surprised us. It surprised us as to how quickly that whole thing just catapulted into something that, you know, it has become today, where ChatGPT happened, and then of course, you know, inference was on everyone's mind, cost-effective inference was on everyone's mind, and d-Matrix emerged as one of the unique companies that is solving that class of problem at scale, and an emphasis on that.

Yeah, so yes, I think, you know, if you look at the market, it has — you know, there are not as many startups now. Looking back at 2019, because it was really a free-for-all at that time. There were so many different problems to solve in the AI computing space, and you know, companies were coming at it from very different angles. There was no clear winning architecture back then, so there were just so many approaches. That is consolidating a bit now. We can clearly see that, you know, folks who are really attacking the data center space, you know, that list is now down to about 10, call it, right? There are companies that are attacking the edge market, and that is probably down to another 10. So there is, I would say, companies that are funded at scale — and this is not for the faint-hearted. This is certainly not. It's a very capital-intensive problem to go solve, so you really need some, you know, you need investors with deep pockets. And companies that are now funded at scale are probably, you know, in the 10, right? There's probably like 10 or 20, call it, right? And I've done that count lately. It's probably a count we're able to count of 15 to 20 companies that are, you know, reasonably well-funded to kind of go to that next step. So let's kind of focus on the data center, because that's where d-Matrix is really focused on today. And you know, there are companies that came before us, circa 2015, 2016, and then there are companies that came after us, right? And you know, the pattern that we are seeing is the companies that came before us were more general-purpose, because they were trying — there was no clear market, there was no clear workload, so they were like, we don't know where this is going, so let's keep this general enough and we can go. We were kind of a tweener. We said, okay, there's enough data to suggest that we should bet on transformers, but there's not enough data to suggest that we should bet the whole farm on this thing. So let's make the chip programmable, let's make the platform programmable, but with a lot of efficiency around transformers, right? So many times people confuse our product as, oh, you can only do this. No, the product can actually do transformers, it can do convolution, it can do all kinds of workloads. We can even do recommendations. It's just that the software won't be optimized, the kernels won't be optimized for that, right? So we are kind of a tweener, right? But then I'm seeing companies that are coming after us, accelerators that are going all in, betting the farm on transformers. I'm like, printing the transformer architecture into my ASIC, you know, I'm just going to — that's my approach, right? And that can be a risky approach because now at this point you are kind of precluding the evolution of how AI models could evolve, right? And so we think our approach wins. You know, something that is generic enough but has got the right optimized level of focus on the largest portion of the market, in which case, you know, in this particular case, it's going to be generative AI type of workloads or transformer type of workloads, right? Now, you know, just to kind of add one more point, you said, okay, it seems like a lot of people are approaching it in a similar way. True, you know, digital in-memory computing is becoming very popular with, you know, other companies too. But I think that is not the core. That is one piece of the overall innovation. It's not just about the compute engine, but it is about how you put the whole thing together. I like to call it a recipe. The d-Matrix solution is a recipe of ingredients, and each ingredient has been very carefully picked. You know, whether it is the digital in-memory compute and how we implement it — it's one thing to say I do digital in-memory compute, but there is a lot of nuance within digital in-memory compute and how you implement and put it together, right? So there is that piece. There is what we call numerics, how we do mathematical representation of models for inference and how we bake it into the digital in-memory computing engine. How do we build chiplets? How do we build chiplet interconnects to connect these chiplets? How do we package this stuff so that the platform can keep scaling over time? How the software is built to take advantage of all of this stuff, and you know, when it's put together right — software, if it is targeted for chiplets, would have to be built differently, compilers would have to be built differently. So it's like a recipe, and no other company has this unique collection of ingredients in one place. They have one piece or two pieces, but not all five or all six in the same place. That's what makes us different. And then you look at the, you know, the performance data you get out of the collection of these ingredients, and that is what gives us the differentiation, because it has been very — it's very well balanced, whether it's compute, whether it's memory bandwidth, whether it's memory capacity, and how we put it and how we scale it. It's kind of a more balanced solution. It's not brute-forced to focus on any one or two things.

So Sudip wants to get to the root of — he's asking some tough questions now. These are great questions. So yes, I think there are three — when you look at, you know, what customers — and you know, we are spending a lot of time with customers these days — there's really three things that feature in their buy decisions, right? The three things are: one is, again, all in the context of generative AI inference, right? When I turn in a prompt to an AI model, how fast can you respond, right? So that is user experience, like can you turn around a response, how quickly can you turn around a response. Number two, how much does it cost to serve that prompt. And number three, how much power am I going to spend to solve that prompt, right? So really, if you kind of condense it down, it covers CapEx, it covers OpEx, and it covers user experience, which is, you know, really the three key ingredients, right? So a lot of the metrics you kind of recited were around those three things, right? It's like I'm very power efficient, I'm very quick, throughput is really good, latency is really great, right, and memory capacity, right? So the way we do it is — and I know it sounds like this is a bit of a cop-out, but this is actually the truth — is we have done a much better job at coupling compute and memory, right? And let that sink in a little bit, right? When I say that, it may not be very obvious, but you look at generative AI inference models, right? They require compute, they require memory. They require a lot of compute on the front end of the processing of the prompt, and they require a lot of memory bandwidth and memory capacity on the back end of the processing of the prompt, right? So the best solution is one that will optimize both compute and memory, and not only that, it will find a way to couple compute and memory. And this is not just about the digital in-memory compute, that's just one piece of it. It is also about how you package memory and compute together. And we have just done a much better job at putting memory and compute together, and not just on generation one, but on subsequent generations. The way we are able to bring other memory chiplets into our solution, package it, and provide both capacity and bandwidth, right? That is easier said than done, right? You have to dig into how that was all put together, how that recipe came together, and how that can continue to scale generation over generation, not just a one-generation thing. How can it continue to scale generation over generation, and how the software keeps up with that, right? So that is really the underlying nuance of why you see those numbers, right?

Yeah, so the energy efficiency in our case is a byproduct, right? It was certainly very important, but we came at it not from like, oh, we want to build the most energy-efficient chip and that's what's going to dictate the architecture. We came at it as what is a customer pain point and what would get them to buy. And again, I go back to those three things that customers really think about when they're making a buy decision. One is how quickly can you respond, what is the user experience like, what will allow me to keep the user on my model for longer — so that is user experience. Number two is how much is it going to cost me to serve that prompt, right? Because I'm trying to operationalize my business. And then is power. So power is, yes, it's one of the decision points, but it's not the decision point. So we came at it from, okay, let's optimize on the first two. And when you optimize on the first two, interestingly, you're trying to do more with less. You're trying to build a solution that can do a lot more, you know, a lot more inferencing in less time. You're trying to do something that is doing a lot more inferencing with less dollars. And when you're trying to think that way, when you try to build a solution with that mindset — I want to do more with less — you solve the energy efficiency problem. You're saying, you know what, the next thing is I want to do more inferencing with less power, right? And so the memory-compute integration that I talked about also leads to a very power-efficient design because you're essentially saying that the traditional architectures have compute, they have memory, and you're kind of trying to go back and forth, and you're spending a lot of energy doing that. We just created something that puts the two together, and that saves a lot of energy. But in the process, your latency went down. In the process, your throughput went up because you're doing a lot, you know, a lot less work while you're generating all that data.

So that's a great point, right? I think there's a couple of things. When it comes to access to resources, you're not going to beat Nvidia, right? That cannot be your grand plan, that I'm going to have more access to resources and that is my winning strategy. You have to attack — the angle of attack has to be something along the lines of I can do something that they cannot do, or if they try to do it, it is a reset for them. It is basically innovator's dilemma has to kick in, right? And hopefully most of you have heard of innovator's dilemma. It has repeated itself multiple times, you know, over and over again, and you have to attack them on something that forces innovator's dilemma, right? And that is, I think, what we have done. We have essentially created something that they can certainly do. Can they do it? 100%. You know, it would probably take them a couple of years to slap a solution like this together. Do they have the resources? 100%. Do they have the capital? 100%. But if they do it, it's going to require a reset to the software, it's going to require a reset to CUDA, it's going to require a reset — their very strength becomes their weakness, right? Because right now they're married to CUDA, and if that is exactly what prevents them from, you know, that's what allows innovator's dilemma to kick in. It prevents them from going off and doing something radically different, which would force a rewrite of the software, right? So that is what we did. We essentially created an architecture that is so different, that really goes back to first principles, that is so focused on inference, that, you know, if they did it, it would force a rewrite. It's a fundamentally different computing paradigm, right? And then on top of that, we say, hey, inference market — we are really selling to application developers. The reason CUDA is so strong is because on the training side they're selling to model developers. That model developer community is actually a very small community. It's a research — you know, these guys were researchers, eventually now they build models. They exist at a grand sum of five companies in the world today: you know, Microsoft, OpenAI, Google DeepMind, Meta, AWS, Anthropic, and I'm missing one, xAI, right? So these are the five companies that have, you know, I would say 95% of these model developers. Nvidia and Jensen personally know each of these guys really well, right? So they go and deliver hardware to each of them long before any of the other companies even get access to hardware, their latest and greatest hardware. These folks start training models a year before any of the other companies even have access to Nvidia hardware. By the time, you know, the other folks catch up on hardware, these folks have spent two years training up a model and launching it into the public domain, right? And that's pretty much how they keep the two-year advantage going, and it's almost impossible for anyone to catch up with that. Right now on the inference side, the dynamic is actually flipped because Nvidia actually becomes an enabler for us. Every time a model out — a new, you know, say GPT-4, GPT-5 goes out into the domain, that becomes an inferencing opportunity for us. Now they're like, you know what, we'll run it on GPUs initially, but you know, after a while, if it really, really breaks out, the cost economics of running this become very important for us. So we're going to either distill it down — so there is the GPT-4 mini that shows up. The, you know, dollars per token have to start falling. They might say, you know what, it doesn't really make sense to run it on a GPU long-term. And the target customers are not the model developers. The target customers now are the application developers who don't necessarily, you know, like to program with CUDA. Many of these folks like open — they want to, you know, work with things like PyTorch and, you know, program in Triton kernel programming. Very different set of guys in a much bigger community.