Welcome back over to theCUBE's live coverage here in the studio. I'm John Furrier with Dave Vellante. We've got a great lineup, three days of wall-to-wall coverage, Robotics and AI Leaders where we're talking about the chips, the software, of course, the energy, power consumption, the data layer. We had entrepreneurs, CEOs, investors in here. Sid Sheth is here. He is the CEO of d-Matrix, theCUBE alumni. Sid, great to see you. Great to see you last night at the event.
Yeah, it was wonderful to be there.
You guys are in the middle of the action. The pressure is on, the market's waiting for the Moore poor performance. Dave and I on our podcast called The Red Zone, like in football, the infrastructure players are close to punching it in and just it's going to be a release of more horsepower, just energy. So congratulations and great to see you again.
First question is where are we? What's new? What have you shipped?
Yeah, I think we spoke, was it summer of last year?
So it's been a year, and I think we spoke again at Supercompute and that was the show at which we announced our first product, a product called Corsair. And this is an accelerator card with inference silicon on it that we have built. It took us a few years to kind of build out that silicon. We have done a lot of groundbreaking world's first type of stuff in that silicon. We can talk about that. But the product is now shipping to customers in pilots with some very large customers. And our plan is that we'll be in what we call general availability or GA by later this year and start ramping the product in 2026.
Yeah, I remember the conversation at Supercomputing. We kind of teased it out. I think we even said a year ago, inference. We hit that up and then we hit it hard in Supercomputing. Supercomputing is coming up again fast. We're already getting... theCUBE's got a bigger booth, it's going to be bigger than ever. GTC at NVIDIA gets the market all hyped up. Everybody wants faster stuff. Inference in particular has gotten everyone's attention, and then the agent wave has kicked in, which is going to put more pressure on more tokens, more action. So with reasoning, and now you've got evaluations, you start to see agents getting some visibility into where that goes, more data coming in. So this is really a demand curve that's hitting, so you guys are on the right wave.
What's changed since Supercomputing on the inference side? Is there more awareness? What would you say would be the market factor or product-specific feature you're vectoring into on the inference side?
It's a great question. So I think we've been saying before 2025 that the world is bifurcating. I mean, for the last 10 years, a lot of the conversation has been all about training, training bigger models, and it has actually worked pretty well. The bigger the model gets, the more capable it becomes. So everyone was on that trajectory like, 'Hey, let's just train bigger and bigger models. That means buy more GPUs and build bigger clusters.' And somewhere along the way, I think it became pretty clear that this is not really an affordable trajectory for a lot of people. Yes, you can deploy all that CapEx, where is the return? Yes, maybe a few companies who are in the pursuit of AGI, that makes a lot of sense for them. But what about the rest? Not everyone is in the pursuit of AGI. A lot of folks want to figure out what they're going to do with all these AI models that are becoming available that have different degrees of capability.
So the world started moving in two directions. There was one set of guys, very few, probably less than five, who are down the path of getting to AGI. And those folks will build big clusters, buy a lot of GPUs, build big models, frontier models. But then there's other section, vast majority, which is really focused on deploying models that are cost-efficient, deploying models that are capable enough to get what they need to get done. And this is something we've been saying for a while, that there is going to be a very large segment of the market that will want that. That happened. The big thing that happened after Supercompute was DeepSeek. And that happened in January and that really kind of put that conversation to the forefront, which is you don't really need to spend a lot of money to build a capable model. You can take big models and distill them down to smaller models, and those models are perfectly capable to do a lot of different tasks. And I think that has kickstarted a whole movement around which people are like, 'I think we are going to use that class of models to build applications with and deploy them in vast majority of the enterprise applications that don't really need those massive frontier models.'
And also you mentioned, so couple things I want to ask you because, one, we know that the demand for tokens is forcing a reset on architecture around bigger clusters. I would put them in the category of mega clusters for the trainings with maybe big workloads, and then I would say large scale clusters for normal data centers. This is another trend we're seeing. So now you're starting to see the use cases. In some cases you don't need a two nanometer, you don't need a six nanometer IoT when hearing that here. But also, DeepSeek points out that the software innovation could deal with the constraints of the market. What are some of those constraints, and that's maybe a bad word, what's the enablement that you're enabling software people to work around? Because constraints that simply, it takes time to ramp up the next gen, so I got to work with what I got and then I'll take advantage of the next generation, but the software is where the action is.
And so with good software and good energy management practices, you can have a data center serving up massive inference.
So, I think first of all, we got to break the training problem and the inference problem, and even training, pre-training, as you said, mega clusters, large clusters, you need lots of compute, many different racks of compute to actually get training a task completed depending on how large the model it is that you're trying to train. When it comes to inference, everything that we are seeing, these clusters are nearly not as large. We are looking at one rack, two rack, maybe eight rack at the best. Most of the vast majority of the inference tasks are really run on much, much smaller clusters, which obviously scales down the software problem. The other thing is inherently inference from a software perspective is easier compared to say a pre-training tasks where you are exchanging a lot of information between various GPU nodes and keeping everybody in sync. There's a lots of orchestration that needs to happen, lots of networking that you need to keep working in sync to get that task done. When it comes to inference, it's really you don't do any of that. It's really shoving data through the compute and all you're doing is quickly making decisions. It's a throughput play and latency play, so you want to do all of that very quickly because users are trying to use this stuff. So the software problem is different for inference. It's nearly not as complex as it is for training.
Now, the things that we have done at d-Matrix to make it more attractive for users to do more with our hardware is it's all around efficiency, because inference is about efficiency. I like to say this, training is a performance problem, inference is an efficiency problem, and you really can't take a training solution and put it into an inferencing solution and hope that you'll get the same efficiency. You've got to build something from the grounds up to be really, really efficient. And that's what we have done is build something from the grounds up, and the software has also been built from the grounds up for inference. So things like using different kind of numerical formats that are very inference-friendly, we call those block floating point numerics, those have now become a standard. I mean, pretty much Microsoft went out and standardized on that called MX formats. The whole industry has formed a consortium behind it. Pretty much every silicon that is trying to do inference uses that, but that allows you to extract more efficiency from the workloads and just do inference more efficiently.
Can you explain the architecture a bit more? Because we think of monolithic chips, big-ass RAMs, and then more efficient chiplet architectures. I think you're the latter, although you do use large SRAMs, and my understanding is blocks of SRAMs. Can you explain the architecture and the difference from sort of what we consider a monolith? I can think of NVIDIA as big chips. Explain the difference.
Yeah. So, I mean, the way we like to put it is there is the von Neumann world, the von Neumann architecture. Traditional von Neumann architecture is you have compute, you have memory, and you spend a lot of time between going from compute to memory, and data has to transfer, you spend time and energy. What we did at d-Matrix, the core innovation, was we said how would we just co-locate the compute and memory? So vast majority of our silicon is these in-memory computing engines that are essentially memory where you can keep the model parameters, but the computation is happening in memory. So you are saving all that energy and time going back and forth to that first tier of memory. You don't need to do that anymore, and that gives a lot of efficiency when it comes to energy and latency. So I think that's one of the core innovations at d-Matrix.
And the Corsair platform, what are the salient aspects of it?
The salient aspects of Corsair are first of all, it's a true generative AI inference platform built for that application. This is not something that was built for something else and then we said, 'Oh, inference is the new hot market. Let's go pivot this thing into that application.' No, the company was started to build a solution for inference in 2019, and Corsair was a product that has been in the making for almost three to four years with an emphasis on generative AI inference. So things like building it with chiplets as opposed to building it with very large chips. And the reason we did that was we said, 'Inference is not a one-size-fits-all problem. People want to do inference on big workloads, small workloads, big data centers, small data centers. You can't really build one solution and hope that's going to work everywhere.' Build chiplets, and you can scale it up and scale it down using things like in-memory compute, which is now becoming very popular for inference, but that is an inference-dedicated compute engine using things like in-memory compute. Using things like we touched upon, block floating point numerics or numerics for inference, all this stuff is built into Corsair. Connecting these chiplets in all-to-all configuration, putting that all on very inexpensive packaging, building a 600-watt PCI card that has dense collection of chiplets that has a lot of compute and memory. So it's one of those things, it's a recipe, I like to call it a recipe where it has a number of these ingredients that have been carefully put together to bake a really good cake, and you really got to build this from the ground up. You can't just think of this as an afterthought.
It's got to be near the compute too, the compute-hungry aspect of inference is another factor.
How are people deploying? How do you see the use cases unfolding? Can you share the thoughts around what this will look like in an environment for a customer? How are they looking at deploying this? Are they changing their rack configurations? Obviously, the card. What's some of the specifications?
So the beauty of the d-Matrix solution is that we do not expect our customers to change anything. We go in with accelerator card and the way we build our accelerator cards is it can plug into a server that is built by, there's a number of AI server folks out there, whether it's the Super Micros, Dells, HPs, Lenovos. There's a whole bunch of these folks who are building AI servers, so we don't want to reinvent that part of the wheel. We are saying, 'Okay, that's been invented. Let's augment, let's find a way to augment that infrastructure that's already out there with our solution so that you can do generative AI inference and do it in a way where you completely change the cost economics of doing it, you completely change the user experience that you can give your users.' That is a big part of the value prop of what d-Matrix does. We never wanted to go in there with a full solution and expect our customers to say, 'Oh my God, I have to rip and replace the stuff I have to accommodate you.' No, that's not what we want to do. We want to go in there and find a way to live in our customer's environment and augment it with what we have built. And that is something that obviously broadens the footprint of how many customers we can serve because there's lots of folks with existing infrastructure that they don't want to throw away and are building new infrastructure that looks a lot like the old infrastructure, or at least is being upgraded. But we fit into any of those. We plug into greenfield infrastructure, we can plug into brownfield infrastructure, but that really opens up the market for us.
