Hi everybody, welcome back to the New York Stock Exchange. This is the NYSE Wired plus theCUBE's AI Factories - Data Center of the Future, and we're super excited to have Sid Sheth here. He's the Founder, President, and CEO of d-Matrix. Sid, thanks for coming on. Good to see you again.
Yeah. No, thanks for having me, Dave. It's always wonderful to chat with you.
So last time we chatted was June, you were in our Palo Alto studio for our AI and robotics leaders programming. Not much has happened since June, huh?
I can barely keep up, Dave. I can barely keep up.
It's incredible. Jensen was in the exchange yesterday. Michael Dell was here. Of course, AI factories is a big theme. When we talked last, we talked about the shift from training to inference. Now, everybody is talking about it, and you said at the time training is a performance problem. Inference is really all about efficiency, performance per watt, cost. And so I wonder if you could sort of talk about how that's, first of all, come to the frontal lobe of most people's minds here, and how that shift is changing the design of data centers and edge. What does it mean for the next generation AI?
Yeah. Yeah. I think when we spoke in June, this was post the DeepSeek moment. And I think my point of view at the time was we have officially entered the age of inference. And I think all these announcements that you've seen in the last six months are further validation of the fact that the age of inference is truly here, and everyone is trying to play catch-up because so much time and money and resources has been spent, call it over the last 10 plus years, on training larger and larger AI models and making them more capable. And then you have this watershed moment in the early part of this year where you realize that we have reached a point in the capability of these models where you can now take them and make them really efficient. You can distill them down, make them smaller, and they're still as capable, and there is a lot more people who can use this, not just a few people. So I think you're just seeing a cascade of events after that DeepSeek movement in January of 2025 that has catalyzed the age of inference, and now it is all about deploying a lot of compute to serve the needs of humanity. And that is going to require so much compute that people are just trying to put their arms around, "Wow, how much inference and compute do I need to run to really serve all of humanity and serve all the applications I want to build?" By the way, the training still continues. It's not like training comes to an end. People are still wanting to train bigger models and get to AGI, so that quest continues. So at this point, you're looking at these mega, mega announcements, hundreds of billions of dollars of compute being deployed to serve both the training needs of the future and, of course, the exponential inferencing needs of the future.
We asked Michael Dell, did you see any signs of demand? I mean, are you sleeping with one eye open? Of course, he's seen bubbles burst before, and he said absolutely no end in sight for this demand. It just seems to be insatiable. So people, I'm sure, yourself, you're looking for those signs, but people talking about the circular reference and taking on a little bit more debt, but it really does feel like the mid-part of the 1990s, '96, '97, than it does '99, doesn't it?
Yeah, everyone is trying to put their arms around where we are in the cycle. And I think, again, I would warn against comparing to the past, right? Because typically, the way the internet bubble rolled out, and there were a certain unique set of events that happened leading to what you call the dot-com crash in the late '99, early 2000. It's not like we are seeing similar patterns of behavior, because obviously, the promise of the opportunity is just so large, everyone wants to get in on it, but I would caution against trying to draw too many parallels to what has happened in the past, because that was a different point in time. Capital needs were very different. Availability of capital was very different. Towards the end of the decade, we were in a rising interest rate environment, not a falling interest rate environment. So yeah, you could be right. I mean, it doesn't certainly seem like we are close to the end here. Inference is certainly just getting started. We are in the very early innings of inference compute, the deployment of AI, the part of AI where people are going to find a way to recoup their massive investments and actually build applications and introduce productivity. So I think that is still very, very early, so it does feel like we are not anywhere close to the end right now. And to me, I would agree. I think I don't see an end in sight yet, and lots more to do actually. Lots more to do as an industry, as the amount of work that needs to be done to get all of this rolled out, so I think there's a lot more to go.
I think that's good cautions. The past is not prologue, and you've got to be there. You've got to be there investing. This is a wave that you want to ride. So do you think that the AI infrastructure is going to bifurcate? I mean, we've talked about this that only a handful of companies are going to be able to go after the million GPU clusters, the mega clusters and chase AGI, but most of the market's going to be focused on your space, cost-efficient, your deployable models. DeepSeek, as you said, was a real watershed moment. How do you think that bifurcation, if you agree with that premise, is going to shape the data center landscape, the edge landscape? Are we going to see sort of training factories and inference factories? Do those two worlds come together in some way? What are your thoughts on that?
Yeah. That's a great question. You're going to see a lot more compute. I think that's the underlying theme here, because our thesis at d-Matrix was always that inferencing compute is going to be 100 times more, 10 to 100 times more than training compute. So whatever you've seen being built so far has been primarily been done for training compute. And what you're beginning to see now in 2025 is these mega announcements, a lot of them are around inferencing compute. So people are beginning to look forward and say, "Wow, with the availability of these reasoning models and with the availability of the capabilities of these models just improving exponentially, you can do a lot more agentic capabilities becoming more readily available." The amount of compute we're going to need is just 10 to 100 times more than what we did on training. So my god, what does an infrastructure build for that opportunity really look like? Do these two coexist? I think the two coexist. I think there's, again, the dynamics has not changed. There's four or five companies in the world who can really afford to go on the AGI journey, and that is where you need 100,000 GPU clusters going to 500,000 GPU clusters going to call it a million GPU clusters sometime in the future. This OpenAI announcement with NVIDIA and AMD is about getting to that point where you are deploying a 100,000 to 500,000 GPU clusters so that you can train the next big model, unleash a new set of capabilities. Because every time you... The hope is that every time you launch a new frontier model, you launch a new set of capabilities, new set of intelligence, new levels of reasoning, and that will happen with GPT-6 whenever that comes out. And that will then unleash a whole new set of capabilities, and smaller models will come out of that, and a lot more inferencing will happen because of that. So I think the two will coexist. You'll have very large factories for training, which will be owned by a few folks because it's so capital intensive, but there'll be many, many, many more inferencing factories. And inferencing is going to be done at many, many different points in the network. There's going to be mega factories, there's going to be edge factories, there's going to be small computing nodes, large computing nodes. The hyperscalers, the new clouds, the sovereigns, everybody needs inferencing compute. So I think that is just going to look a lot more spread out and a lot more distributed. The training factories are a lot more consolidated relatively speaking compared to inference.
Yeah, the deals that are going down with OpenAI are just mind-boggling. First of all, the NVIDIA investment. I mean, Jensen has said, "I wish I could have invested earlier." And then AMD turns around and there's no strings attached, is my understanding. OpenAI turns around and says, "Okay, we're going to now work with AMD." AMD basically given warrants a big chunk of its company actually. And so I feel like the NVIDIA deal was maybe a little bit better and they're in a stronger position, but still. And then you see Oracle, Oracle got hit the other day because there was a report in The Information saying that their gross margins are like 14% on this stuff. And I'm like, "Yeah, well, if you're basically reselling GPUs, that's what's going to happen." That's what happened with Intel. That's clearly what's happening with NVIDIA. Look at gross margins of a company like Dell. So I want to talk about your platform because you're really architecting this platform for efficiency, not just endless excess compute. I think it's Corsair is the name of your platform, correct?
