Welcome back everyone to theCUBE's live coverage, day three, where the show is really cranking. We still got the adrenaline pumping. I'm John Furrier, host of theCUBE, with Savannah Peterson, Dave Vellante, Kristen Nicole Martin. We've all been here on the ground getting all the data, a lot of action. Obviously, it's an AI show, high-performance computing. It's really evolved, and you can see the clear lines of sight into how the cloud's going to play into it, how the data center's going to erupt in huge opportunities as the large supercomputing is coming to the masses. Sid is back on theCUBE. He was in our studio for our AI NYSE Wired community event. He's back, founder and CEO of d-Matrix. Sid, great to see you.
Yeah, I'm back. Pleasure to be back, thank you for having me.
First of all, I remember the conversation we had in Palo Alto at our event, AI Leaders, you were on. You were talking, setting the table for how the data center's going to change significantly, making AI inference affordable, making the scale, changing the architecture. You got a lot of funding in 2023. Now, you launched here at SuperComputing. Okay, let's get right to the hard news first, and we'll get into what we talked about in the underlying scale, but what's the news? The d-Matrix, what's the company do and what's the news?
Well, we are super excited at SuperCompute to announce the world's most efficient AI computing accelerator for inference. We built this product with inference and inference only in mind. When we started the company back in 2019, we essentially looked at the landscape of AI compute out there and made a bet that inference computing would be the largest computing opportunity of our lifetime. For obvious reasons, I mean you can train models only so many times and then you got to deploy them, help businesses operationalize with the help of AI, and that all requires inference. The leap of faith at the time, of course, was this was 2019. It was not very clear when we would have the big watershed moment in AI, and that happened in '22 with ChatGPT. And then as soon as we went past that moment, everyone the world over was talking about what does it really take to deploy really fast inference and cost-effective inference? And d-Matrix as a company is really built for generative AI inference being done very quickly, very fast. We talk about three key things that we focus on. One is, it's all about doing more with less. Doing more inference in less time, doing more inference with less dollars, doing more inference with less power, less energy. The more with less paradigm is really what efficiency is all about, and we have built a computing platform that is efficient on all fronts, whether it is energy efficiency, cost efficiency, or speed efficiency.
Talk about the product, what you guys are selling. Is it the data center? What is the product? How do people use it? How is it deployed and consumed? Price? Give the specifics of the actual product.
Yes, so it's a computing platform, it's an acceleration platform. It's a PCI card that we have built. Today, we have started with the PCI form factor. We can go to other form factors like OAM, but we build silicon at d-Matrix. We build the entire computing silicon at d-Matrix. We've built the acceleration cards to package the silicon together, and we built a software stack that goes along with it to essentially map AI workloads onto the silicon. We sell the whole unit along with the software, and then we work with partners in the server ecosystem that would essentially put these accelerator cards into their servers, and then we can go to data center operators who build out the racks and the cooling. But the big difference in the way we go to market is the fact that we take a very collaborative approach with the ecosystem. We are not going into the data center saying, 'Hey, we've got a whole rack,' or we are coming in with services or anything of that sort. We build the cards and then we let the customers decide which server vendor they want to work with, how they want to build their racks, what kind of cooling they want there or liquid. And that's the flexibility that is so important when it comes to building out an inference computing cluster, because inference again is all about efficiency and cost.
And choice, exactly. It's a lot about choice, and we offer all of that by working collaboratively with the ecosystem. We essentially give customers plenty of choices.
Okay, explain. First off, I get the cards, the PCI cards, the hardware you put into the devices. So inference, I'm going to say open and choice become a big theme. I know you guys are looking at plug and play and making sure you can work with everybody. How does that work? Take me through the nuances. So let's just say I'm a customer. Or well, first of all, are your customers the end user? Are your customers the OEM manufacturers? Split that out for me, who's the buyer of this?
We work through the chain. We don't necessarily work directly with our direct touchpoints because I think inference is still early, so it's very important to understand the end use cases and making sure that whatever we've built really solves a pain point at the end customer. We actually prefer to work with the end customers, understanding their pain points, what are they trying to really accelerate when they want to use inference? And not all inference is built the same.
You're selling directly to the end user customer.
We are directly selling to the end user, and then we work backwards from the end use case.
And then work with the right server partner or the right data center, cloud hosting provider, et cetera.
Just a classic, we've got a great platform, you buy it, you put it in your environment, and then of course there might be other opportunities for you to work and partner. OEM and other thing. Okay, got it. All right, so let's think through. If I'm a customer and I want to have a Dell, I want to have other vendors in there, it's just on the network? Or how does that connect and how do you manage that?
We already are working with multiple system integrators, whether it's take the standard OEM folks or the ODM folks. We have many options already on the system integration side. We announced three at the show: Liquid, we announced GigaIO, we announced Supermicro as three of our partners, and we have others which we will be announcing as we move forward here. But there'll be plenty of system integration partners. And again, we are flexible. The way we have built the PCI card, we do not need any special server configuration for that card to plug into. It is something that is already available as a server configure, pretty standard server config from pretty much a lot of the system integrators, so options are already there. Really, it comes down to what is the end user application that the customer is trying to solve for, and how we can make them better at that.
When we were in Palo Alto, when we had our digital twin event with the NYSE Wired community in theCUBE, you talked about transforming AI workloads. Specifically about unifying the architecture around acceleration of inference. Explain that again, and tie that to the news and how the product works. Okay, you got my attention. Inference is the killer app, but there's different inference use cases and scenarios. I might need different kind of inference for this. And there's a lot of data involved, so whoever's downstream or upstream on the data. That's good, I can pack this. But take me through the unification and then how did the product work? Now I'm going to deploy it.
Yeah. The stuff I was touching upon when we talked about this in the past was around a unified model architecture, which was a transformer architecture, we talked about that. That emerged in 2017 and by, I call it 2020, 2021 leading to ChatGPT, the transformer architecture essentially had proliferated into multitude of different workloads. It showed that it is a very scalable architecture. It's a multi-modal architecture so it works with different modalities, video search, text, et cetera, images. The underlying engine of generative AI inference today runs on transformer-based models. We have built a solution that is highly optimized for transformers. In that sense, we are taking advantage of a common architecture and building an inference computing platform that would accelerate all of those models. That gives us broad reach with our solution. Now, coming to the use cases, what we announced at the show was an accelerator card that is really, really good with what we call low latency batch throughput. What I mean by that is we hear a lot of people talking about, 'Oh, my latency is really great, but it only works with a single user.' Or we hear about GPUs, talking a lot about batching, but they never talk about latency. What we have announced is a product that can do batching if it has to. It can certainly do extremely low latency with single users if it has to, if that's the use case. But really what we think the use cases are veering to are the low latency batched use cases, which is, let me give you an example. A good example would be something like video generation. Somebody wants to interact with video and create a video in real time. Right now, if you see the way video generation happens, you prompt a model, it takes minutes for the model to generate a video, typically generates 10 seconds of video. You may not like it, you come back, you re-prompt the model, you go away. That way of dealing with video is just not working. Where we want to go to is a highly interactive model where you are dealing and interacting with video in real time like you would interact with a chatbot or ChatGPT for instance, where it's interactive. Where you prompt the model, you get a response, you don't like the response, you prompt it back and it responds back in real time. We are not there with video, and the reason for that is memory, bandwidth, memory compute, memory capacity is a real problem. GPUs today don't solve that problem. What we have done is we have essentially created a class of memory that is tightly coupled to compute that allows for these kind of interactive video applications to get created. You typically will never have a single user working with these applications. You have a batch of collection of users, so multi-user low latency. You want to create a use case where you can have multiple users all interacting with a model simultaneously, but not having to sacrifice the latency for each user.
So you're an accelerating situation, not offload or.
We are accelerating. We would be accelerating the video inference use case.
Got it, all right. And what are some of the other use cases? Okay, let me ask the question differently. What problem do I have if I'm the customer that I, what signs in my environment tell me that I need to talk to you and get your product? Is it the latency, is it the batch piece? Or is it just I got certain configurations? When do I know to call d-Matrix up? For saving the day. Basically, because you've become, you're an accelerant.
So if you're dealing with any kind of use case that you need, first of all, number one priority for you would be if you're dealing with a use case where you have multiple users where they're all interacting with a model, but the interactivity with the model is not where you need it to be. Right?
