I was in the prior panel, we're discussing scale up, scale out, scale in. Why don't you introduce yourself and maybe a quick intro, and then we'll go into Q&A with what d-Matrix is doing.
Yeah, yep. So hey, by the way, wonderful to be here. I thought this is my first time coming to Rise and wow, the energy is something else. So at d-Matrix, first, I'm founder and CEO at d-Matrix. We are an inference chip company. We've been around since 2019. So we've been around for 7 years, 7 long years. Feels like a very long time, but in the chip business, it's a very capital intensive business. Building chips is typically a very heavy lift.
We have just announced our products going into production. What that means is our chips are now ready to be deployed. And when we started the company in 2019, our emphasis was on building solutions for inference back then. And I often like to joke, you know, I used to go to investor meetings and the first question I typically got was, what is inference and why do you need to do it? So not many people really even understood the importance of inference, but we started the company with that vision.
We've stayed the course over the last 7 years and here we are 7 years later building silicon. We build the subsystems that we co-package the silicon with. And we also build all the software to deploy that silicon. And I'll talk a little bit about how we take the product to market in a little bit.
Yeah, I mean, 2019 is 4 years basically before GenAI kind of hit the scene.
And so you already had a vision for inference computing then. Obviously the first early days was the training regime. And now inference is up front and center. Especially with all these agents and the agentic workloads, how has this, you know, we're entering, as Jensen would say, we're fully in the agentic era. How different is the inference computing needs versus the training computing needs? And so that's one. And then secondly, why do we need a purpose-built specific inference processor versus a general-purpose processor?
Yeah, that was a great question, Tony. So, first of all, inference is not a one-size-fits-all workload, right? If you really look at it, everyone needs to use inferencing. Our thesis, when we started the company, was around something pretty simple. And I like to give this analogy, so I'm going to try this out here too, which is training AI is very similar to training humans. We spent the first 20 years of our life going to school and college.
But then we spent 40 plus years applying that knowledge. Doesn't mean that the learning stops, right? We keep learning new stuff. So there's continuous learning going on, but it's not like you're spending all your time learning. You're fine-tuning your knowledge. You're spending more time applying your skills and knowledge that you learned in the first 20 plus years, right? And you're monetizing a lot of that. So that's the inferencing piece. So, just intuitively, to us, inference was the much larger opportunity compared to training because everyone needs to use it. Everyone needs to find a way to monetize it. And so when we started, we said, you know what, we're going to build something that is purpose-built for inference, and it is different from what you need for training.
Because training is so much about performance. And just quickly training bigger and bigger models with more data, you want to scale that out very quickly. Big clusters, lots of compute. Inference, on the other hand, was more about efficiency, right? How quickly can you deploy it? How much can you do with less? So how much compute can you do in the least amount of time? How much compute can you do with the least amount of dollars? How much compute can you do with the least amount of energy spent? So it's really about efficiency being the big metric. So you need a different solution. You can't take a solution that was built for performance and then repurpose it and hope that it'll do extremely well on inference, right? Because everybody uses inference differently. So that is really what became the thesis around how we started.
Now, of course, we have entered the agentic age and people want to use inference for throughput. They want to use inference where they have maybe constraints on energy and they want throughput per watt. They want to use, in some cases, they just want very fast inferencing. So milliseconds per token becomes very, very important, right? Cost is always an issue, right? Dollars per token is always important. So how do you optimize around all these different metrics, right? And in some cases, you excel on a few, depending on the application, and optimize the application for that. In certain other instances, you might do something completely different. So you need different types of solutions to come together to make this successful.
It appears that we've seen an explosion of companies that want to build custom silicon, custom purpose inferencing compute. And it also appears that it is around arbitrating different memory architectures to the compute. So you've got SRAM approaches, you have HBM approaches, you have stacked DRAM approaches, you even have now emerging flash and high-bandwidth flash prospectively. And then you also mentioned what are you optimizing for? Cost, energy, speed. Like when you put all of that into kind of a giant bucket, and when you kind of align all the different companies, where is d-Matrix's focus in all of that?
Yeah, so I think that's a question we get very often, right? Where do we fit in? So maybe let me focus the audience on what has become Jensen's favorite chart that he shows in every keynote, which is throughput per kilowatt or throughput on the Y axis versus interactivity on the X axis, right? And what basically the message of the chart is pretty simple. On the Y axis, you have throughput, which is basically the amount of work you need to do. On the X axis is essentially speed, how quickly you want to do that work, right? And the two of them are typically at odds with each other, right? Because when you're trying to do more work, it typically needs more time. That's very intuitive.
At the same time, if you want to do something really quickly, it just means that you're trading off the amount of work you can do. So you probably can do less amount of work if you want to do it really fast. So you have to pick one or the other, right? And most HBM-based solutions today fall on that Y axis. So GPUs today are an HBM-based solution. You have compute, you have memory. The memory is HBM. There is a certain amount of memory capacity, memory bandwidth that is available to the compute. And putting those two together has typically meant that you can optimize for compute, and you will be able to really do well on that Y axis. So GPUs do extremely well on that Y axis, right? When it comes to throughput, batching, using multiple users, it's very, very difficult to beat the GPU. So if you are an HBM-based accelerator, most likely you will run into something that the GPU can do quite well, right?
Now, let's go on the X axis, right? On the X axis, the speed, you fundamentally need a different type of memory hierarchy to address that piece of the equation, right? And that is something we did from the early days at d-Matrix. We said, you know what? We're going to focus on energy and speed. Inference is all going to be about how quickly people can use it. So how do we create a new memory hierarchy that would allow us to really run inference fast? And so we created this approach called in-memory compute, which is putting compute and memory together, and minimizing the amount of data that we have to move, which would essentially lead to not only improvements in how quickly you can do stuff, but also improvements in energy, because you're not moving data around as much, you're spending less energy, right? But that allows you to kind of go pretty deep on the X axis. You can essentially optimize your solution for very high levels of what is now called interactivity. How quickly the application is responding back, whether it's machine-to-machine communication, whether it is machine-to-human communication.
In an agentic world, as Tony was talking about, that we have now entered, machine-to-machine communication has become so important. And so latency matters a lot. So that is the reason you're seeing huge interest in approaches that use SRAM-based architectures. We are now going from SRAM-only architecture to an SRAM plus a custom DRAM architecture. So whereby we stack DRAM memory, not as a separate stack co-packaged with the compute, but sitting directly on top of the compute. So we can actually drop data down into the compute. We call it 3D stacking DRAM or 3D IMC in our case. And we can drop data down into the compute with extremely high levels of memory bandwidth. And so it's a fundamentally different architecture, fundamentally different way of putting compute and memory together that unlocks that capability that we talked about on the X axis to drive to extremely high levels of interactivity. And that is what d-Matrix is focused on and makes us unique.
Yeah, let's elaborate a little bit more on this. This is very nuanced. You know, on the Y axis is HBM and throughput, they're amazing. And you have focused on the X axis, SRAM, speed, efficiency. And now you're doing stacked DRAM. Okay, is stacked DRAM, is that prime time? When will that go? And then, but also because you're using DRAM and SRAM, how unique is that in the industry? You know, Nvidia bought Groq for SRAM. But this SRAM plus stacked DRAM, and then because you're adding DRAM, maybe not HBM DRAM, but stacked DRAM on SRAM, can you do the holy grail, get the X and the Y axis?
