Why People Are Paying 10x More for AI -and What That Means for the Chip Market | Sid Sheth, d-Matrix

It's really an opportunity for everyone to go into something new. I think at d-Matrix we are being very proactive about the use of AI across the whole organization for various different use cases. We have an AI team in the company that is entirely focused on rolling out AI across the whole company. Right. So it is not an option. It is mandated.

The inferencing space specifically, right, is kind of bifurcating, right, because I think everyone looked at inference as one, you know, kind of singular entity, right, but I think there is lots of nuance when it comes to inference. It's not a one-size-fits-all. Something we've been saying for a long time. So, you know, depending on where you are doing the inferencing, what your target markets are, what applications you're going after, you can't really build a single chip that serves all of inferencing's needs, right? And so you've got to really kind of break it down, and not every company is playing in every pocket of the market. You can't, right? Now GPUs, you know, just because they are so general, the ecosystem is so broad, clearly GPUs can go into many different applications for inference, right? But they're not going to be very efficient because of that, right? I mean, so yes, they're very broad in their applicability, they are general purpose, but when it comes to certain breakaway applications where applications need a very specific metric fully optimized, then GPUs will not do well in that market. And I think so, I think the way to look at it is, okay, you have the inferencing market, it's going to be the largest part of AI compute, it's expected to be over a trillion-dollar market, you know, call it in the next five years. But it's, you know, then GPUs will play in that market clearly, but there's going to be portions or segments of that trillion dollars that will need, you know, lots of optimization, highly optimized silicon. And the one that is truly breaking out right now is the segment that is called low-latency compute. And low-latency compute, low-latency inference is specifically around applications that need high levels of interactivity. So, you know, a great example would be Claude Code. This happened, you know, a few months ago, and you have people who are not programmers who are now using Claude Code and are great programmers, right? I mean, they, you know, the joke is, right, the new programming language is English, right? And everyone can program, and you can be an ace programmer and know how to prompt the model and how to use it. So, but that has led to people wanting to interact, and the faster the model is when it responds back to the user, the more likely the user is to stay with the application, right? So I think interactivity has become very important, and companies are beginning to charge more for those high levels of interactivity. So you can always say, look, I don't want that level of interactivity, in which case, you know, I'll pay, call it, $2 for a million tokens. But if I want that high level of interactivity, I'll pay $20 for a million tokens, and people are willing to pay that $20. They want that interactivity. So I think given that this new tier of inferencing is emerging, this new tier of tokens is emerging, right? Where we call it a premium token economy. So we look at the entire token economy, which is inference. This is the premium token economy. Not many companies play in that segment because you need a specific compute architecture that is well suited for fast token generation, right? So it's about generating tokens very quickly, and that typically means you have an architecture that puts compute and memory together in a way where you have an order of magnitude more memory bandwidth available to your compute than a GPU would have. Right? Typical GPUs are based on HBM, and they, you know, have a certain cap on the amount of memory bandwidth. Solutions like d-Matrix, and you mentioned, you know, Groq and Cerebras, have architectures that allow, you know, compute to get access to very fast memory, and it's an order of magnitude faster than HBM. So that is the breakaway category right now, and d-Matrix falls into that breakaway category. And because that category has exploded on us literally in the last few months, we're just trying to keep up, and I think we need to scale up into that opportunity rapidly.

Manufacturing, just delivering on product, deploying product faster, supporting product faster, you know, making more of our product. All of these need to happen. Bringing on more people to help, you know, support more customers. All of this needs to happen simultaneously, and that is why we did this acquisition of a company called Giga IO, which allows us to scale into, you know, faster deployment of servers, faster deployment of racks. We actually had started our rack-scale journey last year, right? We announced the product, Squad Rack, in October of last year at OCP.

Yeah, sure. So rack scale is, you know, you know, you have, you have, if you go into a data center, you know, you have these aisles or rows of hardware, and the unit of compute deployment in that data center is typically a rack, right? And racks are then put together into what they call a cluster. Then the clusters are kind of put together into, you know, so, you know, into, you know, eventually multiple clusters will form a data center. They have, you know, sub, you can actually break it down into pods. I mean, different people use different terms, right? So there's racks and pods and clusters, and eventually you kind of scale up to the data center, right? So the unit, but the unit of deployment in a data center is a rack. And these days, I think just given how quickly AI computing is, you know, the demand is just shooting, you know, through the roof. So I think you need lots of compute, and the unit everyone is looking at for compute deployment is in terms of racks, right? So I think you need to be in their own racks, or even in data centers, right? I mean, like you go talk to a customer, any customer, hyperscale customer or a neocloud or a frontier lab, and they say, okay, how many racks of compute can you deploy, how quickly? That will be the first question they ask you, like, okay, d-Matrix, how many racks of compute can you deploy for me and how quickly? Because then they can do the math in their head, you know, one rack is about 100 kilowatts, you know, depending on, you know, everybody has different ratings.

Somebody has 50-kilowatt racks, somebody has 100-kilowatt racks, somebody has 150-kilowatt racks. So depending on the customer, they can make the math. They can run the math very quickly because they're like, yeah, you know what? Okay, so d-Matrix, you're telling me you can do like 100 racks in three months or five months or whatever the number is. 100 racks times, you know, 100 kilowatts. This is the amount of power and this is the amount of compute capacity I need, which is typically measured in terms of power. So many megawatts of compute or so many kilowatts of compute or whatever. Typically it's always megawatt. Now we're talking gigawatts. All right. So they're kind of working backwards from there. They're saying, okay, I need X gigawatts or X megawatts of compute. That translates into X number of racks. And as a unit of deployment has become a rack. And so I think rack scale is just, you know, is kind of a very prevalent unit in terms of, you know, how quickly you can deploy compute. And so that's, that's really why we...

Yeah. So I think two things, right? One is that model has not changed from our perspective, from the point of view of the hyperscaler. Keep in mind, even when we work with the hyperscalers or the big neoclouds, who also tend to buy trays and cards and will deploy their own racks, you still need to understand rack scale. As a company, we need to understand it because you're talking to people at the other end who understand rack scale. So you can't be like, look, we don't understand rack scale, but, you know, we leave it to you. No, they expect you to understand. We, at the end of the day, have to help them deploy our solutions into their racks. If anything goes wrong, we need to have rack-scale people at our end who can help them. So I think there's an expertise and a skill set gap that we are addressing with this in our company. So that's one piece. But then there is another type of customer that we're going after, which is more the sovereign customers. Longer term, it'll be the enterprise customers who really want racks. They want whole racks to be deployed. Even there, we are partnering with Supermicro today to deploy those racks, right? So it's not like we are going to build our own rack. We're going to have our partners build the racks, but somebody needs to spec out the rack, somebody needs to design the rack, and we would have to do that for our hardware, right? And then Supermicro can go build it to spec, but you still need to have the rack-scale engineers. Now, that whole process has been accelerated, is all I'm saying. It's like the deployment process has just got accelerated because of what we talked about earlier in the conversation, which is the low-latency computing opportunity has exploded on us, and we have a solution that is well suited. So that whole process of taking our solution from where it is today to deploying it in a data center, whether it's a sovereign data center or a hyperscale, it doesn't matter. We need to have the rack-scale people at our end, the system-scale people at our end, who can accelerate the process of deploying the product, and that is what the acquisition is about.

Not everyone, not everyone, because I would say depending on your business model, right? I think in our case, our business model was, and I'm assuming your question is, is every other chip company looking to acquire rack-scale expertise? The answer to that is yes. I would say pretty much every company is looking to acquire, but not every company is looking to build or partner the way we build or partner, right? I think, you know, a lot of the other companies that you mentioned earlier, they're building their own racks. I mean, in many cases, they sell the whole, they sell a rack. Their unit of sale is a rack. In our case, the unit of sale is still a card or a tray. We partner with, you know, folks like Supermicro who would then sell the rack, right? And we would revenue share with them, or there would be another business model, or in the case of a hyperscaler, we would not even worry about selling a rack because they build the rack, right? So...

Claude is, you know, kind of incorporating a lot of agentic capabilities, right? Pretty much every enterprise-class software tool is trying to incorporate agentic capabilities. But I think to me, two things, right? One is Claude Code and the significant improvement in capabilities there, and then the OpenClaw. So this is agentic communication, machine-to-machine communication, where machines are kind of, you know, you can spawn multiple agents across multiple virtual machines, and a task is, you know, broken down into multiple agents, and these agents are talking to each other, you know, essentially orchestrating all the different work or the workflows that you have running on your computer. Now, that is very latency-sensitive communication because, you know, machines are just waiting for other machines to finish tasks, right? And you don't want that to happen for too long. So latency begins to matter.

Yeah. Yeah. It could spawn an agent to write code. So it can spawn Claude Code to go write some code for you on the side. Right. So...

We could. I mean, you know, absolutely we could. We are not currently. The platform, the way we've built it, is a platform that is built with, you know, like a Lego block approach, right? We have these chiplets that are the Lego block, and then we can scale the solution up. We can, you know, put many chiplets together into a very large solution. So for the cloud market, in the data center market, we essentially take our chiplet approach and we package, you know, close to eight of them on a single card, and we have 64 cards in a rack. So, you know, we call it rack scale. We have 64 cards, each with eight chiplets, right? So that gives you an idea of how many chiplets we sprinkle through the rack, right? But you don't have to. I mean, if you went into an edge market application where you're just doing, say, for example, if it's a robot or it's a physical AI application or autonomous driving, right, where you need AI capabilities, you don't need that many chips. You might need, you know, like a few chips, and we actually do that because of the Lego block approach. We just scale it down, sprinkle fewer Lego blocks into a physical AI application or an autonomous application. And the software would not change. The software was essentially built to scale with more chiplets or less chiplets, right? So the platform has been built today to go into the edge market. However, the edge market is very fragmented. So everyone has a slightly different software stack, and, you know, each vertical, I mean, like so if you go to the physical AI vertical, the software stack looks different, autonomous vertical software stack looks different, industrial, the software stack looks different. So we just don't have the scale today as a company to go address all these markets while we're trying to address the data center market. So our goal is to, you know, stay focused on the data center market, then at the right time when the company has scale, we will branch out into other markets.

I wish I could sit here and say, you know, I saw it all, I envisioned it all. No, I think some of it we did, some of it was luck, obviously. And the luck is really about, you know, how quickly a lot of this stuff happened and how useful a lot of this stuff became so quickly, right? So I think that has certainly caught even us by surprise, right? I mean, how quickly this is maturing. And this is because AI is learning fast, right? So AI is learning fast, and AI is actually learning to help itself, right? I mean, so there's this iterative, recursive loop where as AI gets smart, it learns to build tools that it can use in more efficient ways and more capable ways, right? So I think you have this kind of very quick, you know, positive feedback loop that is, you know, the virtuous cycle that is underway, and that's why you're seeing these impacts, you know, the impact is just so rapid, right? But, you know, we had a broad idea, right? We said, okay, the bet was on inference first, right, when we started the company. And then we said, you know, inference is going to be the big application, but then, you know what, it's not just going to be about inference, it's going to be about, you know, interactive inference. And then it's not just about interactive inference, but it's about spawning machines that can take all of that interaction and do something with it, right? So I think we kind of had a broader vision of how this could evolve, and then, but, you know, it even surprised us and pleasantly surprised us at how it's all come together so quickly, right? And I think what we would be most excited by next is, I think we have seen agentic capabilities unleashed. We haven't yet truly seen organizational capabilities unleashed, right? So meaning, you have humans spawning agents and augmenting humans, but can you create teams of agents? And so the abstraction of the task will keep going higher and higher. Right now, the tasks that are getting abstracted are like, okay, I have, you know, I want to make, I want to extract this data from this database, prepare a PowerPoint presentation based on the data for a board meeting that I have in a few hours, call it. And that can, you know, the agents can go spawn up and do a task like that. Now imagine you start abstracting to a higher and higher level, right, where you say, wait, it's not just about, you know, mundane tasks that I want to get done on a day-to-day basis. Imagine if you could describe a task at an extremely high level, like, look, I have a customer that I want to win, right? And you go figure out who the decision makers are, what product, what my product needs to be changed to. And so this thing can cover the whole strategy, like what an entire sales team and a marketing team would essentially do collectively. These agents could go spawn off and create a whole strategy plan. So I think the tasks will get more and more abstracted as you go higher, and very soon, I think, you know, people talk about organizations being entirely run by agents, right? And so I think where are we and how far are we from that, we don't know yet, but I think we are going to levels of abstraction here that, you know, eventually AI could, you know, run large portions of most companies, right, quite reliably.

So we are, we just rolled, we have an AI team in the company that is entirely focused on rolling out AI across the whole company, right? We just launched Claude Code to the whole organization. We have everyone who has access. We are mandating that everyone uses AI, finds ways of using AI, and shares any new tricks or tips that they find with the rest of the organization, right? So it is not an option, it is mandated. This is across the hardware teams, the software team, the operations team, the CTO teams, the marketing teams, the corp comms teams, everyone. Everyone is going to be using AI, right? And, you know, not everybody really knows what they will do with it, but they have to start using it. And the moment you start using it, you know, the use cases will emerge, right? So, yeah, you know, I think at d-Matrix we are being very proactive about the use of AI across the whole organization for various different use cases.

Yeah. So OpenClaw, I mean, we don't use OpenClaw because it's an open-source framework, so there are obviously security concerns around that. There could be, I mean, we don't know what the agents are doing behind the scenes yet, right? There is a certain level of transparency that's missing. But we use, you know, Claude Code, so, you know, now Anthropic is incorporating all the same agentic capabilities that OpenClaw has into Claude Code. So we have Claude Code Enterprise, Claude Enterprise, right? So, yeah, absolutely, we have this agentic layer that comes along with the Claude tools that are available to the whole organization. So, yes, they can unleash agents across multiple different applications. And the good thing is there is a zero data retention policy with all the enterprise tools, so anything that the employees enter into Claude is not retained by Claude, right? It's only used for that session. And, yeah, so I think, yes, we are already got an agentic overlay framework on top of all the Claude tools that we're using.

Oh, yeah, absolutely. I mean, again, a lot of this has happened in the last three to four months, keep in mind. I mean, but even before that, I always used ChatGPT at the time.

Absolutely, and it's surprisingly... Now, of course, we use Claude, we use Claude Code. So Claude Code is, you know, phenomenal. So for both, you know, I'm doing, by the way, we did one acquisition, we are looking at more, right? And so now our M&A strategy, we actually use Claude as a sounding board for our M&A strategy, right? And it's amazing what it can do. It comes up, you know, beautiful reports on how the companies could be integrated. So this used to be entire teams, remember, you know, banking teams and advisory teams that used to come in and help companies integrate and come up with a whole strategy on how to put teams together. Guess what? Claude does it in 15 minutes, right? So, and it presents a beautiful report on how the companies can be integrated and put together. Right. No, you know, it's not perfect...

Because sometimes the data is, you know, not—it's stale. But I mean, once you have a data room from the other company, I mean, you can throw cloud at it and it'll help you create a beautiful plan on how to put what an integration plan looks like, right? Now, till this was not available—I mean, till about six months ago, I didn't—we didn't have access to this. So, you know, I used to use ChatGPT just to run strategy questions. Yeah, you know, we're thinking about a product like this—I mean, what do you think, right? Um, I would say sometimes, you know, ChatGPT tended to be very polite, right? I mean, it never disagreed with anything that I asked. I mean, so I think what you're seeing more of now is the models are just more capable and more well-reasoned and well thought out. Um, and um, they do tend to disagree. I mean, they will tell you stuff that you're missing. Um, uh, right. Of course, you know, the personality that each of these chatbots has is, is, uh, you know, I think they've been curated to be very, uh, likable and amenable. Um, but I think I've seen more debate in the conversation with these chatbots than in the past, right? And that's probably because I think as they acquired the step function improvement in reasoning capability, I think they were able to kind of reason through situations. Uh, you could say, you could argue that they became more confident.

Right. They—they have—the models have now understood that they have better capability. It's almost like a human being, right? You—once you realize that you have acquired a new capability, you become more confident. The same thing with these models, they tend to become more confident with disagreeing in a polite way, but highlighting, you know, things that you might have not seen, right? So, I see more of that in the conversation.

A little bit of that. Initially, I did. So I—I wasn't sure how much to use it. I mean, uh, now I'm feeling a lot more comfortable because it feels more like a human conversation.

I think six to eight months ago it felt more like an echo chamber conversation, right? Somebody out there just echoing my sentiments and maybe adding a little more color to, you know, what, you know, I was already saying and presenting it in a slightly different way. And so if you are looking where—where those tools were more useful for me personally was if I wanted to make a point or if I had an idea that I already wanted to convey, um, a presentation I wanted to do, like it kind of augmented the way I did it.

It didn't change. You know, it's augmented—like, look, you know, you can present this, you know, maybe talk about this this way, this is another way you can say the same thing. So it really was an augmentation tool. Now it has become more of a—it's a co-pilot. I mean, I don't know if co-pilot is the right word, but it's almost like, uh, it's like a, you know, another version of me that is willing to debate with me, right? It's not just an echo chamber, right?

Correct. Exactly.

Could be someday, you know. Yeah. Yeah. Absolutely.