Tech Podcast: D-Matrix CEO Sid Sheth on AI Test Time Scaling | AI With Sally

Thank you.

Yeah, so that's a great question. I think recently, at least, I want to say probably in the last year or so, you know, it's test-time compute, test-time scaling, inference-time compute — all the same thing.

And really the fundamental idea there is you want to be able to throw more compute at inferencing a specific query. And what is very attractive about test-time compute or inference-time compute is the fact that here you have a model that is trained to do reasoning and can reason through queries that it might have never seen before, right? So you are essentially trying to get to answers to questions that you've never seen before, based on what you've already been trained on, because you've learned the ability to reason through questions and the ability to reason through data. And that is obviously a super attractive area, because most of the AI models out there today are one-shot thinkers, right? They either know the answer to the question they've been asked, and that's purely a function of whether they've been trained on that or not. What if you could ask a question that is not a one-shot, you know, a single-shot or a System 1 kind of question — a System 2 thinking question, which is more about really more abstract things, and the model still understands the intent of the question, can reason through different possibilities and eventually arrive at the right answer? But that just means you need a lot of compute. You just need a lot more compute, and you want to do it quickly. You know, turnaround times matter, so latency matters. So suddenly things like having enough compute accessible, but also being able to do it with high levels of interactivity, become very, very important, and a super exciting direction about where AI is going with that.

Yes, that's exactly the idea. I mean, you typically start with a bigger model. What they do is, like, they'll have a frontier model that gets trained with those capabilities, and you essentially want to figure out the solution space of what's possible. And you say, okay, great, this is what's possible, but I don't really need all of that. And then you want to kind of scale down from there, depending on a certain class of problems or whatnot, right? So you focus on math problems, or you want to focus on legal, or some specific domain, and you can distill models down to specific areas of expertise, because not everybody needs access to everything all the time. So you figure out, okay, this seems like a vertical where people would love to use this in the context of solving a bunch of math problems or solving whatever other type of problems you want to solve. Distill it down to that solution space, and those models tend to be a lot smaller, a lot more efficient to run on hardware, and they are just as capable in that specific area of expertise, right? So I think that's certainly becoming a huge trend post-DeepSeek. I think DeepSeek was really when it was the first time people appreciated the importance of taking larger models, distilling them down, and saying, oh wow, this is actually just as capable for everything that I need. And so this is becoming a real, real trend moving forward, right? And when it comes to inference, it's super important because, you know, cost matters and latency matters, and smaller models tend to be more cost-efficient and latency-optimized.

Kind of, except that I think, you know, the models now can spawn multiple chains of thought, right? So they can go off and they can go off and actually, you know, go down a path and say, okay, reason down a path and across multiple paths and eventually get to an answer. And then, yes, there is some kind of an algorithm there that eventually will figure out which answer is the best answer and kind of present it to the user, which you may not like, in which case you can force it to reason again on a totally different vector and different chains of thought. Right?

And eventually converge to the right answer. But again, that goes back to really having the ability to turn this around in a very short amount of time, because these questions are getting more and more abstract. I mean, there was one example — somebody was saying, okay, if I asked you the question, what's the capital of Mozambique, you probably know the answer to that question, yes or no. If I asked you, you know, what does the fiscal policy of Mozambique need to be to do more trade with the EU — that's a very abstract question. Even if you present this to a bunch of humans, they'll probably have to go off for a week and come back with different responses that they may not all agree on. And then somebody with higher authority would probably have to look at all those different responses, pick one and say, okay, this is the path we're going to take. So in that sense, reasoning models are not very different. Probably they don't need a week to do all of that work, right? And you don't want them to take a week. Can this be done in a day, maybe a few hours, right? And maybe present a response, we don't like it, you try again, right? But yes, it is heuristic to a certain level, because if you don't like the responses, the human can force the model to go down a different path.

So in that sense, yes, but the model is actually reasoning through different possibilities and coming up with what it thinks could be the best response.

I think so. One of the reasons, at least the earlier models, presented that option to users was, I think, it's just kind of building trust, right? Like, first of all, why do I trust the answer that came back with, you know? So here, I'll show you all the steps that tell you how I reasoned through getting to that solution, right? And then you can decide for yourself whether you like the steps I took. If you don't, then you can force me down a different path and I'll show you something else. Now, I think post-DeepSeek, I don't know how much of that will be available, because what happened as a result of that is basically DeepSeek was able to take those steps and exactly understand how the OpenAI model was doing all its reasoning, and they were able to emulate that. And it became a bit of an issue for OpenAI in that, look, we lost our competitive advantage in the process. So moving forward, I've not seen too many of these models actually spell out their reasoning steps, at least the ones that have been available of late. But I think it's always good to have that option. And I think my sense is the open-source models will always have that.

The closed-source probably not so much, because if your business is entirely predicated on having the best model, then you don't want to put that out there. But if your entire business is on making the model available to as many people as you want, then you want to build trust and then you want to make that available. So I think that is where the bifurcation is most likely going to happen.

Right. And because there's a lot of learning in that, and you can then incorporate that into how you kind of train your model. So you're a competitor trying to build a model, and your business is predicated on being the best, then you could learn from that and go off and do similar things, right?

Well, I think it is — I think it should apply, I would think. So let's take a simple example of video, right? Video is one of those classic examples where you want to prompt the model and you want to generate video content, and your prompt will be pretty abstract, right? So right now, first of all, video generation in general is not real time, and eventually we think it is going to become real time. By the way, we'll talk about that in a little bit — the hardware that we are building is really targeted for real-time video gen. But strictly from a reasoning perspective, I would think, you know, if you have anything that has a highly abstract prompt that you want to break down and get to an outcome, reasoning is something you could use, or test-time compute is something you could use for that, because essentially you want to spawn multiple chains of thought and figure out different possibilities. It's a solution space — it's just expanding the solution space of what's possible, and you want to try different approaches. So test-time compute is actually great for even multimodal content generation of any sort. So I would expect it's probably more popular for multimodality, especially if you're trying to create a video and you want all the different options possible and you want a larger solution space. Great — I would expect that's going to be more popular for multimodal.

You know, that's a great question, right? I mean, I think, and again, I don't know the answer — I'm just postulating here. I think we've been on this vector for the longest time, that you want to train more, you want to train more, right? And now we're kind of getting to that point where there is another way, right, with test-time compute, where you say, look, instead of training more, why don't we get the model to think more?

Right. So we have this thing at d-Matrix, we say, you know, 'think more versus train more,' right?

And it's not entirely clear that the results are going to be exactly the same, right? So to be seen, but I would argue — my hunch would be — that that is where things will go. It will go towards, why don't we get more efficient in terms of how we get the models to think more. You pre-train once with a very large amount of data and it takes you forever, and it's getting harder. I mean, just look at what's happening with Llama Behemoth — you know, they're just eking out gains.

From these models, beyond a certain point, is getting incredibly hard, right? And maybe there's a better way. You just take a Maverick or a Scout-like model and you get them to reason more and think more, and they might surprise you, is all I'm saying. And I don't think there's enough data to say one way or the other, but my hunch would be that that's the direction things will go: getting models to think more and just do more test-time compute.

I think so. It'll go there very quickly, right? Because again, I think it's a function of human beings — they'll always find a way to use the compute that is available to them and write applications to totally tap into any amount of compute. So I think it's going to be strictly limited by cost.

Right, it's going to be cost-driven. So if the cost curve comes down faster than the amount of compute available, then people are just going to do 100x the amount of tokens, and whatever the number is. It could be a thousand x. I don't know, you can throw a number there. But I don't think it's going to be gated. I don't think it's going to be like, oh, I don't think we need anything more than this. So they'll find use cases that will need more than that, as long as the cost permits, right?

So I think that's what we are focused on: okay, what does it really take to build a world in which people go to that 'think more versus train more' world, where, look, I'm going to only pre-train so much. I'm just going to solve all my other problems by just getting this thing to think more. And oh yeah, as long as I have as many thinking tokens available as I can to solve any question that I have — well then, guess what, if it's cost, then what kind of solution, what kind of architecture do you need to build to drive the number of reflection tokens or thinking tokens up by a factor of 100?

And what is it fundamentally that is preventing us from getting there today? And that is what we are really focused on solving. And yes, it's compute, but actually you realize quickly it's not just compute — it's really memory bandwidth, right? And that's really the choke point and the bottleneck. And are there creative ways to solve that problem so that folks can have access to plenty of thinking tokens?

So exactly. So the first thing we did was — and this was a little bit of insight, a little bit of luck — when we started in 2020, there were no reasoning models, and there was no inference for that matter, and people were not really talking about inference as much either, right? Certainly no LLMs, certainly no generative AI. So the models we were looking at at the time were models like BERT and ResNet, and the only thing we knew was GPT-3 had just become available, right? And this was a 175-billion-parameter model, it was the largest model of its time. And given where the NVIDIA hardware was at the time, the GPU hardware was at the time, this was a very large model, right? And you couldn't fit it on one card. And it was becoming very clear that to run this model effectively, you needed access to a lot of memory, because of course the model is big and you have to shard it, and it's a generative model, so token generation was strictly a function of access to memory.

So we said, yeah, that's interesting. And I said, if the trend continues — okay, we are on GPT-3, then GPT-4, and the models keep getting bigger — then this is only going to get worse, right? So we should go solve for that problem, which is the memory bottleneck problem. And how do we take a GPT-3 model — we don't want to put it into a DDR type of memory because it's going to be really slow. So what class of memory do we need to go create to get this to fit in that class of memory? And then that would really kind of solve the choke point. And that's how we came up with in-memory compute. And we said we're just going to conflate — or basically we're just going to collapse that first tier of memory hierarchy into the computing.

Right. And so we don't have this tier — you know, tier-one cache — it's all collapsed into one computer architecture. So we just make fewer trips to memory. We still have a cache on-chip, but we make a lot fewer trips to memory, and that was the first basic idea we had, which is, okay, look, this is great, let's start here. But then SRAM has not continued to scale the way other things have, and models just got so big that we quickly ran out of SRAM space. So we then said, you know what, there's probably another way to do this. And then we are migrating over to kind of stacking DRAM memory on top of compute.

And now we kind of move away from that SRAM scaling problem. DRAM has a lot more capacity.

And the technology to stack multiple layers of DRAM on top of compute is already available.

And so now we can keep pace with the size of growing models. There's a couple of things that have happened. One is we found a way to increase capacity.

But then folks also found a way to distill models. So models have — so we are kind of meeting each other at the midpoint. And it's kind of nice because now we can keep scaling as the distilled models keep getting bigger.

We have technology that has memory capacity that is big enough to accommodate all of those distilled models. And the more important thing is we solved for the memory bandwidth problem, which is essentially once you put memory directly on top of compute, we are essentially dropping data from the memory on top directly onto the compute tile through, it is, you know, microbumps, right, TSVs.

Because we just have a lot more connections than a chip that is placed sideways. An HBM connection, even through a silicon interposer, will just not have that many connections.

As versus going this way — this is just physics, right? Simple to actually visualize, things in a planar domain versus 3D. So we have almost 10x — in the 2027 timeframe, we'll be 10x the bandwidth of HBM, right? So HBM5, which is not available yet, will be 13 to 14 terabytes per second, and we'll be north of 100 terabytes per second.

Right. So HBM will just not be able to catch up with something like this in our timeframe, right? So I think we'll always be multiple steps ahead. And I think you have to build this from the ground up, because you can't really stack DRAM on top of very power-hungry GPUs, because DRAM is very thermally sensitive.

So you've got to build a platform that will allow you to do that. And our approach with chiplets, where we build these chiplets that are at max, you know, 50 to 100 watts, right. And now stacking DRAM on top of that becomes a lot easier. And as opposed to stacking DRAM on top of a very large chip, we kind of break the compute into tiles and then we stack DRAM across those multiple tiles, and we can still fan out. We can go high and we can go broad.