366: Inside the Age of Inference: Sid Sheth, CEO and Co-Founder of d-Matrix, on Smaller Models, A...

I think what's keeping me awake then it would be really the labor and the talent. You know, we said people have to essentially retrain themselves and reinvent themselves, and that process takes time. And we are going to go through this period of, you know, intense shortages, right? And that is just not a d-Matrix issue. I would reckon it is an issue for pretty much anyone in the space.

Yeah. Hey Dan, so thank you for having me on the show. It's super, super fun to be here. And thanks for that introduction. You pretty much covered everything that I wanted to say. But let me just talk a little bit about what got me to where I am today. Right. You know, I come from a family of entrepreneurs in India. My grandfathers, both paternal and maternal, were entrepreneurs going back to the 1940s. Both of them pioneers in their own right. You know, kind of left home at a very early age to essentially learn the trade and eventually went on to become entrepreneurs, very successful entrepreneurs, started their own businesses. Same with my dad. He was a pioneer in his own right. Came to the United States. So he was the first in the family to come to the US back in 1958. So he stayed here for about 10 years, went back, and then became an entrepreneur, started his own business. So I think it runs in the family. This is something I probably always wanted to do. It was in my DNA, and I started my career at Intel, as you talked about. I was getting just very excited. This is the mid-'90s. I was getting very excited about what was happening in the semiconductor space, and you know, chip innovation was on the forefront of everyone's mind at the time. It still is. So I kind of jumped in and did my master's degree in electrical engineering, learned everything about what it takes to build chips, and then came to the valley, Silicon Valley, and joined Intel. And this was '96, '97. So we were kind of getting into the last three to four years of the previous platform shift, the internet platform shift. Things were very exciting. A lot of startups getting created. And within a few years of coming here, I was pretty clear that, you know, I wanted to try my luck at, you know, doing a company someday, right? I felt I was too young back then. You know, I would have loved to kind of jump in right back then, but I was too young and I needed the experience. Spent the next 20 years just gathering that experience, and then the opportunity came in 2019 when it was very clear that we were in the midst of a second platform shift, at least in my career, as we were transitioning to AI in a very big way. And this was, you know, when I talk about platform shifts in my career, I'm talking about computing platform shifts. So we had, you know, the massive shift to PCs that happened in the 1990s, and then obviously we went to cloud computing and AI in the 2010s and the 2020s. So I think this was an opportunity I could not pass up. I felt this would be a generational opportunity, one that would be the largest computing opportunity of my lifetime. And so I plunged in, and that is how d-Matrix was born. And we kind of decided to focus on inference compute, specifically the portion of AI where you want to deploy AI efficiently, and that's what d-Matrix is focused on. That's our mission, is to really make inference commercially viable and sustainable for everyone on the planet.

Well, it's much bigger for one, right? I think the reach and impact of this is on a different scale compared to what we saw in the 1990s. Obviously, the internet, you know, was massive too, right? I mean, you know, connecting every human being on the planet, getting access to every human, getting them online so that they have access to data that they would never have had access to otherwise, was a big shift. But now we are entering the age where intelligence is available to everyone, right? Productivity gains are available to everyone. So yes, you have information, but you know, what do you do with that information, right? It's all about using that information to make intelligent decisions, and that's what AI helps you do. And imagine if you had that kind of, you know, power available to every human being on the planet, where a lot of the stuff and a lot of the decisions that they make, you know, based on data that they see every day, is automated, you know, with the help of AI, right? I mean, that would unleash huge amounts of productivity everywhere. So I think this is really the next big step in productivity gains for humanity. In that sense, I think this is much, much bigger.

That's right.

So I think, you know, one thing that you hear a lot about, a lot of the narrative out there around AI is really around people still seem to have that sense of apprehension. Oh, what will AI do to me personally, and you know, how does that affect me? And somehow fear is always the predominant narrative, at least initially, in the, you know, adoption phase of these kind of technologies, right? But there's really nothing to fear, right? I mean, if you're willing to embrace it, it's actually a tool at the end of the day that can only enhance what you want to get done, right? So, you know, I think it kind of comes across as a zero-sum game between humans and machines, but it's not really a zero-sum game. It's a net additive, right? A big net additive, in my opinion. And we are not going into a world of fewer humans at work. We are really going into a world of humans with augmented work. And I think that is something human beings just need to embrace, that they created something that can only make them better, right? So if you fear it, you know, obviously, you will not embrace it. If you don't embrace it, you're more likely to get disrupted, right? If you go out and not fear it, embrace it, learn it, and use it, you're more likely to be augmented, and you're more likely to have a very productive career, right? So I think that is a fundamental mindset that people need to adopt, I feel. And it's big in that I think it's probably amongst the first, probably the only platform shift in our lifetime where you are now dealing with a tool that can essentially do things that many humans can do quite capably, right? It can think, it can think, and it can make decisions, right? And this is something that we haven't seen before in any of the big platform shifts over the many centuries before us, right? So this is, for the first time, you're beginning to see a creation, a human creation of technology where the technology can actually think and do things that humans can do, in many cases do better than what humans can do, right? So this is intensely empowering for those who want to take it and use it to let AI do things that it does better than them, and they can then augment themselves to do things that they were meant to do or can do. So it is an opportunity for growth for everyone. So I think this kind of shift is unprecedented. I don't think we have seen anything like this in the, you know, past many centuries. So I think in that sense, obviously, this is much, much bigger.

Yeah. So, a couple of things there, right? First of all, I think it's a very limiting mindset to think that we won't have fewer opportunities moving forward because of what AI is going to do, right? That's certainly, in my opinion, a very limiting mindset. I think the amount of opportunities are going to explode. So it really comes down to, are you as a person willing to reinvent yourself? Are you as a person willing to learn the new tools that are available? And it's not like you are in a unique situation, right? This applies to pretty much everyone. Even folks who are in the field who are dealing with tools every day have to reinvent themselves, also have a learning curve in front of them. So it is not like this is unique to some and not to the others, right? This is something that everyone has to embrace. I mean, I know, you know, I'll just give you the example of d-Matrix, right? I mean, we have very tech-savvy knowledge workers who work at d-Matrix. These are very capable, smart engineers who are working on, you know, writing software or designing chips every day. This is what they've been doing for 30 years plus in many cases, right? They are trying to reinvent themselves, right? I mean, so, you know, you have software engineers now learning how to use tools like Cursor and Replit and GitHub and whatnot to kind of get comfortable with a pair programming model where the tool can do, you know, 50% of the coding for them. It's not like they knew this stuff, right? I mean, they had to learn it. Same thing goes for, you know, chip design. We are slowly introducing tools that would improve productivity in terms of can we use AI to do portions of the chip design. That was very tactical, probably, you know, very repetitive, right, task-oriented. And again, that's a learning curve for the engineers who haven't used it before. So this is not like just because, you know, we are an AI company, we build AI chips, we use a lot of AI software, but again, the team is reinventing themselves. So, you know, folks who are on the outside or, you know, quote-unquote on the outside who feel that, wow, now, you know, this is a career-ending phenomenon for me, are not true, right? Because it's just a mindset. You just have to say, look, everyone's reinventing themselves. I got to reinvent myself, too. And the opportunities are going to be endless. It's just up to me to tap into which of those opportunities I want to take advantage of, right? It's a brave new world. In fact, I would argue that every time there are these changes, it's an opportunity to uplevel your career, right? Because, you know, you were in a dead-end career when things were stagnant and, you know, you really couldn't innovate because really you had no option. But now, guess what? It's the wild, wild west. Everything is wide open. You can go reinvent yourself, uplevel your skills, go and do things that you never thought you could do. So I think there's a phenomenal opportunity for anyone who wants to embrace it. d-Matrix is really at the forefront of enabling the infrastructure that's needed to empower all people and, you know, create all the jobs that we're talking about.

Yeah. So I think, you know, let's take a step back, right? Go back maybe 10 to 15 years. You know, how did this all get started, right? I mean, AI, people have been trying to use the concept of AI and how to use it has been around for, you know, multiple decades. People have, you know, tried it in the 1960s and, you know, back in the 1980s. So there's been pockets of extreme optimism followed by pockets of intense pessimism after that, right? So we've kind of had multiple nuclear winters till the moment in 2012. And, you know, most people consider that the big bang moment of modern deep learning. And what happened there was, you know, essentially there was a convergence of compute capability with the availability of GPUs. There was obviously, you know, the cloud platform shift was underway. Data curating platforms and data gathering platforms had matured to a point where, okay, now, you know, you had large, vast data lakes where you can take all of the data, you can curate it, you can throw it at the compute, and you could essentially create, you know, AI really needs a lot of data and a lot of compute, right? So the graph algorithms that AI is made up of have been, you know, around for decades, but we could not really make much use of them because we did not have enough data, we did not have enough compute. But I think 2012 was the first moment where you kind of had a convergence on all three. You had the graph algorithms which got sophisticated along with data and compute. And for the first time, especially when it came to image recognition tasks, AI demonstrated that it was actually, you know, it took a significant leap forward and within a few years was better than humans, right? So that became the big bang moment, and then that unleashed, you know, the current modern deep learning wave. And people said, wow, I can train a vision model now, let me train other types of models. And there was this snowball effect that kicked in where multiple research labs started training various different models. And what they quickly realized is, wow, you know, the more data I throw at the model, the bigger the model gets, the smarter it becomes, the more capable it becomes. And so we went on this journey for the last 10 years in what I call the training realm of AI. The last 10 years, all about training different types of AI and just unleashing different capabilities. And that's not very different from us human beings. You know, we spend the first 20 years of our life going to school and college, educating ourselves. We learn all kinds of stuff. In some cases, we do PhDs or double PhDs or, you know, what have you, right? But we educate ourselves, acquire a lot of knowledge. And that is really the phase that we were in over the last 10 years, is just training a lot of AI and making it very capable. But then you had the ChatGPT moment, you know, I like to think about that as, you know, the first graduation moment, right? And then we had the DeepSeek moment in the early part of this year, where the conclusion quickly and the narrative quickly changed to, wait, now I think we have enough, you know, model training done. We have lots of capabilities unleashed. It's time to start using this stuff. Not to mention, you know, lots of capex dollars had been already spent on getting to that point. So how do we now find a return on investment on all those capex dollars, right? So that is really what changed in 2025, is post-DeepSeek, what DeepSeek taught the world was you really don't need to be on this gravy train of bigger is better, right? Models don't have to keep getting bigger to get better. In fact, you can take models, big models, capable models, and distill them down into smaller models. And those smaller models retain most of the capabilities of the bigger model, right? And the retained capabilities of the smaller model are actually generally applicable to lots and lots of people who can do lots of stuff with it. Not everybody wants the generic capabilities of a very large model. Most people want the specialized capabilities, you know, of a smaller model, but the smaller models are actually fairly generally capable to begin with. So you can do a lot with them. So that unleashed the age of inference, because now you have these smaller, more efficient models that can run on multiple compute platforms, and you can essentially, it's much easier to write applications around these smaller AI models. And that has led to the current wave of investments, where people are beginning to see that in the next few years you will have tons of applications running on these smaller models, and we'll enter the agentic, truly we'll enter the agentic era, where you'll have machines talking to machines, you'll have agents essentially. You know, agents are really models right under the surface. They'll be smart enough to understand simple commands in English, break down tasks, reason, and go off and do pretty complex tasks on their own. So once you get there, I think then, you know, you can truly start unleashing productivity. But imagine making that available to all of humanity and the amount of compute you need to make it available to all of humanity, which is why you're seeing all these big announcements, you know, hundreds of billions of dollars of compute being deployed over the next, you know, call it five years to unleash that capability and make it available to all of humanity. And d-Matrix is part of that journey. We are part of the second part of the journey, which is we didn't focus on training, we focused only on inference, where we, you know, are now focused on working with various partners and customers to deploy an efficient inferencing solution for the world so that it becomes easily available, accessible, sustainable, viable for anyone who wants to use inference, and everyone will want to use inference on this planet.

Yeah. Yeah. No, absolutely it will continue because, you know, people just want, once you make this available to people, they find ways to use more and more of it, right? And they want, you know, the use cases just explored and you identify things and tasks that you want to do that I think we haven't even thought about today. So we are kind of just barely entering the phase where you're making this available to everyone and people are already finding use cases and things with AI that we hadn't thought of three years ago. So that is what is going to lead to that compute intensity because you're going to have everyone getting it, they're using this multiple times. I mean, if you just look at the token generation trends, right? I mean, token generation is really intelligence, you know, you're unleashing, you know, you're basically mining for intelligence from these models. We have grown that to what was the number I heard, 10 trillion tokens, just Google's numbers were 10 trillion tokens per month going to, you know, close to 500 trillion tokens, right? So like almost a 50x jump in the amount of intelligence mined in a span of one year, right? And that's just one company. So I think, and then what are people doing with all this intelligence? You know, some of it could be just raw search, right? You know, mining for data, but a lot of it is just people trying to get a lot of work done. Most, you know, as you said, you know, coding and reasoning, and reasoning is just barely getting started because post-DeepSeek, that was the first open-source reasoning model that became available to the world. And after that, you know, there has been like a snowball effect of multiple open reasoning models getting launched into the world. So over the last eight or nine months, we had like one reasoning model in all of 2024, and that was o1 from OpenAI, maybe two, o1 and o3. And then we had DeepSeek, and then we have had like 20 or 30 this year, right? So that was a watershed moment early part of this year. So now with the availability of these reasoning models, you are essentially changing the game in terms of the kind of tasks that you can get AI to do for you. You know, this is pretty complex, abstract tasks that AI can go break down for you and get it done, right? And we haven't even put the tool flows around those models yet, right? The models have just barely become available, and, you know, putting all the tool flows, the agentic tool flows around it. We are probably a year or two, maybe, away from really integrating all of this into a seamless workflow that users like you and me can, you know, use very easily just off of our laptops or off of our phones, right? And we haven't even gotten there yet. So I think when that happens, I think you're going to see just a massive increase in the amount of compute used, right? So I think that's what we're preparing for.

Yeah. So a lot of people have talked about that, but you know, I think history doesn't, you know, the precedent from history is very different, right? And people have, and now, you know, of lately everyone's talking about the Jevons paradox. The Jevons paradox, I don't know how many of your listeners know about this, but Jevons' paradox essentially says that every time there is a new technology innovation, it makes things more efficient. And the initial argument is, wow, if this, you know, new technology innovation has made things more efficient, there should be less of it, not more of it. But actually people find a way to use much, much more of the technology innovation, and over time, you are actually using a lot more of that than you originally thought. The usage of the technology far outpaces the efficiency that it brings, right? It's really the efficiency that drives usage, but the usage far outpaces the efficiency gains, right? And so that's what we'll see with AI. Obviously, AI is going to introduce a lot of efficiency gains, and assuming that the amount of usage stays the same, then yes, there's going to be less usage because efficiency will bring the amount of usage down. But no, what we will see instead is because of the efficiency gains, there'll be an order of magnitude, maybe multiple orders of magnitude of usage gains that will far outpace the efficiency gains, right? And so there'll be a lot more of AI usage. So we don't see a flattening. In fact, then, you know, we are already, as I just stated, you know, the numbers earlier on token generation just from one company, Google, it's like 50x increase in token generation or intelligence mining in one year. And that is, by the way, in that one year there was a lot of efficiency gains too. So in spite of those efficiency gains over that one year period, we saw a 50x increase in token, in token generation, right? So I think that is, you know, that is what we expect. We don't expect the curves to flatten. The demand is just going to intensify from here.