Hello and welcome. Generative AI is moving beyond the typical use cases, and increasingly companies want the economics of Gen AI to work out. There are several other issues that the application and its use cases need to address as well. How could they be done? Well, Sid, CEO, and Sudeep, CTO, and also the founders of d-Matrix join us to talk about the opportunities. Gentlemen, welcome. First question, Sid and Sudeep, if I can address it to both of you. You both studied in India before moving to the US and building your own careers, successful careers in semiconductors and AI. What were those key moments in your journeys that led you to start the company together?
Sudeep, you want to go first?
Yeah, sure. So I had a strong foundation in engineering, first starting at REC Calicut, what is now called the National Institute of Technology in Calicut, and then subsequently I went to the US to pursue my graduate schooling at Purdue University in electrical and computer engineering. So I got interested in speech recognition, specifically around trying to get computers to recognize speech, and that was what my final year project was in my undergrad. And then after I finished graduate schooling, I went to work for a number of large semiconductor companies. I started off at Texas Instruments, then I worked at Lucent, Broadcom, and finally at Inphi. And so there, I worked for more than two decades, understanding how to build semiconductors, how to work with team members to come up with solutions to very difficult problems, and then build a product and ship product, right? And so after that experience, Sudeep and I were working at Inphi for a number of years, and we saw the emergence of AI, right, specifically around transformers in 2017. And we saw that the traditional way of building computers, of separating compute and memory, that existed from the early days, what's called the von Neumann classical computing methods that separated compute and memory, were no longer applicable, or not as applicable, to the new models and new AI algorithms that started showing up. And we realized very quickly that the process of moving data from memory into compute took so much energy, orders of magnitude more energy than the compute itself, that we needed to find a new way of doing things. And that's what Sid and I embarked on at d-Matrix, in trying to bring in what is called in-memory compute and technologies related to that to the market.
Sid, you'd like to tell about your story as well. And it just reminds me of, since you mentioned Texas Instruments, it reminds me of the charge-coupled devices, CCDs, and how they changed a lot of the devices that we use, but we'll try and talk about it. Sid, it's over to you.
Sure. Thank you. Thank you for doing this. You know, my background is, you know, I grew up in, I was born in Bombay, grew up in Ahmedabad, educated there, and you know, I come from a family of entrepreneurs. So it goes back, you know, to both my paternal grandfather and my maternal grandfather. This goes back to the 1940s. You know, both left home at a pretty young age, in their teens, you know, rural part of India, went to big cities, eventually, you know, learned the trade. In the case of my paternal grandfather, he learned the trade of, you know, what it takes to, he was a paper delivery boy for a long time, and then he learned the trade of what it takes to actually be a paper distributor for various, you know, businesses, and eventually started his own firm. This was back in 1945, right? And very successful for 40-plus years. My maternal grandfather, the same thing, you know, learned the auto spare parts business, and eventually, you know, had an auto, you know, parts, you know, this was back, you know, when India had, you know, foreign cars that were all imported because India was still not making its own, you know, indigenous cars, right? So everyone had, quote-unquote, a foreign car, and you needed, you know, spare parts for those cars, you know, when the part wore off. And so he had a distribution business that did that, and eventually built his own manufacturing plant. My dad was an entrepreneur too. Studied in the US in the 1960s, went back to India in the '70s, set up his own manufacturing plant. So I think entrepreneurship ran in the family. I don't think I had much of a choice, you know, growing up around, you know, my grandfathers and my father. So watching them, it was just a matter of time, you know, what would excite me and what I would embark on as an entrepreneur. In the 1990s, I think, when I was studying in Ahmedabad, you know, similar story, I was working on a project that was AI plus speech recognition. I think, you know, AI was going through its nuclear winter. You know, it had many nuclear winters, and the 1990s was not very different. So the place where most of the research happened around AI was in schools and colleges and research institutions, right? Because it had really not gone mainstream. So, you know, I had the opportunity to work on one such project around speech recognition, and after that I went off to the US, you know, worked at Intel, but very quickly realized that I wanted to jump in and, you know, build a company, right? Or at least learn how to build a company. So my first startup was in 2001. We were building connectivity chips for optical applications, well before, you know, when people were talking optics and how to connect networks using optics. That was a 10-year run, and that company ended up at Broadcom. And then after that I went to Inphi, where I crossed paths with Sudeep, and we built a business that, you know, essentially was incubated from zero and is currently running at about $3 billion in revenue. It was eventually acquired by Marvell Semiconductor in 2021 in a $10 billion acquisition. So very successful outcome for the team. But, you know, as part of that journey, as Sudeep was saying, we started seeing the emergence of AI, and a lot of the products that we were building at Inphi were being used to connect, at the time, training computing clusters, right? Those clusters needed a lot of bandwidth because you're connecting, you know, thousands of chips to each other, and you need to connect them using, you know, very high-speed bandwidth chips, which is what we were building. But that is what gave us an insight and a front-row seat into what is coming, right? And this was, you know, when we saw that, it clearly felt like a tsunami is headed our way, and, you know, there were many problems that needed solving. And inference, which is now, you know, obviously the whole world is talking about it, was something that was highlighted to us as an opportunity that would be much, much bigger than the training opportunity, right? And, you know, back then, you know, most people didn't even know what the word inference meant, right? We would go into investor meetings and say we want to start an inference computing company building silicon, and they would be, 'What is inference?' right? And we would have to educate them. Today it's a global word, everyone is talking about it. The opportunity has, you know, grown in size to the trillions of dollars, right? Because, you know, all of humanity wants access to AI, and really the only way to get access is to use inference. So we find ourselves really in the middle of a very large tectonic generational opportunity, which, you know, at the time when we started, didn't feel like it would go on to be so big, so fast. But clearly it has. And a little bit of insight, a little bit of luck, maybe a lot of luck to get to this point, but here we are. Ours is a story of, you know, grit and determination. I wish I could say we saw this all and we were all visionaries and we figured it all out well in advance, but nothing like that. We just stuck it out. We had a thesis and we just stayed at it, and here we are.
So that was the time for Intel, and the entire world spoke about Intel Inside, and in your case, looks like it's entrepreneurship inside, if I may say so.
That's a good point. I'm going to use that. Most welcome. Most welcome.
So you left senior roles, identified and started d-Matrix during the times of COVID, there was uncertainty, and also before AI, if I may say so, had become mainstream. What convinced you that AI inference, the inference part of it, was the opportunity to bet on?
Yeah, again, you know, it's mostly a hunch, right? As a lot of the best decisions, you know, at least I have made in my life have been on a strong hunch, right? Because data doesn't exist, right? If the data all existed and we could make a decision based on all the data, then not just us but many others would make the same decision, right? So when you're making a decision well in advance of what is likely to become a big opportunity, you have to make it on a hunch. And yes, there is, you know, you have 20% of the data available, and you're trying to make a decision based on 20% data, not 80% data. So in our case, it was exactly the same. We looked at everything on the horizon. We were clearly, as we stated earlier, right, had a front-row seat to what was happening with AI. And I think, you know, this is, you know, talking to customers, you know, peeling the onion, going back to first principles, and really understanding what is it that once you train all these models, because models were being trained at a certain velocity, right? They were getting bigger, they were becoming more capable. And talking about, you know, back in 2018, 2019, it was pretty clear what the trajectory looked like. And if you extrapolate that trajectory out, it became very clear that at some point in the future, there will come a time when these models would become, you know, sufficiently generic where, you know, pretty much anyone will want to use them and they could do a lot with them, right? So once you get to that point, what comes next? Well, the next thing is, it's pretty logical. Everybody wants to use it, and when everyone wants to use it, that's when, you know, inference really takes off. And, you know, in our mind, not everyone can train a model, but everybody, all of humanity, wants to use AI, and that's inference, right? So that opportunity, logically speaking, has to be much, much greater because people want to use AI 24/7, and everyone wants to use AI 24/7. So the multiplicative effect of that technology would be, you know, exponential when the capabilities were there for people to use it to do lots of different things, right? So we did not know when that would happen. So that was really the bet. It could have taken 5 years, it could have taken 10 years. There had been many nuclear winters in AI. It could have never happened, right? And the bet was that it would happen in an event horizon where it would be meaningful, you know, for us as a company, where we could capitalize on it. And that is where the luck part comes in. It happened literally within like 3 years of us starting the company with ChatGPT, and that was really the, you know, kind of the watershed moment for us, right?
So that AI inference, that focus on AI inference, was at a time when most of the industry was focused on training. What did you both see that others missed? If you can take that question.
