Hey everyone and welcome back to another Semi Doped podcast. Today we've got a special guest, R K Anand, co-founder and chief product officer at Tensorine. If you haven't heard of Tensorine, Tensorine is building an inference chip, an inference rack that does its math in a completely different number system and has other interesting architectural tricks that we'll talk about. So R K, I'm pleased to have you here to talk more about Tensorine.
Austin, it's a pleasure to meet you and I'm very glad to be here. I watch a lot of your podcasts and I'm a big fan. So I'm glad to be here today.
Oh, right on. Awesome. Awesome. I love that. Thank you. Okay, so let's start with you. You have a fascinating background in networking, which is really interesting because I think now that we are at rack scale, obviously networking is a big part of the AI inference story. So, you know, maybe tell our listeners more about you. Take us way back. You were at Sun, you were a founding engineer at Juniper. Kind of tell us about your career arc that led you all the way here.
Yeah. So, you know, this is my 37th year in Silicon Valley. I started my career at Sun in 1990 actually, designing microprocessors. So I was part of a microprocessor team and Sun was building high-end servers for the market. So it was a really fascinating time. There's a lot of competition. There was RISC versus CISC, and you got exposed to building really large multiprocessing systems. Fast forward a few years. So 1996, I was fortunate to join Juniper Networks as one of the founding engineers and led the first silicon efforts there. And the reason I was able to do that was because Pradeep Sindhu, who's the founder of Juniper, was one of my early mentors at Sun. He was part of a team from Xerox PARC research center helping us build these big machines, and he gave me the opportunity to join Juniper. And I spent almost 17 years there with a little bit of a break, and saw the company from zero to slightly over four and a quarter billion dollars of revenue. And the most fascinating part was that we were in the early days of the internet. The internet was doubling every six to nine months, and I think Juniper transformed networking pretty significantly. So it was a very good learning opportunity to start from scratch, grow with the company, and eventually be very successful in building the biggest routers in the world. But most importantly, I think, building great relationships and great friendships and being able to mentor a lot of the talent there. So I think it was a great learning experience for me.
Oh man, it sounds like it. Okay. So interesting. So you got to work on compute back at Sun and then you got to work on networking for a long time at Juniper and then here we are in this era which is both compute and networking. So it seems like you have learned all the right things that you need for this era.
Yeah, it's fascinating. You know, I think Silicon Valley, if you've been here long enough, goes through cycles, right? And we're in one more of those really great innovation cycles. And yeah, you're right. You know, computing, you think about computing and you say, okay, general purpose computing or high-end client-server computing, and how was that built? You know, because we started from PCs and, you know, mainframes, then PCs, and they went to client-server. And Sun was a premier company, a great innovation, a great place for innovating and building great technology. And then Juniper ends up being the premier company in networking, building high-end core and edge routers, right? The fact that we have the internet today that works so flawlessly is because of all the work that companies like Juniper, Cisco and others did. And then we come back to the world of AI and it's again back to computing, but then we start hearing for the last couple, three years like networking is so central to AI. So it's a coming together of all of these things, and you know, you can then reflect back and see if you can take those learnings and apply them to what you're doing today, I would say.
Sure. Yeah. Totally. So what I think is interesting, and you know, correct me anywhere I'm wrong, but when you guys started at Juniper, I mean, I assume that Cisco was probably the dominant player in the space.
They were. You know, Cisco was started in the early 80s and by probably mid-80s, I don't remember the exact time, but when Juniper started in 1996, the browser had just appeared, right, out of Urbana-Champaign and Netscape, the Mosaic browser and Netscape had appeared, and we were trying to put a lot of our data on the internet and we were trying to start sharing our, think of it almost like sharing your hard drives basically. And they were these internet service providers like UUNET was an example of them, and they were using Cisco routers and Cisco was the dominant player. I think Cisco had like 93, 94% of the core router market share. And fortunately for us, Pradeep, who was the founder of Juniper, looked at the way routers were getting built and he was trying to wonder why are they not getting built like high-end servers.
So he had a general thesis that he could actually build routers far more efficiently. And then we got to hear from people like Mike Odell, who was CTO at UUNET, that the internet was doubling every six to nine months and he was not able to keep up. And so Juniper fundamentally rearchitected how routers were built and that completely transformed how the growth of the internet happened. Cisco by the year sometime 2000 was the number one market cap company in the world, you know, and John Chambers was on the cover of every magazine in the world. And Juniper appeared in the landscape and we built the highest performing router and then I think within the span of a couple of years we had 30% market share. So there is proof positive that when new markets evolve there are opportunities for startups to come in and if they have technology that's differentiated they can have an impact in the market.
Nice. Awesome. Yes. You took it exactly where I was thinking in my head, which is like, wow, this actually feels very similar to back then where there's a new technological revolution happening that's impacting the end users, which was the internet at the time and it's LLMs today. And there's an incumbent that of course, you know, is the largest market cap company in the world, and is quite dominant. And I'm sure when you guys started Juniper, there was probably a lot of people that just said like, what are you doing? How could you ever possibly compete with Cisco? But then what I heard you say was the interesting insight is like, well, if you think that there's a different way to tackle the technology that ultimately you do think you can compete on technical performance and then probably TCO and other things.
Yeah. And you know, of course, the battles only quickly end there because the size and scale of the market is kind of way, way different now. And the competitive landscape, you know, I mean, in those days the internet, you know, information was more constrained. Everybody knows we live in a different world today and information is instantaneous, everybody knows everybody else. But in that period between say 96 and 99 there were many other companies that were funded to build routers. I mean, all those names are etched in my mind, right? There's Avici and there's Pluris and Nexabit and there's a bunch of companies. So there's an opportunity, a landscape opens up, an opportunity opens up. I think if you look at today the scale is orders of magnitude larger. The market capitalization of some of these companies is quite significant and then the technology power that is there, especially with Google or a lot of hyperscalers also building silicon systems, is that there's a lot more information. But I think there's a, you know, just like anything else, there are waves and there are changes that happen. I think that if you look at the last five, six years, the initial phase was all training, AI training, right? And then we have slightly shifted from training to inference. And inference was initially we were doing monolithic kind of models and now we're in the world of two trillion pluses. So the architecture demands for inference are different from training. And then for monolithic models to, you know, two plus trillion models or 2.8 like Kimi K2, the pressures on the system are different. So there's a window of opportunity that opens up if you think about it and how can you build a system that addresses demand for today's market that looks different from something that's four, five years back.
Yes. Perfect. Okay. So lead us into from there then into Tensorine and what we have this window here. It's inference at just kind of massive scale. It's mixture of experts, two trillion parameters. We want high interactivity these days. It's agentic AI, right? So agents want to run at thousands of tokens per second ideally. And so therefore, so we've got this gap, we've got this window. There is definitely a way that everyone's doing it now which is mostly like running this inference on racks and racks of GPUs and Tensorine is taking a different approach. So like tell our listeners what is the technical differentiation that you guys are pursuing that you think is a different bet?
If one were to break down the problem, you know, the models are immensely large, they, you know, so they don't fit on single devices anymore. So now you have to have many devices that somehow operate in a kind of a synchronous or orchestrated mode. Well, how do they do that? Well, you have to connect them together. So there's a networking problem that's in front of you that makes it apparent that the model ain't fitting on a single device. You got to spread it across many devices. Then there are power and other constraints that are in front of us, right? We can't get power quickly enough. These data centers are large and, you know, we didn't, I mean, we didn't start at this point, right? If you rewind a little bit, Tensorine was prior to this named Recogni, and we had our Series A funding in 2019 and we were more at that time, the world was about image and image recognition and ResNets and ResNet-50, ResNet-101. And we started as an inference company. So we, but because it was image recognition we had to think, okay, how are we going to build a piece of silicon and a system that can do really exceptional, deliver high performance for AI, for inference, for image recognition. And the market was automotive that we focused on. Automotive has a power constraint. The car can only give you so much energy. So there's a power constraint and you have to deliver inference and you have to process, let's say, incoming frames from cameras that are at 8 megapixels. So we set that up as a problem statement. And then the realization was that if you took traditional math and tried to do it in the car, you would blow the power budget of a car. You know, rather than using it to move the car more miles, you would basically, you know, shrink the range of the vehicle. So our approach to inference at that time was like, let's think about mathematically it being different and let's approach math from a first principles math perspective and that's how we converge on using logarithmic math for AI. So that set up, if you want us to be fortuitous, that set up the foundation for Tensorine today. We didn't build a perfect device, but we validated our math, we validated that it really worked for AI. And the analogies were that, you know, first of all, log math has been around since John Napier invented it over 400 years back, so it's not something that we dreamt of. And log math has been, you know, if you think about like the Apollo program and all these programs, people used to use slide rules and log math to determine escape velocity and re-entry and all these things. And then over the last two, three decades, we've been using log math in DSPs. So it's not like we were the inventors of log math, but our realization was that can we use log math in AI? And so that helped us, right? And set us up for today.
Gotcha. Okay. Okay. Interesting. So for listeners who haven't heard of Tensorine before, Tensorine actually started, it's a different company. What did you say the name was?
Recogni. Okay. So probably image recognition. Maybe it came from that.
Yeah. Yeah. Yeah. Exactly.
So you were looking at like convolutional neural networks, that kind of stuff pre-transformers. And you were asking, okay, how can we do this at the edge? Thinking automotive. So, you know, self-driving cars, that kind of thing makes a ton of sense. And you obviously a power constraint there, which I'm sure we'll get to it. Same power constraints in the data center. Same but different, but like thinking about power. But you were taking a, you were asking how can we do this incredibly efficiently. And the idea was to use log math instead of the traditional like matrix multiplication, multiplies and adds. Like tell our listeners a little bit more about like what does log math unlock as the benefit.
So you can't avoid matrix algebra. Matrix algebra is the fundamental for all of AI. So it's not like we could avoid that. But if you take a number and convert it to a log number then, and most of the operations are what I call MAC, right, which is basically if you think about it, it's a multiplication followed by an accumulate and addition. So we're doing trillions of multiplications and additions in AI. Whether we're doing it in convolutional neural networks for image recognition or in LLMs, it's exactly the same. So when you take a number and make it a log number then log base 2 of a times b is log a plus log b. A multiplier becomes an adder. And you know, let's think of ourselves as young children. Let's say we are in elementary school and we're trying to multiply two numbers or add two numbers. Adding is easy, right? You just take the numbers and add them. But when you want to multiply, you know, you have to do this long multiplication and add a bunch after that. It's the same thing in silicon. If you think about it, you have to have a lot more gates. You're going to take a lot more area, a lot more power to multiply two numbers. But you want to add two numbers. It's more easy. Adders are, you know, like a building block in silicon. So you solve the first part of the problem. The second part of the problem is really hard because now you have to take the result of the multiplication and add it and the minute you are in the logarithmic domain adding becomes a problem. And if you were to take a traditional approach of a lookup table or Taylor series, all the gains that you have in going to addition in place of multiplication are lost. So our fundamental innovations that we have patented was to figure out how to go back from a log to linear space and perform the additions without losing the gains in area and space and power that we got from multiplying the numbers instead of actually adding them. Right? And so we spent inordinate amounts of time in mathematics. So almost half the company spent almost half a decade in math, right? Because that's how you can be a revolutionary startup rather than an evolutionary startup. Because if you do the same thing like everybody else does and you have access to the same silicon geometry, 7 nanometer, 5 nanometer, you're going to have no advantages. You can't compete against the biggest guys. So you actually have to think about this differently. And that goes back to how, you know, in my thinking reflecting back in 96 and 97, how Pradeep and the other co-founders of Juniper thought about routers, right, and do it differently, forwarding in silicon versus forwarding with, you know, microprocessors. So the similar analogies applied here and so we spent a lot of time thinking about that.
Nice. Okay. Awesome. So you reach for log math because you realize, okay, we have a bunch of MACs to do and if we instead, and multiplication is expensive from a power and a die area perspective and ultimately probably latency as well, and therefore if we can convert it to the logarithmic domain we can turn these into adds but then eventually it has to get turned back, converted back and you guys thought really hard about how to do that efficiently, effectively, cleverly. And so then you said, okay, we figured this out. This is great. This is more power efficient, small, fast. So then you ultimately at least built chips for automotive to kind of prove out essentially the R&D side of things. And okay, now connect me from there to getting into data center inference.
All right, let's do that. Let's do that. So think of a convolutional neural network as like a 3x3 matrix. So you have to perform nine operations, nine multiplies followed by all the additions that add up those nine numbers. So we had the building block for it and we had functioning silicon in 7 nanometer that was fully functional and we were able to demonstrate taking camera streams and doing that. We hadn't incorporated a lot of the safety features that we needed for automotive. And so at that time in, you know, late 22, early 23, ChatGPT happened and we realized quickly that our building blocks were really set up perfectly for this moment. The market was 50 or 100 times larger. The ability to have a product from power on to getting to revenue is much, much faster in the data center, in networking rather than in automotive because automotive has a long call cycle, right? It takes a long time to build a car, you got to certify it, you got to make sure it's safe, there's a number of things that need to happen. So we coalesced as a team and said, let us look at our technology and see how applicable is it for LLMs. In LLMs you do one-by-one multiplication, not 3x3. Great. Well, LLMs. So if you look at ResNets and all, thinking of 100 million parameter systems and now you're going from 100 million to multiple hundreds of billions to trillion parameter. So the memory hierarchy, the memory architecture has to change. You have to think of memory slightly differently.
But your fundamental building blocks for math you can build upon. And so that's what we did. And so we started architecting in 2023. So we had the good fortune of retrospective view on math and 100 million parameters. But we also had a view that what was the problem statement in front of us that is that you could go from multiple hundreds of billions to maybe trillions of parameters and so how do we build and architect a chip for that? We still hadn't solved the networking problem though. So we said, okay, let's understand how to build first an accelerator that could be world-class and deliver world-class performance for LLMs and then the multi-hundred billion parameter problem now became a multi-trillion parameter problem.
And so now you have to think about the networking problem next.
