Thank you to the Irish Open conference for having me back again this year. This is really a wonderful conference for a great cause and I'm honored to be invited back with Jonathan.
Jonathan, you guys might have seen, has been in the news a little bit lately. He's the chief software architect of Nvidia, founder and CEO of Groq, and inventor of Google's TPU. He's had four first-time right silicones. He's one of my dearest friends and someone I'd consider to be one of the greatest engineers of our time.
Well, thank you. And for those of you who don't know John, although probably most of you do, John's one of those rare hedge fund managers who both gets great returns and provides immense entertainment value. Watching you hold those shorts like you've got diamond hands. I don't know how you do it. Everyone else is like sweating for you.
Not everyone's supposed to know I'm like a crazy short seller, so...
Crazy. Can't believe you just docs me like that.
So, you know, Jonathan called everything that's happening today like over 10 years ago. Everyone thought he was crazy. I was one of the lucky few that believed him back then. And, you know, up until a few years ago nobody even knew what inference was and so, I think that's a good place to start given it's now one of the most, probably the most important thing about AI today.
Well, for once I don't have to explain what inference is, that's nice.
Yeah, and so, you want me to get into the economics of what do you want?
Yeah, like, you know, one thing I think is very important is that, you know, portability across architectures is diminishing and porting is no longer really an engineering inconvenience but has evolved into more of an economic handicap. So, maybe we can talk through some of the tokenomics of, you know, inference.
Yeah, a good parallel here is we landed people on the moon about 60 years ago, roughly, and that was 60-year-old technology that got us there. What we're doing in AI is brand new technology. It's cutting edge even for today. We finally have enough compute to do these things with models. We've had the algorithm since the 1970s. So, what I'm seeing a lot of is people will look at one small part of the stack and think that that is the most crucial part, that's the bottleneck. And I don't think people understand just how much you have to build to make AI work. It's the chips, the packaging, the systems, the networking, the data centers, the servers, the racks, the power, all the stuff. But then also on top of that, it's both training and inference, which are two completely different problems. A little bit like, you know, getting to the moon and then landing on the moon were two very separate problems. And in terms of bottlenecks and all this, the focus I see a lot in investors are asking questions is what is the bottleneck? What's the next thing that's going to be constrained? What should I do next? And that's not how this industry works. Every time a bottleneck gets big enough, people solve it. So, as a person running a business, you're always going, 'Hey, what is my biggest problem and can I solve it?' And when you look at some of the components that are limited, when they're a problem but not a huge problem, people can charge a lot of money for it. But as they start to become a bigger problem, people start solving that problem. And so it's this constant shifting dynamic of where in the supply chain is the biggest problem, but don't become too big of a problem because then it'll get solved.
Yeah. So like memory is like the big theme today, right? It's the biggest bottleneck in AI. But so what you're saying is if memory continues to become a larger and larger bottleneck and continues to be extremely supply constrained that you think it's going to be solved?
Yeah, exactly. So memory used to be a commodity.
Correct. Yeah, it was the most commoditized segment of the semiconductor supply chain, pretty much.
Yeah, maybe a good way to explain this. There are two kind of goods that you can charge a lot for. One is a Veblen good and the other is a Giffen good. Veblen goods, the more you charge, the more desirable it becomes, like luxury goods. Giffen goods are a little bit different. A Giffen good is, an example is rice. So if you start to charge more for rice, some people can no longer afford steak, so they actually spend more money on the rice. So by raising the price, not for a luxury good, but by raising the price for a staple, you can actually increase the value. And that happens to some extent until someone goes, 'Let's stop eating rice and let's eat corn or something else, right?' And so there's this inverse sort of headwind where if memory is too expensive and if people don't build enough of it, people are going to solve that problem technologically.
So you think algorithmic efficiencies like, you know, for example, like Deep Seek's latest model release, the V4 release, it compressed KV cache by 90%. But the argument around that has been Jevons paradox, Jevons paradox, Jevons paradox.
