Nvidia is going to have better networking than you. They're going to have better HPM, better process node. They're going to come to market faster, ramp faster, have better negotiations with TSMC, SK Hynix, memory, silicon, rack people, copper cables, everything. They're going to have better cost efficiency. So, you can't just do the same thing as Nvidia. You have to really leap forward in some other way. You have to be like 5x better.
Dylan, welcome to the podcast. We've been trying to get you for a while. You're a busy man, but it worked out. Gu, I want you to introduce why we're so excited to have Dylan on the podcast and what we're excited to discuss.
Look, I mean, Dylan, you've done an exceptional job in covering what's happening in the AI hardware space, AI semi space, and now more and more data center space as well. And you know, just looking at it currently, the most valuable company on the planet is an AI semi company, right? The biggest IPO so far in AI was an AI cloud company. This is currently where it's happening. Right, in any gold rush in the early days, it's the pick and shovels that make money, and I think this is the stage that we're in. So super excited to have you here today.
Awesome. Thank you. Happy to talk about my favorite topics.
Amazing. Well, maybe let's start with GPT-5. You know, we just had some of the researchers from OpenAI on here last week. You said it was disappointing. Why don't you share your reactions or what capabilities you were hoping to see or overall?
I think it depends on what tier of user you are, right? And so like, right, if you're just using GPT-5 and before you were a $20 or $200 a month subscriber, you no longer have access to 4.5, which in my opinion is still a better pre-trained model for certain things. Or you no longer have access to o3, which would think for 30 seconds on average, maybe, right? Whereas GPT-5, even when you're using thinking, only thinks for like 5 to 10 seconds on average, right? Which is an interesting sort of phenomenon, right? But basically like GPT-5 is not spending more compute per se. The model did get a little bit better on a vanilla basis, right? 4o to 5 is actually quite a bit better. But when you think about, you know, what is this curve of intelligence, right? It's like the more compute you spend, the better the model gets. And that's whether it's a bigger model, which GPT-5 isn't, right? You can see it's not a bigger model. It's roughly the same size. And you know, or you think more, right? But again, like this is something that OpenAI's first thinking models, you know, the first few generations of o1, o3, would think for a long time and waste a lot of tokens, if you will. And when you look at like, for example, Anthropic's thinking models, they actually think, even when you put them in thinking mode, they think a lot less, right? To get to the same results or better results, right? As OpenAI was. And so OpenAI, I think, optimized a lot of like, well, if I ask, like, I think the silliest one I had asked was like, I asked o3 once like, is pork red meat or white meat? And it thought for like 48 seconds. I was like, what are you doing? Like, this should just tell me the answer. And so like, you know, the nice thing is that GPT-5 will think a lot less even if you select thinking manually. But more importantly, they have the sort of auto functionality, the router, which lets them decide whether or not, you know, hey, do I route to the regular model? Do I route to maybe mini if you're out of rate limits, or do I route to thinking, right? And how much do I think? But in general, the thinking model will think less. So there's less compute going into a power user's average query than before.
But isn't it even more interesting? OpenAI can now control how much compute it wants to allocate to you, right? If we're in a high load situation, maybe tune the router a little bit so it's less, right? Maybe I have no idea what they're doing behind the curtain, but there's this meme out there at the moment that basically all they did, which is a meme, right? It's not true, but all they did is, you know, take all three plus a couple of smaller models, put a router in front, and offer that at a lower blended price essentially, right? Is I think there's a little bit of that, right? Cost suddenly matters and they figured out a way how they can steer that.
I think yeah, that I mean, and they talked about how they've been able to dramatically increase their infrastructure capacity because I myself was just regularly using o3 or 4.5, right? And now I'm forced to use auto which sometimes gives me, you know, the o3 equivalent thinking model but sometimes gives me, you know, just the regular base model which sucks. But like, I think for the free user it's actually quite interesting, right? The free user was not getting thinking models pretty much ever, not using them, or in many cases they just open the website and ask their query and now sometimes their query gets routed there so sometimes they get a way better model but now sometimes, you know, OpenAI can gracefully degrade them if they need to, right? And it's like, I think the router points to the future of OpenAI from a business, right? Like you can look at sort of the model companies, right? Anthropic is fully focused on B2B, right? API, code, etc., right? Where, or Claude Code, whatever it is, right? OpenAI, yes, they have that business, Codex and API business, but really their majority of the revenue is consumer, right? And it's consumer subscriptions, but they have no way to upsell, like, you know, to make money off of all the free users, right? In any other application, consumer app, the free user still pays via ads. But this is not compatible with AI, right? Like it's a helpful assistant. You can't just make the user's result worse by injecting ads. Banner ads don't really work in AI either. So it's like how do you now monetize them? And I think they're getting with the router, they're getting really close to figuring out how to monetize that user, right? With the new CEO of applications, if you saw her product that she launched at Shopify, I think it was Shopify, was an agent for shopping, right? And now this like immediately clicks like, oh, if the user asks a low-value query like, hey, why is the sky blue? Just route them to mini, right? Like, you know, the model can answer perfectly fine and that is a chunk of queries, right? But if they ask like what's the best DUI lawyer near me, right, all of a sudden this is like, you know, you're in jail, you have one shot, you know, you're like screw it, let me ask ChatGPT what the best DUI lawyer is and now all of a sudden, you know, the model's not capable of it today but soon enough it'll be able to, you know, contact all the lawyers in the area and figure out what their results are and maybe search their court filings and whatever, right? And book the best lawyer for you. Or an airplane. Negotiate a cut, you know, as part of that. Yeah, of course they're going to take a cut, right? But this is a much better way of like monetizing the free user. It's like, you know, it's like Etsy, 10% of their traffic now comes from ChatGPT and OpenAI makes nothing off of that, but they really will soon, right? And partially that's because Amazon blocks ChatGPT, but like, you know, there's a way to make money from shopping decisions, whether it's booking flights or looking for items. And those you now say free user, I don't care. I'm gonna send you to my best model. I'm gonna send you to agents. I'm gonna spend ungodly amounts of compute on you because I can make money off of this. But if it's a query that's like, help me with my homework, I'll send you like a decent model, right? I don't need to spend money on you. And so this is how I think OpenAI can finally make money off of the free user. And I think that's the biggest thing about the router, right?
Like this is super interesting. I think this is the first time that we've seen that there's a launch of a new model where to some degree cost is the headline item, right? I mean so far it was always like who has the smartest model, who has the highest MLU score. Now we have suddenly people who use models for coding for 8 hours a day and, you know, surprise, that, you know, if you take a large context window and the best model, creates, you know, thousands of dollars of cost a month. So cost matters. And so to some degree, sway on the frontier between cost and performance is the new benchmark for model competition. No longer where cost alone is that. What we're seeing here, or?
I mean, I think definitely, right? Like OpenAI said they doubled their rate limits for, you know, for big amounts of users. They've dramatically increased the number of tokens they're serving from this launch which effectively says this is an economic release. And I think like, and probably also means the tokens are all cheaper, right? Otherwise, yeah, for sure, for sure. I think the funniest thing is like, you know, this whole cost thing you mentioned is like we've seen this in the code space, right? Like Cursor had to pull away the unlimited cloud code, you know, initially they had like this super expensive plan and it had like unlimited rates and then they or only like a weekly rate limit. Now they have like hour-based rate limits. And I saw the craziest like thread on Twitter where this guy said he changed his sleep schedule, right? Modeled after like how sailors in the bay, like, you know, like if you're sailing you can't sleep, right? Like solo sailing. They'll take like power naps when they get to the right spots so that they can like still be safe in the morning when it's not very windy. Well, but like they can't sleep uninterrupted, right? And so because Anthropic had to put rate limits that are like not just week-based but like a number of hours based, he's like, he basically sleeps, you know, multiple times a day, but small chunks just so he can maximize the usage. And there's also a leaderboard on Reddit where people are like competing to see how many tokens they're using through their subscription. And there's like a dude spending like $30,000 a month.
So I'm going to find some developer in India that I do pair programming with so I can get the day cycle, he can get the night cycle, you know, and we both can maximize together the quota for the account. Is that the future then?
I mean, but it's clear like people are taking advantage of the negative gross margin, like sort of, you know, subscriptions that are offered. You know, I think Anthropic probably makes a positive gross margin off of my subscription. Like I don't code enough, but like there's plenty of people that are definitely losing money. And so as you said, it's an economic, it'll push more and more to I think just usage-based pricing, right? I think if you have an underlying commodity that you're reselling to some degree that is a large part of your cost of goods, right? You need to go to usage-based pricing.
How much do you think the like customer capture and stickiness for these code products is like? Yeah, I'm curious what you think on that, right? Like, you know, once you use an IDE, once you integrate, you know, one of the CLI byproducts, like how sticky is it or is it a billion dollar question?
That's a very conservative estimate. Look, and Andrew Parthy has this great slide where he basically says if you're building an agentic system today, right, but fundamentally what it is is this loop, right, where half of the loop is the model thinking, right, and trying to do something. The other half is then the user verifying what did the agent do, is it the right thing, providing feedback and trying to steer it in the right direction because, you know, we can't run forever, eventually you need to steer it back. One half of that is the model provider, right? They're trying to build the best models. The other half is really about, I think, designing the best possible UI to enable user to give feedback. And I think there's value in that. So, I think there's a certain amount of stickiness in there, right? So, what are all the different tools like in terms of, say take code editing, right? How can I most easily visualize what the code changes are? How can I most easily visualize, you know, what they impact, which files, you know, how can I for small changes get very quick feedback versus for complex ones, you know, get complex feedback. There are some tools that actually draw diagrams for you of what they do, right? So I think this will be the battle. So I think there's stickiness in that, right? You know, how much exactly, this was a great question. So in that sense like people should be doing subscriptions to get people locked in, right? Instead of moving to usage-based pricing.
Well, I think it's the customers that don't want to do usage-based pricing because it's so hard to guarantee, you know, it's so hard to get it to get away from them. And you actually want guarantees and you're willing to like, say you're willing to commit to pretty high spend in order to not have usage-based pricing. I think it's the model companies that want usage-based pricing.
I think with consumers, it's frankly very hard to not have usage-based pricing just because the variability is so massive, right? If it's like, you know, us coding versus somebody who does this as their full-time job, right? You just have a factor of 20 or so difference in usage. And if that costs a lot of money, right? I think it's just, I think for enterprises, we could see like more flat fee pricing because you can average it out more. You have a developer that's using it all day. You kind of know in a general sense like how many hours a day they're programming and what that sort of looks like. The vibe coders are harder. Yeah. Yeah.
Before we leave OpenAI, I want to ask a broad question which is if someone was sitting here and saying, hey, Dylan, I'll listen to anything you tell me to do. Any advice you have as long as it makes OpenAI more valuable? What would you tell them?
I would say like immediately launch a method for you to input your credit card into ChatGPT and agree that for anything it agentically does for you, it'll take X cut and then launch that product because where it does shopping, right? Is like we know everyone knows that like Anthropic and OpenAI and all the other labs are buying RL environments of Amazon and of like Shopify and of Etsy and of all the different ways to shop on the internet, of airline websites. Right now just like, hey, integrate my calendar, I want to fly to there on Thursday, make sure I don't miss a meeting, cool, book, right? Like do that integration like super well, know my preferences on whether I like aisle or window, all this stuff, right? And just take a take rate. I think like this will make them so much money the moment they launch it. And I think they're working on it already, but like I'd like to hear how he thinks about it. Because he's shifted his tone massively on like ads over the last 6 months, right? He used to be like, no way. And now he's like, maybe, you know, there's a way to do it without, you know, harming the user. And I think this is how you monetize the free user, right? So I think that's probably what I'd tell him slash ask him about like a whole line of questions around this.
I want to shift to Nvidia. Nvidia is having a monster year. They're up almost 70%. What are the possible paths from here? How do you see it playing out?
Um, depends like how bullish you are on like the continued growth, but like I think you guys have a good vantage point. We have a good vantage point of how fast revenue is growing for a lot of these companies, especially the code companies, but even many other applications. I think we can clearly see the demand side is accelerating, right? And then if you look at the training side, I think the race is on. You know, Meta's upping hugely. Google's upping hugely. You know, if you just look at like again just OpenAI and Anthropic and the compute that they have and are getting this year, from Google and Amazon for Anthropic and from Microsoft, CoreWeave, Oracle for OpenAI, 30% of the chips are going to them just those two companies. But that's actually like, okay, well like 70% of the stuff, like who's making it? Well one third of it is like ads, right? Whether it be ByteDance or Meta or many of the other people who are doing ads. So then it's still like, okay, well where are the rest of these one-third of the chips coming from? Well they're like mostly uneconomic providers who I don't think it's like an obvious bet that they're going to keep growing, you know, keep raising bigger and bigger rounds. So what happens there? I think with the, you know, we talked about coding earlier, actually the Qwen Coder 3 model is actually super cheap if you're running it on prem or if you're running it in the cloud with all these inference libraries. And so like there's stuff like that as well. So I think the question is like how much does it keep growing? Because clearly, you know, I think the first third is definitely skyrocketing, right? Of OpenAI, Anthropic, lab spend. The second third of like ads is gonna grow, it's not gonna grow like crazy but I think there's definitely an inflection point that could be hit with GenAI ads. I know Meta has been experimenting with it a lot. But I could totally be convinced that there's going to be a huge inflection in like take rate there, right, where you start showing me personalized ads. Like every person that's an ad looks like me and I'll be like, okay, yes. Except like slightly better, so I feel better, right? And I'm like, I want to buy it. Yeah. It's interesting, I have no idea how this is going to scale, right? But if you ask the question how much could it scale, right? Like how much value are we creating here? Can we create enough value to actually keep growing for a long time? If you just take AI software development, right? We know we can easily get about 15% more productivity out of a...
I don't think that's right. I think it's way higher.
No, no, that we, with a straight, like I talked to a lot of enterprises, like a classical enterprise, straight up, you know, GitHub Copilot deployment that gives you about 15%. We can do much more than that, but bro, like you know how bad GitHub Copilot is, like...
Like how did they, how, look at the revenue ARR, it's so funny.
It's so funny if you look at the revenue ARR chart and it's like, it's like Claude Code in like 3 months has surpassed them and like Cursor, you know, easily surpassed them and then like even like companies like Replit are like and Windsurf/Cognition are like going to pass them. Like it's like you're preaching to the choir. So look, let's assume we can get this to 100%. So we can double the productivity of a developer, right? About 30 million developers worldwide, give or take. Yeah. Right. Let's say 100k value add per developer. This might be a little high worldwide. US is low, but worldwide is high. So it's $3 trillion. Yeah. Yeah. Right. So we're probably building technology here which adds three trillion dollars of GDP value. In theory, we could put that into GPUs because that's the main, just from a coding model, like just from this one use case. So at least in theory, the value generation is here to keep growing, right? Now how that translates to the industry is much more complicated. I think we've already seen AI's value creation exceed, so sort of there's like the whole like the famous like oh 300 billion problem or 200 billion problem, now it's 600 billion problem. I'm sure someone's going to put out like the 1 trillion problem right soon enough. But like there is some like reality in that, of course, but you know, ignores that like infrastructure spend today is accounting for five years of revenue not like one and the revenue looks like this not like flatline. But I think the main thing is that AI is already generating more value than the spend. It's that the value capture is broken, right? Like I legitimately believe OpenAI is not even capturing 10% of the value they've created in the world already. Just by usage of ChatGPT, right? And I think the same applies to, you know, Anthropic and Cursor and whoever else you're looking at. I think the value capture is really broken. Even like internally, I think like what we've been able to do with like four devs, in terms of like automation, like our spend on like Gemini API is absurdly low and yet we go through every single permit and regulatory filing around every single data center with AI and it's like, and we take satellite photos of every data center and we're able to label our data set and then recognize what generators people are using, what cooling towers and the construction progress and substation. All this stuff is like automated and it's only possible because of GenAI. But and we do it with like very few developers and then like the value capture that I'm able to generate by selling this data, by consulting with it is so high but the companies making it is like, they get nothing out of it, right? Like I think this, like there's a value capture challenge here that far out exceeds the sort of creation. And as you get models like GPT-5 or open source models like continuing to drive it down, it's like the value capture is just harder and harder and harder for these companies because they're making, you know, 50% gross margin on an inference if they're, or less in many cases.
In so many words, you're saying we're getting commoditized and therefore you can't capture the value and thus you should temper your expectations of how much you can spend on GPUs.
Well, no, I think you can, I think there's still ways to like inflect hugely on value capture, right? But I mentioned the ads are a huge value capture that needs to happen before we see massive increase. No, I think the other thing is like there's a lot of capital that's not been spent, right? Like the hyperscalers still can grow capex 20, 30% next year, right, from what they're doing this year. In addition, companies like CoreWeave and Oracle because they're tapping capital markets can raise way more than 20 to 30% capex. And then you go down the list further and it's like, oh, the largest infrastructure funds in the world like Brookfield and Blackstone. Well, actually they're turning all of their eyes to investing even more into infrastructure, AI infra. And then you're like the sovereign wealth funds of the world like the G42s or, you know, the Norway one or GIC and Singapore, like these people have barely started touching AI. And so I think there's a whole lot more capex that can come without it being necessarily like economically motivated day one. I'm more so saying like economically motivated capex can only grow like so much, but there's so much other like, where it's not clear from, you know, if you have a spreadsheet, you know, and you're basing it on real business that you should actually spend this much. But people will because they believe, I believe, I think you believe, like infra, you know, people believe that this will be, you'll get profit out of it, but there's no like 100% certain like way to argue it. Yeah.
How threatened, if at all, is Nvidia by custom silicon?
I think that's the biggest thing, right? Is when we look at orders from Google and from Amazon, right, especially, and Meta, their custom silicon is not, not Microsoft, their custom silicon kind of sucks, but the other three, they're really upping their orders massively over the last year. You know, Amazon is making millions of Trainium, Google's making millions of TPUs. TPUs clearly are like 100% utilized, right? Trainium's not there, but I think Amazon will figure out how to do that. And Anthropic will. So I think that's the biggest threat to Nvidia is that people figure out how to use custom silicon more broadly. And this sort of becomes the sort of like, if AI is concentrated, then custom silicon will do better. And that's not even talking about like OpenAI's silicon team and stuff, right? If AI is really concentrated, then they'll do better custom silicon. But if it gets dispersed broadly because there's all these open source models from China and there's all these open source software libraries from, you know, Nvidia and China and it makes the deployment costs like rock bottom, then potentially, hear me out here, if Google's TPU is able to compete with Nvidia, in theory, could do it on the open market, is worth more than Google these days. Shouldn't Google start selling their chips to everyone?
I mean, in theory, they should be able to achieve a higher market cap.
I absolutely think so. I think Google's even discussing it internally. I think it would require a big reorg of culture and a big reorg of like how Google Cloud works and how the TPU team works and how the JAX software team and XLA software teams work. I totally think they could. It would just take them like shaking themselves pretty hard to be able to do it. Yeah, but I totally think Google should sell TPUs externally, not just renting, but like physically.
It's kind of funny if a side hobby in theory has a higher company value potential as your entire business, especially as you think about the degradation of search business. I mean, yeah.
I think but I think like if you were to ask like Sergey, right, like, hey, do you think selling chips and racks is more valuable or cloud or Gemini? He'd be like, no, no, no, no, no. Like Gemini is going to be worth way, way, way more. It's just not yet today, right? And so I think like today you say Nvidia is the most, again it's like a whole concentration thing, right? If the world is super concentrated in terms of customers, then Nvidia will not be the most valuable company in the world, right? But if it gets dispersed more and more, which arguably we're starting to see with a lot of these open source models getting better and better and better, and with the ease of deploying them getting better, then you would see, I think you could argue Nvidia will remain the most valuable company in the world for a long period of time. And historically, no pun intended, software has eaten the world in most markets, right? I mean like if you look at early networking days, Cisco was the most valuable company on the planet, right, for a while. It's no longer, right? The guys that built services on top like Google or Amazon or Meta eventually eclipsed. Well, which is why Nvidia is like making all these software libraries, right? That's, and they're trying to commoditize inference, right? Like, you guys don't, I think even have an inference API provider investment, do you?
Well, we have all kinds of model providers.
Model providers, but I'm talking about a pure API provider investment, I think, right? Is that correct? I think I talked to one of the team members, maybe Rajeev or someone, about like why you guys didn't invest in like, you know, like a Together or like a Fireworks. And sort of the argument was like, well, we think just serving models alone without making them will sort of be commoditized, right? We have some in the stable diffusion ecosystem like with Fal.
Yeah, it's a little bit different dynamics there. I think they tend to make much more compound models than the LM folks. I think that's a little bit different.
But like you guys don't have one of these like, you know, Base10 or any of these like sort of like API investments because you think, this is from someone on the infra team, that you guys think it'll get commoditized because the software Nvidia is making, because vLLM and SGLang, which is like open source software coming out of Berkeley and now, you know, sort of has their own environments now and supported by many, like this being commoditized means that like API providers aren't necessarily worth a ton, right? Is sort of your argument maybe. I think that's relevant to this whole thing which is, you know, why, why, right? Like why would you do this?
Shifting gears, what about the silicon startups? What's your take on those? I mean, there's a ton of capital flowing into that, right? We've seen, I have not numbers, but probably billions being invested in chip startups.
Yeah, for sure, for sure. I mean like whether you're looking at like, you know, companies like I think it's like pretty impressive that a few companies like Etched and Rivos and a number of other companies, you know, MatX and others, like have gotten the amount of funding they've had without even launching a chip, right? You know, in the past like, yeah, silicon companies would make money or raise money but they would at least launch a chip before they get a, you know, a big round. But like Etched and Rivos like have raised, you know, a lot of money without ever launching a chip publicly, which I think is, I mean, it speaks to, well, like yes, silicon is super capital intensive if you're building a chip, especially an accelerator which has so many moving pieces. And there's like 10 different AI accelerator companies out there, right? Like that are newish in the last few years. I think there's a lot more that are like, yeah, yeah, yeah, that's fair. And then there's the old guard which continues to raise money, right? Like Groq and Cerebras and SambaNova and Tenstorrent and so on and so forth, right? Like, or Graphcore getting bought out by SoftBank and SoftBank dumping money into this effort as well, right? There's a lot of capital being invested to dispel sort of Nvidia's top dollar or top position. But it becomes challenging, right? It's like how do you beat Nvidia, right? Like the hyperscalers I think are like kind of lucky in that they can do mostly the same thing as Nvidia, right? They're a captive customer which is themselves, right? It's a huge, and they can just win on supply chain, right? Like I'm using cheaper providers. It's a margin compression exercise essentially. Yeah. Yeah. And maybe for certain workloads like recommendation systems, they'll have a better, you know, they can specialize more. But for the most part, it's like no, we're targeting the same workloads. We can just simplify supply chain or in-house a lot of it and compress margin and it'll be fine. But in the case of, you know, these other companies, it's like, well, they don't have a captive customer so now you have to contend with, well, I'm using the same ecosystem and either I can use some custom silicon provider who's going to take a margin anyways on top and that's going to compress my, like what I can sell for, or I can try and in-house everything but then it's like this is really hard, right? Like I'm going to do all the software design, I'm going to do all the silicon design, I'm going to build all this different IP, I'm going to manage the supply chain pain on chips, on racks, on everything, right? Ends up being a huge effort in terms of team size. All in the end, like, hey, I make a 75% gross margin as Nvidia. AMD sells their GPUs for 50% gross margin. And they have a hard time out-engineering Nvidia and they're great at engineering, right? Like, they, but yet they still take more silicon area, more memory to achieve the same performance, and they have to sell for less. So, their margin gets compressed. That makes sense.
But look, I think historically if you look at it, typically if new entrants in markets didn't win by marginally improving on something existing, that happens sometimes, but more likely they jumped up some kind of disruptive technology leap, right? Where it's like we have a different approach, we have different technology. Is that possible here?
I mean, to some degree, maybe simplifying a little bit, but I think part of the reason why the transformer model won was because it runs so incredibly great on GPUs, right? Like recurrent neural networks are similarly performant, it looks, but it runs terribly on a GPU. So did we sort of pick the model for an architecture? And now it's hard to come up with an architecture that, you know, really, well, it's hardware-software co-design, right? Like there's all this hype about neuromorphic computing, right? Like theoretically it's amazing and super efficient. It's like, okay, great. Like there's no ecosystem of hardware, there's no ecosystem of software. It would take like tens of thousands of people who are the best at AI today focusing on that to even prove out if it's worthwhile or not, right? On a hardware side, on a software side, on a model side. And so like you look at like Groq, Cerebras, SambaNova, they all like sort of over-indexed to the models that were leading at the time when they designed their chips and so they made certain trade-offs, right? They put a lot more memory on chip and Nvidia was like, well, we're not going to do that, a lot faster at least, right? Well, more like, if you compare the amount of memory of SRAM on Nvidia's chips, it's much, much lower. Yes. Correct. They went SRAM instead of DRAM, but then they usually have less DRAM. So, there's a trade-off there as well, right? There's less DRAM, there's more SRAM, and because there's more SRAM on the chip, you have to have less compute on the chip. And so, they ended up losing, right? Because the model sizes got too big and all this, right? And so you have this like super weird dynamic where they bet on something that was actually better, right? Like I have no doubt that Cerebras would run certain types of models better than Nvidia or Groq or, hey, Dojo, right? Tesla's Dojo would run certain types of models way better than Nvidia's chips because they're optimized to that. But then it's like, oh, well actually even in vision tasks you use vision transformers now, so it's like, okay, cool. Model sizes grew and all these things. So it ends up being a catch-22 in that like you optimize for something and so now like today you have this new age of AI accelerator companies are like, okay, we're going to optimize for transformers. By the time they started designing, they're like, okay, transformers are dense models that are this big. What's the best, you know, the hidden dimension is 8K and your batch sizes are this big and your sequence lengths are this big. So, let's just make a super large systolic array. So you can, you know, create the maximum efficiency. And that turns out, oh, look at DeepSeek or, you know, go look at what the labs are doing. Actually their shapes are much smaller. Actually you need to do a bunch of small matrix multiplies not massive, massive, massive singular matrix multiplies per layer. And then it ends up, you know, oh well that chip you're designing for that is actually not super effective for that. And so the software is evolving constantly because of what works best on Nvidia. And you see that with, you know, whether it be what DeepSeek's doing or Alibaba's doing or what the labs are doing internally. And you even see this like for Google, right? Like their open source Gemma models make different decisions because the shapes of a TPU are different than a GPU. And those, the GPU and the TPU are actually not that far apart, right? Like you would say yes, they're very different, but like Blackwell and TPUs are very, very, they're converging on similar designs actually. Whereas like to beat Nvidia, you can't just have this supply chain win, right? You don't have this captive customer. So now you need to do something, you know, that will give you 5x advantage, right, in hardware efficiency for a certain type of workload. And then pray the workload doesn't shift, right? Because Nvidia is...
Also optimizing their architecture generation. They've added a lot of stuff to make their chips way better for the existing models, but it's like they're taking large steps every year, every two years towards something, whereas you have to go way over there in left field and hope that models stay over there, right? Because you have to win by 5x because Nvidia is going to have supply chain efficiency over you. They're going to have time to market over you in terms of a new process node or new memory or whatever technology. Right, even AMD, they got to 2 nanometer before Nvidia, they had higher density HBM, they use 3D stacking, all these things on supply chain that should be better than Nvidia and yet they still lose. There's still the software angle, right? Nvidia has fantastic software and then there's software as well, right? But it's like Nvidia's going to have better networking than you, they're going to have better HBM, they're going to have better process node, they're going to come to market faster, they're going to be able to ramp faster, going to have better negotiations with whether it's TSMC or SK Hynix and the memory and silicon side or all the rack people or copper cables, everything. They're going to have better cost efficiency. So, you have to be like 5x better.
But to be fair, if somebody had a viable competitor, which would even be marginally cost competitive, my guess is many of the big consumers of GPUs would immediately shift some revenue there just to have a number two, right? Just to... that's AMD today, right? And Microsoft...
The there's still pretty limited traction though, right? Sure. Meta continues to buy from them and Microsoft did buy a bunch and then they stopped because it's like well yes they're, you know, AMD is giving you all these advantages but it ends up still not being better on a performance per watt basis and they have a way bigger software team. They're somewhat competitive on like all these dynamics that I mentioned, right? So you can't just do the same thing as Nvidia and do it better, right? Or try and execute better like AMD. You have to really leap forward in some other way but that's the design cycle takes so long that models will shift, right? Because they're like, 'Oh, what's the next generation of TPU and GPU look like? Okay, let's optimize for that.' And the research path is, you know, like great, like yes, neuromorphic computing could be the most optimal thing for us to do. But no one's working on that because you have to advance in the tech tree, you've chose it, right? If you restart the tech tree, you're going to be like, 'Well, this sucks.' And so like if it branches this way and you're over here, you're screwed because you have to be 5x. Because the supply chain stuff means that 5x actually turns into a 2.5x and then Nvidia can compress their margin a little bit if you're actually competitive and then that 2.5x becomes like a 50% better and then yeah so it's like it ends up being way too difficult. And the software stuff, right, everything takes your 5x and makes it like oh you're actually only 50% better. And defense supply chain for sure.