當SemiAnalysis的Dylan Patel遇上AI硬體軟體協同設計，人工智慧真的能獲得100倍增長嗎？

I think it's really fun inside of SemiAnalysis because we have 90 people and like a big chunk of them are technologists, engineers across the whole supply chain. And then a big chunk is people who are formerly at hedge funds and you see these arguments like people are like, 'Oh well that doesn't matter' and it's like then someone's like, 'Well but cost' and then the engineers are like, 'No no no but this technology is the coolest.' You see this organically like fight it out and we're pretty informal and you know given the fact that I was a forum moderator, you can imagine what the enjoying it is.

Exactly.

It's as accurate as the information is, yeah. You know, you never know.

Well, when I was a young boy coming out of the womb... So okay, so I grew up in like a small business. My parents had a motel, we lived in the motel, we had our gas station. So you know, I was selling, you know, I joke a lot of times the first neural network I trained was racially and visually profiling people based on when they entered the gas station which cigarette to pick. Basically, you know, the cigarettes were all extruded across the top and I was too short to actually like reach them and technically it wasn't legal to sell cigarettes at that age, but whatever. I had to move the step stool over to the right area.

Well, I didn't get paid, right? It's a family business. Yeah, we had our motel and then across the street was our gas station. So, you know, sometimes someone would walk in and so like if an old white lady with curly hair walked in, I'd move the ladder or the step stool over to where the Camels are. And different age, demographic, profession, race, etc., I would move the step stool over and I joke this is the first neural network I trained because if I waited for them to tell me, I'd have to like move it over and then I'd step up versus like just being ready. So, you know, menthols versus 100 slims and all these things. You know, I think I joke that's the first neural network I trained, but I grew up in family businesses. Lived in a motel. And it all really goes back to when I was like, you know, it was my 8th birthday. My birthday is in May. And it was April when the Xbox 360 was announced. For my birthday, I didn't ask for the Xbox or I didn't ask for a birthday gift. My parents asked what I wanted. I asked for it for Christmas. We celebrated Christmas, but there was no way, at least at the time, I thought there was no way they would give me the Xbox 360 for Christmas, and so I asked for my birthday for Christmas. Anyways, Christmas comes around, I get it. You know, fast forward a couple months, my cousin who lives in Alabama, they also lived in a motel, was going to come over for spring break, for his spring break and we're going to hang out at my house and he's in between me and my older brother in age. Brother's a bit more jockey, so he didn't really care too much about the Xbox. He played sometimes, but didn't really care. But my cousin, you know, I wanted him to think I was cool, right? So I bragged many times on the phone. I was like, 'Yeah, I got an Xbox.' And then the Xbox broke. There was a hardware defect called the red ring of death. But long story short, I had to open it up and short the temperature sensor and it fixed it. But there was many other tricks I tried first and none of them worked. And so that's sort of how I got into hardware. It was like opening Pandora's box. By the time I was 12, I was like on these forums a lot, reading, posting a lot. And this is around the time when Reddit ate all other forums. And so I became a moderator of Android and Apple and Google as well as hardware and was looking at Intel, Nvidia and AMD and all these other forums, right? I was building a PC. All these forums I was watching, reading, posting a lot. But some of them I was moderating a lot. And so, you know, smartphones, watching smartphones develop from very simple to speed racing to being technologically more advanced than PCs in many ways architecturally and same with all the GPUs, just tracking and watching and reading every comment. Always having the economic tinge because I grew up in a small business. So, I was always looking at the economics, right? There was a time where all the neckbeards on the internet loved AMD GPUs and I personally had bought an AMD GPU too because price performance but then when it came down to what's technically better I'd always be like no no no Nvidia's better because they use a smaller chip to get better performance at better power efficiencies and their margins better and so I would always talk about how Nvidia's margins were better than AMD's in the GPU landscape and so it's very fun.

I started moderating when I was 12, but this is all through my teenage tween age and high school years, right?

I played a ton of StarCraft. At one point I was Grandmaster on the North American ladder. StarCraft 2.

Yeah, I mean obsession is good.

They were decent. I would say like I had mostly A's, but there's classes that I thought were really boring or, you know, I just didn't enjoy. Like Spanish I got like not the greatest grades. You know, but it was like I speak fluent Spanish by the way, so it's really dumb. But like it's just sort of...

I didn't learn Spanish till later to be fair. But yeah, so sort of my grades were fine, right? Like I mean they were fine enough for Asian parents. I was better than most of school but you know it wasn't like tryhard maxing for all A's.

Yeah so I went to school I got a few degrees in stuff that wasn't related to semiconductors. Was a quant for two years at a small quant risk firm. And then basically, you know, there was a culmination of events that happened, right? One was that I got screwed out of a bonus. I'd made my company many millions of revenue of risk-free revenue because I exploited a risk thing in the market. You know, I think well over 10 million and then someone else took credit for my work and all this sort of stuff. But eventually I did get rightsized. But you know, I lost a social contract with the company I was working with. Had some, you know, my grandparents grew up in my house with us, right? Or in the motel with us. They lived with us and so very close with them and my grandmother got dementia and she had forgotten who I was and she fell down some stairs and had like a tragic accident and passed away. So all of that happened in early 2020. Additionally there were some girl things and so there's a few things that happened that made me kind of very sad. And so all of those things sort of culminated. Then COVID happened and my brother's like dude just come stay with me. He lived in Nashville so I came and stayed with him in Nashville. We were like, 'Oh, lockdowns will be a few weeks. You can stay with me while they happen and then you can go back home and whatever.' Famous last words. Lockdowns lasted much longer. But, you know, living with my brother for a few months was like sort of like, 'Okay, didn't know what I was doing. I was now at my brother's home. Everything was his rules.' You know, him and his fiance at the time, now wife, were like there. And so, like I basically had to tiptoe around, but I didn't care about my job. And so, I was like posting even more than normal. I'd always been posting a lot on the internet. I'd always been trading stocks a lot, but like I made a lot of money shorting COVID and longing COVID and all this stuff. Semiconductor shortages happened around then too. And anyways, I was very much obsessed with posting and things like that. And eventually around that time I got into an argument with someone on the internet and they doxed me, right? They publicly revealed my identity for my anonymous account. And at the time I was like, 'Oh no, I'm scared.' I stopped posting for like three weeks and I was like, 'What am I doing? Why do I care?' So then I just started posting under, I had like blogs and stuff as well. I made a real blog, SemiAnalysis, and on my 24th birthday, I posted two blogs and then from there it just like it was not a newsletter, but I got so much traction because now instead of posting on an anonymous name as a real name and I put a lot more effort into those two posts than I usually did instead of like posting on the internet, it was like real effort into the blog. You can actually go back and read those if you want. They're not that great, but you know, they were good for the time. They were the best stuff you could find on the internet about semis. And I just kept posting, posting, posting. I started getting a lot of consulting business. You know, 2020, I also sort of I was again crashing out. Didn't know what I wanted to do. So, I packed everything up or I took my truck, I bought a tent that fits on the back of the truck, bought an air mattress, whatever, and would drive around all these national parks all around America. And so, like two or three or four days of the week, I'd stay in a random motel where I negotiated the price to be like $30 a night for a room. And I would work on something else. And then the weekends I'd read books and oftentimes read textbooks while in some random national park or hiking and listen to audiobooks about semiconductors, about AI, about all the things that I cared a lot about and got way more educated over these six months where I'm just going to every national park. And the whole time I was alone, the whole time I was posting blogs. Everyone was like, 'Dylan, what the are you doing?'

Pre-Starlink. Yeah, so it was like very much like what are you doing? I traveled around Latin America again for a year initially with my friend and then with my ex for about a year and then end of '21, '22, '23, and '24 I'm completely, I'm still completely homeless since mid-2020, right? But I'm traveling around to every conference in the world. I go to 40 plus conferences a year no matter where in the supply chain it is. I'm like oh that looks interesting. I guess I'll go to that. And I went to one conference like wow this is amazing. You get to talk to the experts and they're gonna talk to you because and then you're so excited and in the case of semiconductors everyone's a boomer so it's great to like they don't see young people who are excited about it so they're really happy to tell stuff and so you just have to ask.

I think the trade shows and conferences range really widely. Obviously some of the ones I have the most fun at include NeurIPS. So why is that? Because it's 20,000 AI researchers and they're generally in my distribution of age range. So it's a lot of fun but they're also leading AI researchers and you learn a lot. There's also a lot of parties and then it ranges all the way to like there's a random chemical conference in Japan where it's 300 Japanese dudes. It's like 20 guys from ASML, 20 guys from TSMC, 20 guys from Intel, and those are the only people who speak English. Everyone else speaks only Japanese and you're like, 'Huh, I guess there it's still pretty interesting and fun.' I think one thing that I have as a skill set is like I'm able to bond with anyone regardless of their background and who they are. I'm able to talk to them, find something interesting to talk about. Oftentimes it's the tech stuff, but you know, and so I think the most interesting conferences are oftentimes the really big ones because that's where the biggest stuff is happening. But I think the niches that are really really exciting is like SPIE. So there's IEEE which is International Electrical Engineering something and there's SPIE which is another ecosystem. SPIE conferences are super super deep in details. Every single one that I went to, especially like SPIE Advanced Lithography or SPIE Photomask, I went to them the first time I didn't even understand 90% of what I heard and then I read, read, read, I made some context of course and then next time I went I understood like half of what I went to. Third time I went, I understood like 75% of what I went to. Even now I went and I was like, I still don't understand everything that's going on. Whereas like you go to NeurIPS a couple times you can understand, okay, what's neurosymbolic reasoning? Okay, what's this? What's that? You can kind of get a mapping of what everything is pretty quickly. But some parts of the supply chain are so arcane and so deep and so technical, it takes a lot of times for you to even understand what's happening. And for every research paper doesn't necessarily mean you go to a conference for a few reasons, right? You understand the research, but like it's all the research that's being published but what you really care about is understanding how does that research intersect with technology, also how does that research differ from what's there today and none of these research papers tell you what's happening today but then you just ask people and you build contacts and you learn and then you learn about the supply chain and oh this company supplies this company even though it's not publicly stated anywhere or you learn that this chemical costs about this much and a tool uses about this much and you just learn.

My favorite one is I learned a Japanese guy at that specific Japanese conference that I went to where almost no one spoke English, in very broken English he told me about how his father worked in this industry in the 1980s that the only factory in the world that built this chemical burned down and that caused memory prices to like double or triple. I was like, 'Wow, not too different from today.'

I mean obviously use of tokens is going to be the biggest market. Yeah. And the value that's created from tokens is going to be the biggest market, but I think tokenomics, sort of the use of tokens, adoption of AI sort of is the most important thing that's happening. And inference whether it's open models or closed models will be like one of the biggest markets in the world much bigger than oil I think much bigger than many other parts like inference of AI will be many percentage points of the GDP.

Yeah. So, to zoom back, right, like SemiAnalysis, we do a lot of stuff that's like, a lot of it is like research for institutional clients and our subscription first products, but a lot of it is also like, hey, you know, this would just be cool to figure out. Let's figure out how to figure it out and just post it publicly. And that gets more and more scale. And so, we've done this with a lot of GPU benchmarking and testing and training performance and inference performance. But, you know, ultimately we saw like inference benchmarking was like point in time. You know, you test it and you take some time and you release it and it's like slow and arcane and outdated because models change all the time. I feel like every week there's a new model whether it's a Chinese model or today Mythos 5, Fable dropped and new models are coming out all the time. On the software layer, PyTorch, vLLM, SGLang, new drivers, new something drops, you know, in fact the update cycle for most of these libraries is twice a week. So you basically have the software updating all the time and therefore performance changing. New inference optimizations are coming out and those get updated and so I feel like it's a relentless breakthrough after breakthrough after breakthrough that keeps driving efficiency and cost down which is why we've seen model costs drop for equivalent quality by like 60x a year. It's incredible. But to stay on top of that you can't have point in time benchmarking. You need to have benchmarks be living and breathing, i.e. constantly running on the latest hardware on the latest models. And so we embarked on a project and we got a lot of buy-in from the ecosystem. This was only possible because we had enough aura with some of the ecosystem where we're able to get CoreWeave and Crusoe and Nebius and Oracle and Microsoft and Amazon and Google and OpenAI to contribute compute to us and then we were able to work with SGLang and vLLM and now Radix and Interact which are the private companies who are sort of leading those efforts, the open source efforts, to collaborate with us. So, we were able to get Nvidia and AMD and Google and Amazon now because we're adding TPUs and Trainium to collaborate. Now, we've got all these people collaborating. We've got over $50 million of hardware donated to us. Once we launch TPUs and Trainium, it'll actually be over hundred million dollars of hardware. You know, maybe about like 15 different chip types, all running these benchmarks every single day on all the latest models, right? The best model from Moonshot, the best model from Alibaba, the best model from, there's about five different Chinese models, the best open source models, the best Chinese labs there. We run benchmarks on their models every day and then also the best US open source models, GPT-OSS, Neotron, etc. So we're running these benchmarks every day in an automated fashion and they run on these servers that are dedicated to us for inference benchmarking and we sweep across so many different configurations and optimization types and then what it creates is and all the results are public and all the configurations are public. So now we have the Pareto optimal curve because a lot of times when people are comparing inference performance they're taking a suboptimal curve or point for someone else and comparing it to their optimal one. It's like, well, yeah, I can make, if I drove a Porsche versus some race car driver, obviously, I'd drive it slower. This is the same thing with inference benchmarking. And so, what we did is we created open-source containers for the optimal points across every point on the interactivity, i.e. how fast is it responding to me versus batch size, i.e. how many users am I simultaneously serving curve? And so now anyone who wants the optimal point can just go to Inference X download it and run that as the optimal point and they can check every day if they want or they can even auto-download the most optimal point for that model and their inference performance will be near peak.

Yeah, I think most things in hardware, infrastructure, model, application layer, everything is downstream of that curve, right? Is it something that needs to be super super fast, super low latency? And I don't really care about the cost, so I make batch size very low and I use techniques like speculative decoding or multi-token prediction heavily and there's so many possible techniques there. Or is it something where actually I'm batch processing a ton of documents and I don't really care about all these things. I don't use these techniques that actually are worse on cost efficiency but help you with speed for an individual user because I just want to pack a bunch of users. I don't care if the document takes all night to process, right? And right now the way we treat AI infrastructure, it's like one size fits all, but over time we're going to get to the point where there's stuff where you have batch workloads or you need instant response and there's the whole curve that's going to matter for users. And so we see this with Anthropic, right? Claude Code fast mode costs way more than regular mode. And same with OpenAI's priority queue thing.

So if I let's say imaginary example, I have a batch size of 100. And I can do 10 tokens per second per user. So in total I'm doing a thousand tokens per second off of that one piece of compute. That's one side of the curve. Super slow 10 tokens per second. Other side is I have 500 tokens per second, but I only have one user. And so maybe 250 tokens per second one user and then there's points in the middle that are more Pareto optimal, right? The average person actually wants like 50 or 100 tokens a second and maybe the number of users I can batch together so the curve is okay a thousand tokens total per second or 250 tokens total per second depending on how many users I batch and there's a curve in the middle and so ultimately some workloads will actually want the 4x cost decrease because the same unit of hardware can do a thousand versus 250 and some users I'll pay 4x because I don't care about the price. I care about time because the person using the tokens is expensive or the feedback loop that I have here is expensive.

This is a tough one. So I think the non-consensus or at least against SpaceX thing, you know, I love SpaceX by the way and I would totally buy the IPO if I could buy stocks. Not investment advice. Thank you. Not investment advice. From either, I don't think that space data centers will really matter in the next three to five years. With that said, I think in 20 years, I think the vast majority of compute will be going in space. And so the real factor there sort of what's the cost?

And I think obviously my views of where inference, you know, how many gigawatts or terawatts are devoted to inference is it's a crazy curve for me personally.

Yeah I think by 2030 just OpenAI and Anthropic will have over 100 gigawatts combined and then you'll add Meta and Google and so on and so on and so forth. It's a humongous amount of compute that will be dedicated to inference and by like 2040 it'll be terawatts, right? The curve of like productivity that we're going to get and so inference deployments is going to be huge and so if you look at like 2040 I think probably more than half of the incremental compute will be going in space but if you look at 2030 I think it's sub 1%.

Yeah, I think it often depends on what you're doing too, right? Like a TI-84 is way more intelligence per watt in terms of doing math than us. That's like 30 years old, right? So obviously this is like a dumb, sort of...

Yeah. But general intelligence wise. So one of the things Inference X does is we also measure the power and cost of all of this hardware. And so we offer not just throughput versus interactivity, we offer cost versus interactivity, we offer power versus interactivity. And so as far as has intelligence per watt been increasing? I mentioned it's been a 60x cost decrease for same benchmark level. We've also seen the same on intelligence per watt. It's not been exactly 60x, it's been closer to like 40x. Some of the efficiencies are non-power ways but there's been a humongous improvement in intelligence per watt on an annual basis at least so far this year, last year, year before. And I expect that to continue. As far as where we are from the human brain, we're many orders of magnitude away. Thankfully, doesn't really matter. We can devote a lot of power to computers. Much easier to power computers than human brains. Like, you know, we have sickness, disease, and like food preferences, sleep.

Yeah. Yeah. Sean I completely disagree with you by the way.

Okay. So I think one way is to look at as these three different layers. And in that sense like okay from Hopper to Blackwell which is all we've had over the last three years roughly 30x improvement on DeepSeek on the most optimized deployment which is you can see on Inference X there's about a 30x improvement. But over the last three years we've had way more improvement intelligence per watt. A lot of that coming from the model layer, right? If you look back three years, it's GPT-4. Now it's like maybe like Qwen, one of the smaller Qwen models that's like 27B parameters total and like two billion active is like way better. And so you've got this huge improvement on model, you've got this pretty sizable improvement on hardware, but it's that co-design layer and I think that's what's important, right? If you look at the architecture of any of these models, but DeepSeek is the most famous one at least that's public and people have seen.

Yes, I think it's kernels of course, but it's actually you build the hardware architecture for the chip. So if you look at the shapes of all the experts in DeepSeek V3, they were all optimized for Hopper. And if you look at for V4, they're optimized for Blackwell and Huawei's chip. And what's interesting is despite the fact that TPUs are objectively an amazing chip, you know, and they run all of DeepMind and they do all the training for Anthropic as well on the pre-training side at least, TPUs suck at running DeepSeek but they are really really great at running other kinds of models that don't run well on Nvidia. There is some level of such deep optimization that has been done whether it be shapes, network IO patterns, how you do the collectives, how you do things around the arithmetic intensity of the attention mechanism, all these different things are co-optimized between the model and the hardware and the infra software in between and it's hard to say you can disentangle the gains.

I don't necessarily think so. I think it's more so that the west doesn't tell people what they do, right? Like OpenAI didn't tell people that GPT-4o was how sparse it was, what the shape size was, all these things but GPT-4o is roughly the same size slightly smaller than DeepSeek V3 and 4o came out a little bit earlier, right? If I recall correctly.

I would say there's been more gains on the model layer than on the software infrastructure layer and the hardware layer. But there's been innovations on every layer and really the biggest gain and the beauty of the best labs is when they co-optimize all three, you know, and that's what like when Anthropic is, even though they use many different kinds of hardware, they don't really inference too much on TPUs. They mostly train on TPUs. And they inference a lot on Trainium and GPUs and GPU is more the jack of all trades, but they've optimized their hardware. They optimize their model. They've optimized everything so they can do that. Whereas OpenAI, their prior models were optimized for Hopper more. Now they're more optimized for Blackwell. And you step forward through time, these labs and the same with Google, right? They've optimized, Gemini 2 was really optimized for the TPU V6E, Gemini 3 was, and then the next Gemini that's coming out is really optimized for TPU V7. And so sort of a lot of these things are being co-optimized and actually when you pull that model and run it on the old hardware, it's really not that great. And so I think a lot of this co-optimization is the most important thing. It's called software hardware co-design and that's what's really exciting about sort of what I think my day-to-day is like great you get to look at one layer there's all these innovations happening here there's all these innovations happening on every layer but the real breakthrough innovation is when you leapfrog a few layers you co-optimize and co-design them and now all of a sudden you've taken what could have been a 2x here 2x here 2x here and instead of being multiplicative to 8x it's actually 100x because you've optimized across all three layers and so that's what's really exciting about sort of what you see at the labs which you see at a company like Nvidia who's not co-optimizing on the model layer per se but a little bit from the model layer all the way downstream to silicon or you look at a company like TSMC they're co-optimizing not just fabrication but all the way from the components and the consumables and the tools all the way upstream to what the designs their chips are the customers are telling them is this co-optimization across many layers of the abstraction stack.

And band-aids to act.

So memory is an easy one that everyone's talked about, but I'm not going to talk about from a supply chain angle. I'm talking about from a technology angle, right? Memory capacity and bandwidth have been improving very slowly. The NAND cell was invented like 25 years ago. The DRAM cell was invented like 40 years ago and there's been no major breakthrough in cell like what a NAND cell is. Obviously NAND is like a very simple gate or a DRAM cell. There is stuff that could come down the pipeline that could be hugely innovative. But even over the last five years, all we've really done is make the HBM more stacks faster, but actually there's like new innovations coming in the next few years where instead of stacking the HBM separately from the chip, you stack the memory directly on the chip and that makes your bandwidth explode. And so there's interesting companies in that space and interesting plays that companies are trying to do there. I think like memory bandwidth is one of the biggest. Another one is for the history of silicon basically for the last two decades at least how many watts a chip is can be easily predicted just by looking at it for a data center or desktop chip it peaks up at one watt per millimeter squared and so if a chip is 100 millimeter squared generally the power consumption is around 100 or a little bit less and if you look at the newest Nvidia silicon, the newest TPU silicon it's still on that range of one watt per millimeter squared so chips are now getting to 1400 watts. Next generation is 2,000 watts for Nvidia. With Rubin and such, and you move forward to Rubin Ultra, it's going to be like 4,000 watts or something like that. But really, there's increasing the amount of silicon. What's exciting is we're now finally doing things and it's in development right now where you actually can pump the amount of power into the silicon to be way more than one watt per millimeter squared. And now that all of a sudden means you need less silicon. Obviously, it's running at higher power. It's less efficient in some cases, but you reduce the amount of silicon and you're able to like overcome...

Thermal issues. There's interference of like electrical interference issues. There's all sorts of different issues that crop up and that's why it's a hard engineering problem. That's why we've stuck at about one. But what's exciting is the world is trying to change these things. I think interesting like in a different part of the supply chain is sort of like people will talk about like energy is hard and we have energy bottlenecks and it's like yeah but there's actually like very simple solutions one could think of, right? Take the millions of diesel engines for trucks that the US has the capacity to make, you can very trivially convert them to be using gas in the assembly line and then stick them up to an electrical motor like back driving it so the electrical motor generates electricity rather than the electrical motor causing the rotation of the wheel for example, but doing it the opposite direction. And now you've generated electricity by pumping gas into something that US can make millions of. And then okay, well that sounds like a pain in the ass to service, right? Because now you have to have hundreds of these on a data center site. Well, actually, you can just pull people out of car mechanic shops and have them run around and repair truck engines. Actually, it's actually pretty trivial to not, I don't want to say it's trivial, I couldn't do it.

Why would you want to go work in hardware when you can make ads to ads?

I think like everyone wants to pick one or the other for this but it's really like a function of like look you look two years from now Google's going to make 10 plus million TPUs through their supply chain and Nvidia is going to make many more million, tens of millions of GPUs and both are going to be 100 plus billion dollar, well Google's going to be 100 plus billion dollars of TPU created a year and Nvidia will be 500 plus or whatever I'm not making a specific estimate.

Or research. Even media trans absolutely, you know getting ready for the SpaceX IPO.

Awesome, awesome. So I would say the case of sort of like Google TPUs versus Nvidia GPUs they both have points that are really in their favor, right? Nvidia will be like oh well we have switches and we're general purpose and TPUs will be like well we're more optimized we're actually more energy efficient and our network is actually more optimized for certain types of network architectures. And so you have these counterpoints that both would really get into and I could with a straight face argue with you like that GPUs are way better than TPUs or TPUs are way better than GPUs but it comes down to hardware software co-design. So actually the way OpenAI's models are headed it would be a terrible decision for them to use TPUs potentially and the way that Anthropic and Google's models are headed it's actually a terrible decision potentially for them to train with GPUs. I mean it'd be fine for them to...

There's various things right like the size of the matrix multiply unit is different as a very simple thing and therefore the shape of the matrix multiply you do, the attention mechanism you use, the way that attention mechanism is structured...

I think they have quite different model architectures in fact. Interesting, you know, OpenAI's are much more sparse and that has benefits, and then Anthropic's are, you know, they're still sparse but more dense in general and that has different benefits. And there's many other things, right? The network topology, right? Nvidia, all of their chips are connected to switches, NVLink switches. For Google, they have no switch. But what they've done is they've been able to, you know, Nvidia, the NVLink can only connect 72 GPUs. For Google, their ICI can connect 8,000 chips at super high bandwidth, but you have to pass through other chips to get there because there's no switch. And so, there's like there's trade-offs there. There's positives and negatives, and that influences the model architecture. It's not necessarily that you should, you know, claim one is better than the other because at the end of the day how do you say that this is better than that when you can't measure them in isolation because it also extends up to the model layer, right?

Yeah. I mean, certainly the CUDA moat and software moat is at least partially disentangled because, you know, models are just great at coding and all software gets commoditized in that case. I do think there is some level of like open-source and, you know, what people call the CUDA moat is not actually anything to do with CUDA but it's like the fact that DeepSeek, Kimi, and Zhipu, and Alibaba, and Tencent, all these companies, Xiaomi had an awesome model recently, their models are co-designed for GPUs and therefore if I want to run them on TPUs, actually in some cases they don't run really well on TPUs. Now Google just has to create their own open source model ecosystem or open source models themselves, so they have the Gemma models. And so you end up with like, well, that's not really CUDA as a moat, it's that the downstream product is more optimized for Nvidia and in these cases these companies are just open sourcing them or like Nemotron is just open sourcing it. And then the users of it, for example, the open, you know, the inference API providers, the RL companies that are trying to take open models and customize them for companies' business use cases, all these different companies are downstream of the fact that like, okay, well, I guess I need to use Nvidia because the ecosystem uses Nvidia even though I don't particularly care about writing CUDA kernels because the models are great at that. But it's like the shape of like, well, this expert, the demux is this and, you know, the hidden dimension blah blah blah is this, right? And so therefore it's better to run on Nvidia GPUs than it is on TPUs and vice versa, right? If Google were to actually open source really good models, you know, this would be the same thing, right? People would take their models and they'd be like, oh wow, these don't run that well on Nvidia GPUs, I should actually just rent TPUs and or buy TPUs and do it on there. For small teams, you're going to want to use all the open source software like vLLM, SGLang, PyTorch, all that stuff. The big labs, they don't necessarily need to use all that, right? Open forked PyTorch long ago and, you know, Anthropic and all these other people don't necessarily rely heavily on the open source implementation of, you know, these things. They forked things or built it on their own already and so they don't need to rely on the open source. And therefore now it's more like, you know, I'll choose the best hardware and I'll co-design my model and infrastructure software through and through for that hardware that is the best and most cost-efficient. And, you know, I'll AI help me write all that software.