EP10: Nvidia vs AMD & the AI Power Crunch — Doug O’Laughlin, SemiAnalysis

Something new under the sun is what we like to say, but we'll say consulting research. I think it's the first full-stack AI semiconductor research firm. That's really how we think about it. But yeah, from the chips to the models to the data centers, we cover it all. That's SemiAnalysis.

I think it did have the impact. I'm not going to say that we're the only reason, right? There's definitely internal and external will have to come together. I wasn't on that call, but the apocryphal story I think was like the day after Christmas because I think we wrote the article the day before Christmas last year and Dylan and the technical team were on with Lisa and in the beginning she kind of was really dismissive like, hey, this is like, oh, whatever, is this true? And essentially Nishant, who was head of Rock and doing a great job, we meet with him all the time on X and he's like, yeah, it's actually all true and we helped bug fix these little gem shapes and all these things just to make a working accelerator. And our takeaway because we spent a lot of time doing and bug fixing and working with AMD is like, dude, yeah, it's probably impossible to inference on it and nothing's ever been trained. And so we wrote this big thing and AMD took it seriously like there was a huge sense of urgency to catch up and yeah, I think obviously that's the outside. Now, the inside is I know some of the jokes are like we have some friends at AMD and they're like, yeah, we hate SemiAnalysis. Now we work 996. And so they definitely spent a lot of effort catching up and I think they've done a really good job and part of it is just like highlighting kind of looking at the truth of what was actually happening versus the story right at the time especially in late December or man was that two years ago like the MI300 story when it first came out was we have better silicon with higher performance and more memory bandwidth and more memory right and you're like well obviously X and Y you would always choose Y you choose the better product but the reality is getting transistors to work is a whole other problem. And it wasn't the AMD story of old, which is like not copying, but doing a better x86 CPU, which had an entire mature library that both AMD and Intel could use. I think AMD thought if we hit parity at a GPU, we're going to have a better GPU. But the reality is they hit parity for the silicon and the transistors and the package, but they didn't hit parity for the software ecosystem. And so that's why the NVIDIA or the CUDA moat is real because you got there with the chip, but you didn't actually get there with the developer ecosystem or a usable chip. And so we've done a lot of work since then. Actually, InferenceMax is our new benchmarking suite of tools that kind of through that show real world performance, not just specs. Like that's something you see over and over. Hey, this one has more flops, this one has memory, it's better, cheaper, whatever. But the real final rubber to the road proof is how many tokens comes out of this GPU at what rate and at what estimated cost. And that's really what we did with the InferenceMax team and product. So yeah.

Okay, so I definitely feel like that story has to start with Dylan Patel. I knew Dylan before he became famous. Actually we were just buddies on Twitter when it was still Twitter, right? So I started a newsletter called Fabricated Knowledge and I got like kind of merged in, joined with Dylan about, I feel like it's a little under two years, a little under two years feels like 10 years ago at this point. And Dylan, dude, Dylan at the time was like, and he's always been this way, he's like a crazy poster who loves technology and will do whatever it takes to get to the truth. He will argue to hell but he's very focused, he's very truth-seeking and that's like an important part of SemiAnalysis. And so, dude, at the time I was working at a hedge fund. Like my background originally is finance. And I wrote my newsletter. I left to go do, I left the first job I had to do this big sabbatical kind of, it was during COVID. I didn't really, you know, my sabbatical started in 2021. I went like this big hike. Did like a thru-hike. It was awesome. But I had some free time on my hands. And so, I just was really passionate about semiconductors and semi stocks. I was like, this industry is completely changing. And the person who I found most compelling to talk to on a day-to-day basis was Dylan. And because he just always knew what was up. And so Dylan, I definitely, it was funny. I think at one point in his life, he wanted to do something else. I was like, dude, get your head in the game. Make a Substack. You need to get started on this. He was originally on WordPress. And so yeah, I think I was there early, knew who he was, have always kind of collaborated, worked with him, talked to him. Our spheres of influence have always been similar. And that's why I chose to join him because like, dude, competing with Dylan is pretty like, he's relentless. He's ruthless. And so good.

Yeah, 100%. But I also think what's important is like, you know, if you want to go far, you want to go with someone else. And so that's the thing we wanted to make this a sustainable awesome research business, right? And I think that SemiAnalysis currently, I think already today we are a world-class AI semiconductor research firm, right? But I think we're just starting this journey. We have a lot more to go and the entire space and ecosystem is pretty hot right now, maybe even too hot, but who knows. But I think the thing that's important is we're both really young motivated people and we want to build something. Right. And so that's what SemiAnalysis is and what it will continue to be. We're really truth-seeking. We actually do research. That's quite crazy, but like you know, you can't pay to be in our Gartner quadrant, right? That's not how our business works. And so, yeah, that's kind of the blurb of SemiAnalysis and how it came to be.

Yeah, it's a great question. I feel like it's one of those things where it feels overnight, but at least when I first was DMing Dylan on Twitter in like 2018, I was like, oh my god, everything's going to change. Like the long-term bet was Moore's law was dead and that because of that, the new technologies of the future would cost a lot more. And so, we had to do like something had to change. And so, a myopic obsession with semiconductors is where we both started. We're like, dude, this is the most important industry in the world. And so, watching 2020 happen, right, and having the chip shortage, that was just a confirmation of what I already knew was true. That chips and hardware are the most important technology that has ever been invented. We essentially made the information in our brains into the physical around us, right? By we're able to program and use physics to manipulate information at will. That is magic as far as I'm concerned. So we always knew it would be so valuable and pretty much compute continued to chug along and then obviously the magical new thing that came in is AI. A lot more demand for compute and now all of a sudden chips goes from something that is kind of just like a boring plumbing behind the scenes that no one even thinks about because of the CPU world to critical front row most important thing. And I mean it feels like it's overnight. But part of it is it's been a longer story to get here and I think by having conviction and always knowing that that was true and positioning ourselves to be there, it felt inevitable. So yeah.

Yeah, so Taiwan's crazy. Because for better, for worse, all the good chips are made there, man. It's kind of like one of those interesting facets of the entire semiconductor industry and the entire semiconductor conversation, right? Is that we outsourced chip manufacturing overseas and we literally lost the keys of the kingdom and no company has ever come back from falling off.

Oh dude, you don't even. Okay, so my favorite, I got this in the Wall Street Journal I thought was really funny. Dude, Andy Grove wrote a book called Only the Paranoids Survive, right? And the entire book is about strategic inflection points that essentially these big things come in your industry that changes everything and it's your job like screw hell or high water. You cannot miss them. Okay. And then Intel proceeds to miss every single strategic point from that point onward, right? They missed mobile and then they missed AI and then they totally like whatever and so there's like this deep irony, right? But yeah, 100% these are like really critical technologies and I think the reality is Intel was not a young company when these new inflections came and it just wasn't nimble enough to capitalize on it. So, yeah, I think that's a huge part of the conversation because let's just say maybe not 100%, 85, 90% with some of like TSMC Arizona, but 95% of the chips that really matter for AI are made in Taiwan. And 85% or 75% by value, I think it's 80% by value is made in Taiwan. Like, these are the chips that are not even part of AI. It's part of our mics, our computers, the phone in my pocket. Every little bit of technology that is thinking or working hard, it's made in Taiwan and we've kind of lost that key technology. So it's really important because what happens is that little island is, you know, if we're talking about like compute being the most magical thing we've ever done, right? That is the source of the magic and obviously it is 100 miles off the coast of China and China really does want that territory back for like a long list of reasons, right? But we're not going to, I don't want to go into that. And then obviously for America, for better, for worse, if Chinese chip production or Taiwanese chip production was to go down, it would be catastrophic like truly our lives would be really impacted and I just don't think people appreciate the concentration. So yeah, the magic of making it.

Man, I could go on this one over and over and over. Look, my favorite analogy is apparently, and you're going to have to fact check me on it, but I think 2,200 blood cells fit inside like a single transistor at 2 nanometers. So, you can't even fathom how small of the scale. And this is the precision that we need, 99.99%. Because if there's like a miscontact of the metal, like your chip won't work. So the precision that we're talking about is just insane. We've never made a more precise smaller technology at scale and it's at massive scale, billions of devices, right? And making that whole thing is thousands of steps, it takes months for it to be done and each one of these steps are like the cutting edge of science fiction or science but it almost is like science fiction, it really is. And so each one of these steps have a special manufacturing process, a special supply chain, a specialty in knowledge, right? And so you have all of these thousands of steps and everyone who is the best at it lives in Taiwan. And so all of the chemical suppliers who are best at supplying chemicals to making semiconductors live in Taiwan. Everything, the entire supply chain, the best builders of clean rooms in the world live in Taiwan, right? There's this ginormous talent density there that we are trying to recreate, but we're having a hard time. And yeah, part of it is like we just do not build things as quickly here. There was like some during the Chips Act, there was some conversations about how much cheaper it is to build abroad, how much quicker, how much subsidies, whatever. But we have lost that magic for whatever reason. And bringing it back is like, you know, when you lose the fire, I feel like bringing it back from embers, it's a lot more effort than people appreciate.

Well, the Chips Act has been completely scrapped as you know and it's mostly gone to Intel. I am cautiously optimistic on Intel for the first time in a long time. I can kind of talk about that if you guys would like to go into that. Look, Intel was a story where there was a true belief that if they built it, customers would come but the reality is they built let's say 80% of the stadium. Okay. And the last 20% is the PDK which is a pain. And they've only ever, it's like building their entire process was built for a customer of one. A one-of-one design process. It is like, oh, you know, if you want to start the car you have to bang the dashboard. All these special little design tricks that only Intel knew because they built and grew it all the way till now.

Yes. For the fabrication. Yeah.

Actually, the story is actually more complicated than you thought. So on the process side was a whole problem too. 10 nanometer, which I guess is seven because it got renamed, was so complicated and had so many little tricks shoved into it, it just got delayed four times, right? And so what happened was all of these delays allowed the competition to catch up. And also the competition was building an open ecosystem when Intel was building a product suited for one. It's like a chef with only one customer and then someone else orders a different meal and they're like, I don't really know how to make that. I've only been making spaghetti and meatballs and you're asking me to make a pizza like sure but I just can't do this. And so that's kind of the problem is that they lost process density meaning the density of the transistors at the same time as another competitor came in with a much broader ecosystem. And then also on top of it on the process side, right, this is like Intel's place in the world not just in a manufacturing way, right? Because they also lost it on the design side. The entire world went the way of accelerated compute. GPUs were the next big thing and CPUs, a generalized compute, was not getting better at a quick enough speed and so because of that they lost their strategic relevance and they also lost their process technology in one go. That's a terrible setup. And so that's kind of the two sides of it. And so I think the most strategically valuable part of the business is the manufacturing portion, right? Like I was saying, manufacturing these things is like science fiction, but they're still kind of within a viewing distance of the leading edge. And so there's a belief that they could probably get their things together with 18A and then 14A, their next generation process. And I think it's at the point where the process is mostly fixed. But the issue now is how do you make it so that like this process that you are selling to your customers is exactly what they're getting, right? You know, it's like a quality control thing where it's like you don't have those idiosyncratic bugs that Intel had designing in this process in their chips for like 20, 40 years, right? And that's kind of their challenge. And so they got there, if they built it they would come, they mostly built it, people weren't really going to come because why would you risk your most important product on an unproven process technology with your former competitor who's been underexecuting for the last 10 years? I wouldn't. And so what happened was the Chips Act got completely rolled into what was a ginormous investment into Intel. Okay. And so the carrot was if they build it will they come. Now welcome the stick. Okay, you got to invest in America. The largest shareholder in Intel is actually the United States government. 10%. Yeah, 10% is the United States government. And I think the need and the force via tariffs, whatever, is a really good counterweight to push customers from the top down to force them to choose Intel. And I think that's unfortunately the right way to do it, right? These relationships get built from the top up. And someone making a big bet in an organization that is not performing doesn't make sense to me. But someone making a big bet with knowing the counterparties with I guess the human capital element where like Trump is saying like you got to do this or else is a big deal and I think that's, I'm the most hopeful I've been on Intel for a long time.

Yeah. Okay. So let me think about this. So I think one of the biggest things, if you guys are familiar with the term, is that GPUs aren't general purpose anymore, right? Or like they do graphics, right? In a lot of ways they've been like since at least the Hopper generation a little bit, but definitely the Blackwell generation. You can argue that Blackwell is a transformer processing unit. It's been very specialized and focused on making the most of transformer throughput. And so there's this belief because if you look at like Bitcoin mining for example, right, the old analogy is that the ASICs will have another 10x speed up. I don't know if they quite have that much architectural speed up. And then also the problem is too GPGPU, which is general purpose GPUs, has flexibility so that if an architecture changes you can still recycle some of the processors, the tensor cores, right? Yeah, the tensor cores to still be productive but it doesn't have to be like a very defined matrix multiplication shape. And so like that might be the problem for example with Etched, right? They're like, hey, we're going to bake a transformer in, right? But what happens if like some kind of clever algorithm trick, state space models, yeah, state space models, there's so many, there's like many, many, many ways this can go that it is not like just the biggest systolic array possible that is like the correct way forward and then all of a sudden their bet ends up becoming brittle. And so that's kind of the thing. You can argue that this is, you can look at history of compute and see kind of a folded in fractal of the same problem. Why did CPUs win? There was actually a lot of ASICs in the beginning, in the very beginning. Why did CPUs win? Because they got so much better every single year and they supported a broad use case, right? And so as long as Nvidia continues to have a killer pace, what happens is you don't need to have these like orders of magnitude increase. You just hold a steady pace until your competitors run out of money and fuel. And that's what's been happening and what we've seen. So a lot of the early ASIC companies, they tape out a first chip. As all first chip tapeouts go, it's delayed. They have to do a respin. And their original thing that was going to be 10 times better than the A100 is now kind of competing with the H100. And then because when it came out, the H100 is like just 6 months away. And then they also don't have a software stack and they don't have like all of these nice-to-have developer tools. And so by the time you get it and you purchase it and you have like a hundred of them, H100s already exist and you have your entire ecosystem completely written out and ready to go. And so that creates a lot of workload inertia. And so that's why like Jensen internally is obsessed with making sure it's backward compatible so that as long as they continue to push it forward, all the work they've done to this point doesn't need to be rewritten and then that can keep the inertia into the GPGPU ecosystem. And so that's kind of the bet. I think there isn't a special skunk works that I'm aware of, but they are pushing the limits of what is possible at a transistor, package, data center level, right? They are dragging forward other people. And I think a lot of the innovation that we've been seeing from Nvidia, ironically, is not at the nanometer transistor magic level that I love and truly appreciate, enjoy, but it's actually at the package level, at the rack scale level, right? Being able to move some of the power. If you think about it, this is just a ginormous distributed problem. And they say it's like it's not just a single chip, it's over a whole data center. One of the things you don't want to do is you want it to be as dense as possible. It's the same problem as Moore's law. The reason why Moore's law works so well for so long is that if you move two transistors closer together, it takes less energy to move between it and it's faster because it's less distance. That same problem is what they're doing with liquid cooling. By moving and densifying these racks by having more MSI, like millions of square inch of scale out on the rack level and then having more liquid cooling, you're able to do what you're effectively doing by moving it all together into a denser and denser rack. And what they're doing right now that they're talking about is we're talking like, I mean mega, I've heard megawatt racks. That's crazy. A megawatt, like you know, a thousand, that's a gigawatt, 6,000. So essentially like if you think about it, 6,000 racks, my favorite analogy I've been using is all of New York City since we're an AI NYC show is like 6 gigawatts.