It's a honking big PCB. Like I've never seen a PCB this big. Like usually PCBs are something you can hold in your hand. This thing requires like two people to lift this PCB. Okay. And from what I've seen, this midplane PCB has like 52 layers and is like a work of engineering to be honest to even manufacture a board this complex, you know, is fantastic. And so what this does is...
Welcome to another Semi Doped podcast. I'm Austin Lyons from Chipstrat and with me is Vik Sekar from Vik's Newsletter. So, if you're wondering why does it look like Austin's in a hotel room with a crappy webcam and terrible audio, that's because I'm in a hotel room with a crappy webcam and terrible audio. Coming at you live from San Jose, um, outside of Nvidia's GTC. I'm not too far from the SAP Center where they had the keynote today and also, uh, the conference center where they'll be having a lot of stuff tomorrow. Um, but I will say, Vik, I missed the keynote, so I had to stream it from the plane because we had some weather in Iowa and my flight got canceled yesterday. Um, so Vik and I wanted to come to you people and listeners and give you a keynote debrief and sort of digest what we learned together. And we wanted to get it to you ASAP, sort of like an emergency podcast. Um, and that's why it is 8:50 p.m., uh, Pacific time here. So, it feels like 10:50 p.m., which is past my bedtime. But, uh, Vik, how are you doing over there?
I'm pretty good. You know, usually we record this podcast like your morning time, my evening time, and so I'm kind of used to doing it late at night, but like after you've traveled like all over the place with delays and all of that stuff and then landing up and meeting with all the other analysts and all the things and now you have to record a podcast. So, yeah, I feel for you. It's far worse than what I usually go through. [laughter]
Yeah, totally. So, yeah, for listeners, we had a blizzard in Iowa. It came through Sunday afternoon, so they canceled my flight and then it got colder and colder and the wind was blowing overnight. And so, this morning, everything kept getting pushed back. Um, I think when I left Iowa, it was 7 degrees Fahrenheit, feels like minus 7, and then when I went down to Dallas, and then I went over to San Francisco. And in San Francisco, it was like 78 Fahrenheit when I landed. So, I was like, 'Oh, the feels like temperature is like a difference of 100 degrees Fahrenheit from when I left to when I arrived.'
You know, remember I was joking on an X comment on your tweet basically like how you're like basically one of those frozen shrimp that you take from the freezer and you put in the microwave to thaw it out, but you end up cooking it. So, that's basically Austin today.
Yes. Exactly. Exactly. So yeah, I landed, um, came down to the hotel and then there's... I'm an analyst so I got to go to the analyst relations dinner and there were some Nvidia folks there. Ian Buck was there. Um, Gilad, the head of the networking business was there and a couple other folks. So that was cool to get to mingle with people quick. Uh, had some good California wine. Now I'm plugged in. Got some decaf and Vik and I are going to unpack the keynote with you.
Okay, let's do it. He said so much stuff like, you know, two and a half hours. So I kind of in my time I got up early in the morning and like I was like on Jensen's keynote before I even brushed. So I've been listening to it like the whole thing immediately because yeah I was asleep when he actually did the thing. So yeah he said a whole lot of stuff but I'm wondering if it's really worth like two and a half hours of talking. He had all these like science fiction videos of cool stuff. I know it's nice to see when you're in there probably, but you know, I skip past a lot of science fiction videos.
Yeah. So, you know, I have an interesting take here, which is when I was talking to the analyst relations team, they were saying like [snorts] because Nvidia is so big and they have so many businesses, you know, autonomy, telco, data center, client, gaming that they are trying to be respectful of all of their customers and they want to like talk to each of them. But on the other hand, they're like one of the world's biggest companies and therefore even in three hours they can't talk about everyone. And so I thought it was interesting. You know, for example, the keynote opened up talking like very early on about gaming and it was like a really cool demo of showing how they use generative AI to make their gaming better and then they talked about it for like a minute and then that was that. But, you know, it was probably more than they talked about it at CES. So, I think it was like a nod to the gamers like, 'Hey, we got you covered. We're thinking about... So, on our podcast, too, you know, we do care about consumer stuff once in a while.' Uh, and I think this was cool because of all the videos that I saw that Jensen was playing. I think I liked this the most because...
I don't know, they showed this like before after their deep learning super sampling technology, which is like DLSS.
Yeah. Which is like an AI powered suite that, you know, mixes their...
And it was really good too.
It was awesome. Yeah, I think anybody watching this should at least see the first part of the keynote. So you can see like how, you know, you have these typical video game characters. You can always tell they're video gameish, but what ends up happening after DLSS is like awesome. It's just like a real person playing these games. It's amazing. I'm excited to play games now. I literally want to go buy a gaming, you know, card if it's not a million dollars because of like memory problems. Memory... actually play some games.
Totally. Yeah. Yeah. It went from like kind of blocky faces to just very realistic and like stubble on people's chin and stuff like...
Yeah. You got like strands of gray hair like we do. It's awesome.
Yeah. Totally. Totally. So now then Jensen quickly went past consumer GPUs and ultimately this was a data center conversation and what I liked how he teed up the framing. I liked the framing and the business case talk. So let's start with that. Um, you know, I think Jensen did a really good job of reminding people that there's been sort of like three big sort of step change eras with generative AI. At first it was like the training and post-training era where it was like whoa, GPT-3, GPT-3.5, like this is amazing, this is a thing, it's a chatbot but people were still concerned about hallucinations and whatever so it was like oh this is a cool party trick but you can't trust it. And then of course in like... and that was like late 2022. In 2024, late 2024, um, the o1 model came out and now all of a sudden you could reason and that of course helped solve the hallucinations problem. And then we started to get into test time compute scaling where you're just like dude if this thing can think just let it think for longer at inference time and you'll get better intelligence. Um, so you could start to see we're shifting from the training era where everyone needed Nvidia GPUs for training to inference to like even more inference because now all of a sudden it's going to have real demand because it's trustworthy and it's going to have maybe 10x the tokens because you're going to have it think for longer to make sure it's right. And then Jensen was pointing out like we've now entered honestly truly in 2026, um, like late December 2025 into 2026 this third sort of era which is the agentic AI era. So think like Claude Code and as we've talked about think like OpenClaw and again it's like another 10x, um, you know, another order of magnitude in need of tokens because all of a sudden it's not humans talking to a chatbot, it's humans talking to AI and then AI spawning all this other compute, tool calling, CPUs, you need long context, you need lots of storage. And, um, and actually Ben Thompson even wrote a great, uh, strategy article this morning I think is when he published it saying like hey I used to think this was a bubble and now I'm actually not so sure it's a bubble because with agentic AI it feels truly useful and it truly feels like we're going to need all the compute not just GPUs but CPUs. And so I think Jensen was sort of taking the same similar framing and just saying like guys you have to believe us we're in a new era and it's not that AI was a party trick but especially with OpenClaw like you're seeing the world is changing. Um, so let me pause there. Like what did you think of that framing? Did you think it was on? Does it resonate with your experience?
Yeah, I mean this is why and I wrote that article about CPUs and the agentic AI boom and all that. Uh, it is a real need for inferencing now. And what Jensen pointed out is that inferencing is what will generate revenue and this is why people, you know, AI has become a true revenue generator only because of inference. All this time like people are deploying GPUs for training and all of that. And with the inference problem people initially thought ah, you know, inference is an afterthought but what Jensen said was inference is an incredibly hard problem because of the kinds of spaces it has to address. You need low latency, you need high throughput. It is a big problem and you need long context and you have to work with so many different tools and it has so many different use cases and it has to work on so many different kinds of hardware. So it's actually an incredibly difficult problem that now we are really deploying and he calls it the agentic AI inference inflection and so yeah we are here right now.
Yeah, totally. And okay, so what I liked also about the framing and you made the point starting to call out where Jensen talked about these different tiers. The question is like as you're inferencing and you've got the Pareto curve of throughput on the Y axis. So, you know, how many tokens can you generate at a time versus token speed or interactivity on the X axis and this is at ISO power. Um, and we very early on, you know, with these sort of AI supercomputers first we were in the training area so the talk was about like hey Nvidia has scale up and scale out networking and you can have huge clusters and you can train faster. Then as it started to shift into inference, there was this simple Pareto curve and it's just like, hey, no matter what you're optimizing for, we own the frontier. So if you want the most efficient frontier, it's the cheapest or the most efficient to use Nvidia as you know that's the Pareto frontier. Everyone else is behind us now. And what I really appreciated about Nvidia is acknowledging that there's really one workload that matters at scale. 60% of their customers are CSPs, hyperscalers that are serving AI labs. And it's transformer-based LLM inference at scale. But as we've seen over the last three months, it's not just about throughput or about like tokens per dollar because, um, yes, there are some use cases where you're like, 'Oh, okay. I'm fine with a 70B model. That's good enough and I just want as cheap as possible.' But then there's also use cases where you're like, 'No, no, no. I want Opus 4.5 or 4.6 and so I want the biggest model and oh by the way or I want the fastest model and then especially with coding you're kind of like actually I want the biggest model and the fastest model.' And so Jensen and Nvidia started to break down that Pareto curve and they introduced this concept of tiers of like oh hey maybe you have a free tier and it's very high throughput very low speed like Qwen 3 was the example they gave. You know, a decent sized model. You've got a very small context length and you actually just give it away for free because it's like response to those simple questions for those simple use cases. And this is just like top of the funnel, get you new customers, get them hooked because it's some free inference. Um, but then you want to introduce tiers beyond that where it's like a little bit larger model, it's a little bit faster, it's a little bit longer context length. So maybe you have like a medium tier where now you're charging them $3 per million tokens. And then maybe you have a high and a premium tier which he said that, you know, Hopper could maybe only serve like those lower tiers. And then Grace Blackwell unlocked these like higher tiers where you could have an even... because of the HBM capacity and especially with the NVL72 scale up. Now all of a sudden you could legitimately do inference and do like really big models and have them fast and have like a pretty long context length. And then, you know, finally he transitioned this very nicely when they introduced Groq which we'll get to which is to say well what if 400 tokens per second is not fast enough and you actually want to unlock a thousand tokens per second or longer. And so I'll pause there. I'll get your reactions, but I just want to say like what I liked was this was an implicit acknowledgement of like we're past just GPUs are great for training and GPUs are perfect for inference kind of one-size-fits-all to acknowledging like there are different use cases, there's different user demands from intelligence, context, speed and therefore we are going to start optimizing our system so that you can hit different points and as we'll get into he talked about like maybe you need Groq to unlock some of those certain user experiences.
Yeah. So, I have a couple of things I can say about this. So, first thing is like I like the free tier, you know, the way he described it. You know, everybody uses it. You get a Qwen 3 model or whatever at almost zero cost. This is great because this is like something I can get my mom to use like just use a, you know, don't Google anything. Can you just like use an LLM to get your answer? And it provides a good enough answer. You know, she's not going to be like, 'Oh, go off and launch some coding, you know, task with some database queries.' You know, she impossible. She just wants to know like, you know, basically some kind of a travel plan that she's planning to go next month or whatever. It's a perfect tier for that. I mean, it's going to solve like, you know, it's going to rewrite the basic use case tier for everybody in the world. You know, everybody should be using this for at least basic queries if you're not already. So that out of the way, I think then we can go to the extreme end of Groq, which is what I've been saying to many people who ask me about this is that Groq is at like you say an ultra premium tier. Okay, it's ultra premium because it is not that everybody needs this. There are only a few people who need it and I think Meta initially said that they would like this to be like 10% of their inference needs. In Jensen's talk, he actually said like maybe 25% would be this kind of Groq thing. Okay. So yeah, you got some delta of numbers here. But we can say like bracket it, you know, 10 to 25%. That's all about that's all the ultra fast inferencing will need and maybe it's the, you know, enterprise that, you know, needs this because for them time means money and they are willing to pay for the ultra premium tier and that will accordingly cost an ultra premium token cost which they are willing to pay because their ROI is significantly different from somebody like my mom who's just trying to plan a trip she does not need Groq. So between these two use cases and everything in between you still have a massive need for HBM. It's not going anywhere. Like I've had some investors ask me like what happened to HBM? Like is Groq going to take over the need for HBM? Is the memory pressure going to ease off of HBM? I don't think so at all because for majority of the use cases this is not the technology that will be most useful and this by that I mean the SRAM-based thing. So it addresses a particular market where you like you say what if 4,000 tokens per second per user isn't enough? What if you want thousand, what if you want 10,000, you know, so those are the kinds of questions that like the agentic AI world brings to the table but there still for a majority of use cases I think the HBM based inference systems are here to stay so all of that where Rubin GPUs they are not going anywhere and whatever else anybody else is developing based on HBM systems those are all still going to be needed right so that's where I think yes this chart was really useful in setting the stage on what kind of inference is useful where.
Totally, totally agree. And so should we dive more into like the full Vera Rubin system and talk about all the components including Groq?
Yeah, let's do that. I think what this GTC introduced was the seventh chip. Like the last conference, CES, was actually like the six chips that he showed. But this time it's like you got the Groq LP30 chip that is now part of a rack unit. So you can basically slot in this compute tray with eight Groq chips I believe into one rack unit and you can build a rack of Groq chips which is pretty sweet. And I was closely looking at the picture of that tray actually even has like a CPU and like four DIMM sticks. That's interesting. So it's not just like you... there is a CPU on that tray.
No. So yeah, speaking of that, so I'm going to pull it up on my side and look at it. I don't have the screenshot right in front of me, but yes, there's the Groq chips. There's a CPU. There was a call out of an FPGA, which I thought was interesting and I was like, I wonder what the FPGA is used for. Would love to know more. Um, but I saw Patrick Kennedy from Serve the Home has a really good YouTube channel. He had tweeted like, 'Hey, that looks a lot like an Intel CPU.' And people are like, 'Well, how do you know?' like they never said anything about Intel CPU. And he's like, 'No, I guarantee you it's a... yes, they didn't say anything, but I guarantee it's an Intel Xeon because I've just dealt with a lot of CPUs and it looks a lot like an Intel Xeon CPU the way that they have a heat sink on it and the way that they like kind of physically attach it in.' And actually down at this dinner that I went to right before, he was down there. And so I was asking him about it and he's like, 'Yeah, man. I don't know what it is, but just based on experience, it looks like an Intel CPU.' So, for all the Intel bulls out there, maybe the Groq rack is gonna sell some extra Intel CPUs, but it's probably... I don't remember which version of Xeon he said. It's probably an older version that maybe they don't have enough supply anyway because Intel has some supply issues with their older CPUs. So, we'll see how this all shakes out. It'll be funny if it is actually Intel because then I'm like, didn't you just say that like CPUs are all the rage and you know, even Jensen said, 'Oh, I don't think CPUs were going to be such a big deal, but they are.' And so, he's introduced basically this CPU rack, compute rack, which like has I don't know how many CPUs, four CPUs, two CPUs, I can't remember. Uh, but yeah, there is like... eight CPUs, two or three. Yeah, in a tray maybe in a tray. I don't know. Someone had a great picture, a great tweet and it was like cuz I think there's like eight CPUs and just a ton of memory. It's just like all memory and then someone's like, 'Oh, I see why memory is so expensive right now.' I see. Okay. Yeah. So, whatever how many other CPUs are to a tray basically now you can get these like CPU trays that are based on Vera systems and you can build a rack of CPUs.
It's nothing new. I mean we've had rack of CPUs for years but now the fact that you need a rack of CPUs tells you something about CPU demand.
Totally. So yes for people I don't know if everyone has seen the keynote end to end but basically now the Vera Rubin system is actually a full data center and it has yes it has GPU racks where each GPU tray has GPUs and a headnode CPU but it also has a CPU rack or maybe multiple I can't remember of just Vera CPUs and that is for the agentic AI and orchestration processing the head nodes all about like feeding the GPU and keeping it fed but all of that tool calling and thing you want it to happen on CPUs, but you don't want to have to reach out to a different data center or to the other end of the data center. You still want that CPU rack as close as possible. So, Nvidia introduced the Vera CPU rack. And I had some quotes from Jensen. He said it's extremely high single-threaded performance, incredibly good at data processing, extreme energy efficiency. He said it's the only data center CPU in the world that uses LPDDR5, which is like mobile memory. So therefore, the performance per watt is unrivaled. He made sure to say it's in production. And then to your point, Vik, um, he said, 'We never thought we'd be selling CPU standalone. We are selling a lot of CPU standalone. This is going to be a multibillion dollar business for us.' And for reference, you know, I think Nvidia or AMD and Intel do something in the order of like tens of billions of dollars a year in CPU revenue. So to say you're going to do multi-billion is not too shabby.
Yeah. And I know he likes to play up the Vera thing, but it has good single threaded performance, but its Olympus cores are like there are 88 cores and they have the equivalent of multi-threading. I forgot what their marketing term for it is. But yeah, basically it has like 176 threads. Uh, when you compare that to AMD's Venice dense that is due I think later this year, uh, it has like 512 threads and like extremely good single core performance. I mean so there are some really competitive CPUs out there and I like had like a CPU yellow pages that I created that I compared various data center CPUs according to various metrics including memory. So if he's saying like LPDDR5 is the thing that is unique, I'm going to go back to my yellow pages and compare it because I don't know why it doesn't seem like it to me. I don't know. I feel like they have other things going on.
