Oh wait, 200 gig? Yeah, we can do that in copper. No problem. And then he also drops the bombshell saying, "Oh, by the way, we can also do 400 gig sis, which is like years away." Like we are not even talking about it right now.
Welcome to another Semi-Dope podcast. I'm Austin Lyons of Chipstrat and with me is Vik Sekar from Vik's newsletter. So, Vik, a lot to talk about today. Let's quick plug our newsletters and then we'll get into it. For listeners who don't know, we write Substack newsletters, hit on some interesting topics this week. Vik, you wrote about decode hardware for inference and specifically how Groq's LPUs could fit into the mix for Nvidia. I thought it was pretty levelheaded. You know, there's a lot of people on Twitter kind of going crazy with all the possibilities of what Nvidia could do with Groq. And the way I read your article was like, guys, it's only been two months. There's probably not going to be some sort of crazy engineering announcements at GTC because literally it's been two months. But yeah, what maybe a really quick like high-level take on your side?
Yeah. So I wasn't going to write about this at all, but somebody on X suggested like, what do you think they're going to announce in GTC? Like, do you have any ideas of what will happen with their acquisition of Groq? What are they going to do with that chip? How is it going to change things? And then I started looking at it and then one thing led to another and I had so many things to talk about in the article. And so basically the idea is like, yeah, they got this chip called the LPU from Groq, which is like a Christmas Eve announcement last year, and then it uses this unique architecture that is very deterministic. We spoke about it on our very first podcast episode, and so if anybody wants to go back and listen to that, I highly recommend it. So yeah, all the details are in that episode, but basically what can Nvidia do at this point? It seems like they can put all those chips into a rack and stamp it something like Nvidia branded. People think it's going to be called the LPX. Might as well be, but if it's called something else, whatever. But yeah, basically I think it creates the missing piece of the puzzle that Nvidia has been building up to because in last GTC they introduced a Dynamo, you know, inference software platform where that software handles the disaggregation of prefill and decode. It handles the movement of KV cache, all of that kind of stuff. And then late in September, I think last year, they announced the CPX, the chip for prefill only, which uses GDDR and not HBM, and that because you know prefill is more compute bound, and then the missing piece of the puzzle is like you need now decode specific hardware which is almost entirely memory bandwidth bound. It's not compute bound, and how do you get the best possible memory bandwidth? You don't go off chip, and the only way to not go off chip is to use SRAM, which is what the Groq chip does. So this is the missing piece of the puzzle, but then I started imagining what other companies should be doing or you know what AMD should do, things like that. So yeah, so like it ended up being a fun article to write, so I really enjoyed writing it and think it's been doing okay.
Nice. Nice. This reminds me, so when you talk about last GTC and the introduction of Dynamo, that was the start of a shift that's happening where the narrative used to be, oh, GPUs, they're very parallel processors, so they're better than CPUs, but they're supremely flexible, and Nvidia has CUDA software libraries for everything, so everyone should use GPUs. And when there were companies who came out and said like, "Oh, we're going to build, you know, XPUs or AI accelerators," everyone's like, "No, no, no. Those are not flexible enough. What are you doing?" But interestingly, a year ago already, Nvidia basically by introducing Dynamo and saying, "Oh, you should actually put parts of this workload on different hardware, prefill and decode," and then later coming out with a SKU for prefill, that's actually Nvidia like shifting and saying, wait a minute, actually it makes sense to optimize your systems for particular workloads. Transformer-based inference is here to stay. It's growing. It's massive and it's actually worth optimizing for. And I know that they might not come out and say it so broadly, but that has really kind of ushered in this year, especially the past six months where now all of a sudden we're talking Groq, we're talking Cerebras, Talos, Etched, MatX, and of course even just like NTIA or Maya or AMD's Helios, and it's like a proliferation of hardware from other vendors. Of course, some of that's due to supply and demand. But I really think it actually in hindsight all really started to usher in when Nvidia themselves came out and showed Dynamo. So I know conversation for another time, but again what you're writing about just has me reflecting on like we are in a new era where it actually makes sense for inference especially to have hardware tuned to specific requirements and specific workloads.
Yeah. So I wanted to tell you three things about this basically. First thing is that yeah GPUs are really not the best thing for all parts of inference because during the decode phase the GPU is sitting idle basically and it's like a big waste of time, you know, it's like this is not like a one-size-fits-all thing. So clearly that came into this prefill decode disaggregation concept. So that's one thing. The second thing is that basically it came about because in a sense like China had all these export restrictions at one point and they, you know, I think Jensen mentioned this in last GTC, he said that wait, people are using the H800 for prefill which was more memory limited and they were using the H80 for decode which was more compute limited, right? And he was like, wait, what's going on? So it turns out that like a bunch of papers before that that mentioned like it's called, um, anyway, I forgot the names of the papers themselves, but yeah, DISC was one of them, DISCER is one of them, and they basically these papers explain how you can use these different topologies to do this, so you know it kind of comes about from the whole thing of doing it this way.
Fascinating. I'd never realized that the export controls which created different SKUs where they're trying to clamp down on China's abilities and so therefore you have one SKU that you know that maybe with more memory and one SKU with less memory forced the innovation to say how can we best distribute our workloads across these for prefill and decode. And I think you know we would get there anyway but it's interesting how geopolitics played a hand in shaping where we are today.
Yeah. And the last thing I wanted to say about this was that it is actually a backtracking like you mentioned about how Nvidia has been beating the GPU drum and they've kind of backtracked now into like separate hardware for each kind of thing and then they were also beating the like the GPU better than CPU drum. Now that has also changed around and now everybody is wanting CPUs. So it's all like going backward in time it seems like. Yes, it's, I don't know, you could call it pendulum or maybe sort of circular like what's old is new. You know, CPUs are dead and now it's like long live CPUs. Yeah, for sure.
Now, I want to actually also mention what you wrote this week because like we spoke about a lot of companies last time and how all these companies are doing like Lumentum and Coherent are making all these lasers and indium phosphide lasers which everybody wants. But what was interesting about what you wrote is basically Broadcom is actually one of the biggest supplier of lasers and I think the reason you don't see it being talked about so much is because they don't sell externally and they use it for their own products and they are so vertically integrated and self-contained that you know other people are not going to have access to that laser source. Is that right?
Yes. Yes. Exactly. And it's a really good point and I frankly didn't even realize it myself for that exact reason until I had someone reach out and say, "Hey, sharpen your pencil on this." And yeah, as I did a bunch of research and wrote about it, yeah, I was shocked. But it makes perfect sense. It's like, okay, Broadcom has great switches, you know, great transceivers or all sorts of great digital logic chips. And inside of it, they actually built are fully integrated and not only building the logic which we focus on and the switching but actually the laser components themselves.
Yeah, it's a good piece. I think people should go read it. I learned a lot of stuff out of it actually about how Broadcom has all these facilities where they make stuff and you know how they diversify supply chains and Yeah, it's amazing. I liked it a lot. I think it's a good article.
Oh, thank you. Yeah. You know, it's fun to ramp up in optics as, you know, the world is always changing. There's always more to learn. And it's fun to have an excuse to dive deep and share it here with folks and write about it. Definitely.
Yeah. Speaking of going backwards in time and the world changing, what did you see the whole thing this week about how it's like all optics optics all over the place and then suddenly, oh yeah, copper is here to stay?
Yes. Yes. So this week, Monday, some big announcements from Nvidia investing in Lumentum and Coherent. So the optics bulls are out there beating their chest and saying it told you so. It's all optics. Here's Nvidia, you know, locking up supply. And then last night Broadcom's earnings on Wednesday, Hock Tan was very pro-copper and just like hey we have great serdes and we want to use copper as long as possible which is funny because they also make CPO switches. But and we'll get into all of this. And then of course now all the copper folks are like yo and I think Credo was up big time after hours and Lumentum and Coherent were down and everyone's like see I told you copper still has a leg. So, set the stage like we should talk about that today.
By the way, I'm one of those latter guys who was like, I told you copper's here to stay. I told you copper, you know, I never like believe physics wise copper has reached its limits. Anyway, we'll talk more about it, but basically what is happening is this, right? We have had so much news over the earnings calls of all these companies that have reported in the last month or two, Lumentum, Coherent, and everybody's like oh the laser is sold out, indium phosphide is all the rage and we can't make these things fast enough. We have like a 30% back like you know waiting line, you know people are standing in queue for this stuff and we can't make these things fast enough. The wafers are too tiny, they're you know 3 inch but 6 inch isn't yielding well. So everybody has been thinking okay wait so optics is here right like definitely right look at the backlog look at the revenues look at the projections of all these companies everything is growing like 100% or more and then Nvidia comes along and says oh how about I invest 2 billion each for Lumentum and another 2 billion for Coherent you know because we love lasers right Nvidia loves lasers too apparently and then in each of these earnings calls these companies went and said like oh yeah we like scale up is now going to go to optics because that's where all the orders are coming from and then when Nvidia invests 2 billion into each of these things you immediately look at it and go obviously like scale up is now obviously going to optics right look at Nvidia's money why would they put the money down so anyway that's where the people's mind was until like a couple of days ago where two companies reported earnings like in almost consecutive days I would imagine which is I think Credo followed by Broadcom. Yeah, Credo goes, oh no, like copper is going to have a long life still. Like AECs are doing great. Like active electrical cables are doing amazing. So what's the problem here? Why do we need you know we've heard this CPO story for many years. I don't think it's going to come anytime soon. Like you know they've been saying this even in the last earnings call CPO or we've heard that song. We've heard that song into so many different names. You know, this is just another phase, whatever. We've heard this. We've been around the block. And then Hock Tan shows up and says something like really amazing, like he's like, "Oh, wait. 200 gig? Yeah, we can do that in copper. No problem. 200 gig per lane translates to 1.6, you know, which is even now just only ramping up." And then he also drops the bombshell saying oh by the way we can also do 400 gig sis which translates to 3.2 ter like generation 3.2t which is like years away like we are not even talking about it right now anyway. So this is how this is where we are right this is the landscape of optics first yay everybody's going to scale up all these optics companies we spoke about the company doesn't even make any money but it's like up like a thousand%. And then comes like these companies and say yeah yeah yeah copper right so that's where we are.
Yeah, totally. Totally. And you know, I think kind of what you're pointing out is that every company has a different message and framing and no one is saying that there'll never be optics for scale up ever, but it's all about timing and if the optics and I think that's where maybe a lot of investors are investing heavily if they think that the timing is like it is happening right now. Optics for scale-ups like it's coming in 2027 all the I should invest in these companies and get like ride the wave. But then the copper companies to your point, Credo, you know, Bill Brennan, I think he had a quote exactly to what you were saying which was like show me the CPO deployments. Oh yeah there are none. Okay, next, you know, and now to be fair, Credo's investing in optics for scale up as well. But they're building out a portfolio and they're staggering it in with respect to time. So it's like, you know, again, it's still like, dude, we're still just deploying like we're literally deploying 800 gig copper still right now, you know, and 1.6 will come and these things will happen over time. And then yeah, to your point, I think Hock Tan surprised everyone and said like, hey, maybe that timeline when everyone thought CPO scale was like 27 or 28 and maybe it's actually even later into 2030. And so I think the correct thing to do when interpreting all this information is to think about the incentives in this company's business models. If you're a CPO company or you know, any sort of optics company, of course, you're going to say yes, it's coming. It's immediately coming. And to be fair, they are talking to customers and customers are probably saying like, "Yeah, we like get ready. We want to do it, right?" So, I'm sure from their point of view and from the people their set of customers they talk to, I'm sure they are working to that end. So, I would assume that no one's like wrong per se. And then on the other end, obviously, you know, Credo has customers who are telling them, "Yeah, we're going to continue to deploy your active electrical cables." And so it is that tough like mental state where I don't think there is like a black and white right or wrong a certain date that everyone agrees on but there's all sorts of moving components moving pieces and really it is all about like the you know five kind of hyperscalers at the end of the day depending on how you count it in their deployments but even they are deploying different technologies in different places at different times. So you just have like all these moving parts. You kind of have to wrap your head around all of it and then zoom out and sort of get a sense for it's really sort of like again an adoption or deployment curve of like and we've been talking about this like the stacking S-curves of you know 800 gig 1.6 and eventually 3.2. So it's kind of knowing like who's deploying what where and at what scale.
Yeah. By the way, by the end of this episode, remind me if I don't say this, but I have a total 4D chess move hypothesis conspiracy theory for why Hock Tan is all about copper when he is actually a CPO company guy. So interesting. Okay. Yes. Well, I'll we can only explain that when we go through whatever we have to say because otherwise it will not make any sense. So I think we should first begin with why like Nvidia is investing into these companies. Kind of dig into that a little bit and then we'll hit up a little bit of the copper stuff and then I'll give you my theory.
