Oh wait, 200 gig? Yeah, we can do that in copper. No problem. And then he also drops the bombshell saying, "Oh, by the way, we can also do 400 gig sis," which is like years away. Like we are not even talking about it right now.
Welcome to another Semi-Dope podcast. I'm Austin Lyons of Chipstrat and with me is Vik Sekar from Vik's Newsletter. So, uh, Vik, a lot to talk about today. Let's quick plug our newsletters and then we'll get into it. Um, for listeners who don't know, we write Substack newsletters, hit on some interesting topics this week. Vik, uh, you wrote about decode hardware for inference and specifically how Groq's LPUs could fit into the mix for Nvidia. Um, I thought it was pretty levelheaded. You know, there's a lot of people on Twitter kind of going crazy with all the possibilities of what Nvidia could do with Groq. And the way I read your article was like, guys, it's only been two months. Uh, there's probably not going to be some sort of crazy engineering announcements at GTC because literally it's been two months. Um, but yeah, what maybe a really quick like high-level take on your side?
Yeah. So I wasn't going to much write about this at all, but somebody on X suggested like, what do you think like they're going to announce in GTC? Like do you have any ideas of what will happen uh with their acquisition of Groq? What are they going to do with that chip? How is it going to change things? And then I started looking at it and then one thing led to another and I had so many things to talk about in the article. Um, and so basically the idea is like, yeah, they got this chip called the LPU from Groq uh which is like a Christmas Eve announcement last year and then they uh are it uses this unique architecture that is very deterministic. We spoke about it on our very first podcast episode uh and so if anybody wants to go back and listen to that I highly recommend it. Uh, so yeah, all the details are in that episode but basically what can Nvidia do at this point? It seems like they can put all those chips into a rack and stamp it something like Nvidia branded. People think it's going to be called the LPX. Might as well be, but if it's called something else, whatever. Uh, but yeah, basically I think it creates uh the missing piece of the puzzle that Nvidia has been building up to because in last GTC they introduced a Dynamo uh you know inference software platform where uh they basically um that that software handles the disaggregation of prefill and decode. It handles the movement of KV cache, all of that kind of stuff. And then uh late in September I think last year they announced the CPX, the chip for prefill only which uses GDDR and not HBM and uh that because you know prefill is more compute bound and then the missing piece of the puzzle is like you need now decode specific hardware which is almost entirely memory bandwidth bound. It's not compute bound and how do you get the best possible memory bandwidth? You don't go off chip and the only way to not go off chip is to use SRAM which is what the Groq chip does. So this is the missing piece of the puzzle but then I started imagining what other companies uh should be doing or you know what AMD should do, things like that. So yeah, so like it ended up being a fun article to write so I really enjoyed writing it and think it's been doing okay.
Nice. Nice. Uh, this reminds me, so when you talk about uh last GTC and the introduction of Dynamo, that was the start of a shift that's happening where it used the the narrative used to be, oh, GPUs, they're very parallel processors so they're better than CPUs but they're supremely flexible and uh Nvidia has CUDA software libraries for everything so everyone should use GPUs. And and yet and and and when there were companies who came out and said like, "Oh, we're going to build, you know, XPUs or AI accelerators," everyone's like, "No, no, no. Those are those are not flexible enough. What are you doing?" Um, but interestingly, uh, a year ago already, Nvidia basically by introducing Dynamo and saying, "Oh, you should actually put parts of this workload on different hardware, prefill and decode," and then later coming out with a skew for um prefill. Uh, that's actually Nvidia like shifting and saying, wait a minute, actually it makes sense to optimize your systems for particular workloads. Transformer-based inference is here to stay. It's growing. It's massive and it's actually worth optimizing for. And yeah, I I know that um they might not come out and say it so broadly, but that has really kind of ushered in this year, especially the past six months where now all of a sudden we're talking Groq, we're talking Cerebras, Tensilica, Etched, MatX, and all of and and of course uh even just like um NTIA or Maya or AMD's Helios and it's like a proliferation of hardware from other vendors. Of course, some of that's due to supply and demand. Um, but I really think it actually in hindsight all really started to usher in when Nvidia themselves came out and showed Dynamo. So I know conversation for another time, but again what you're writing about just has me reflecting on like we are in a new era where it actually makes sense for inference especially to have hardware tuned to specific requirements and specific workloads.
Yeah. So I wanted to tell you three things about this basically. First thing is that uh yeah GPUs are really not the best thing for all parts of inference because during the decode phase the GPU is sitting idle basically and it's like a big waste of time you know it's like this is not like a one-size-fits-all thing. So clearly that that came uh into this prefill decode disaggregation concept. So that's one thing. The second thing is that uh basically it came about because in a sense like China had all these export restrictions at one point and uh they you know I think Jensen mentioned this in last GTC he said that wait people are using the H800 for prefill which was more memory limited and they were using the H80 for decode which was more compute limited, right? And he was like, wait, what's going on? So it turns out that like a bunch of papers uh before that uh that uh mentioned like it's called um uh anyway I forgot the names of the papers themselves but yeah, DISC was one of them, DISCER is one of them uh and they basically these papers explain how you can use these different topologies to do this so you know it kind of comes about from the whole thing of of this doing it this way.
Fascinating. I I'd never realized that the export controls which created different SKUs where they're trying to clamp down on China's abilities and so therefore you have one skew that you know that maybe with more memory and one skew with less memory um forced the innovation to say how can we best distribute our workloads across these for prefill and decode and I think you know we would get there anyway but it's interesting how geopolitics played a hand in shaping where we are today.
Yeah. And um the last thing I wanted to say about this was that um it is actually a backtracking like you mentioned about how Nvidia has been beating the GPU drum and they've kind of backtracked now into like separate hardware for each kind of thing and then they were also beating the like the GPU better than CPU drum. Now that has also changed around and now everybody is wanting CPUs. So it's all like going backward in time it seems like. Yes, it's um I don't you could call it pendulum or maybe sort of circular like what's old is new. You know, CPUs are dead and now it's like long live CPUs.
Now, I want to actually also mention what you wrote this week because uh like we we spoke about a lot of companies last time and how all these companies are doing like Lumentum and Coherent are making all these lasers and indium phosphide lasers which everybody wants. But what was interesting about what you wrote is basically Broadcom is actually one of the biggest supplier of lasers and I think the reason you don't see it being talked about so much is because they don't sell externally and they use it for their own products and they are so vertically integrated and self-contained that you know other people are not going to have access to that laser source. Is that right?
Yes. Yes. Exactly. Um, and it's a really good point and I frankly didn't even realize it myself for that exact reason until I had someone reach out and say, "Hey, sharpen your pencil on this." Um, and yeah, as I did a bunch of research and wrote about it, yeah, I was shocked. But it makes perfect sense. It's like, okay, Broadcom has great switches, uh, you know, great, uh, transceivers or or, you know, um, all sorts of great digital logic chips. And inside of it, they actually built are fully integrated and not only building the logic which we focus on um and the switching but actually the laser components themselves.
Yeah, it's a good piece. I think people should go read it. I I learned a lot of stuff out of it actually about how Broadcom has all these facilities where they make stuff and you know how they diversify supply chains and Yeah, it's amazing. I liked it a lot. I think it's a good article.
Oh, thank you. Yeah. Uh, you know, it's fun to ramp up in optics as, you know, the world is always changing. There's always more to learn. Um, and it's fun to have an excuse to dive deep and share it here with folks and write about it. Definitely.
Yeah. Speaking of going backwards in time and the world changing, what what did you see the whole thing this week about how it's like all optics optics all over the place and then suddenly, oh yeah, copper is here to stay?
Yes. Yes. So this week, Monday, some big announcements from Nvidia uh investing in Lumentum and Coherent. So So the uh you know, optics bulls are out there uh you know, beating their chest and saying it told you so. It's all optics. Here's uh Nvidia, you know, locking up supply. And then uh last night Broadcom's earnings on on Wednesday um Hock Tan was was very pro-copper and just like hey we have great serdes and we want to use copper as long as possible which is funny because they also make CPO switches. Um but and we'll we'll get into all of this. And then of course now all the copper folks are like uh yo and I think Credo was up big time after hours and Lumentum and Coherent were down and everyone's like see I told you copper still has a leg. So, uh, set the stage like we should talk about that today.
By the way, I'm one of those the the latter guys who was like, I told you copper's here to stay. I told you copper, you know, I never like believe physics wise copper has reached its limits. Anyway, we'll talk more about it, but basically what is happening is this, right? Uh we have had so much news over the earnings calls of all these companies that have reported in the last month or two, Lumentum, Coherent, and everybody's like oh the laser is sold out, indium phosphide is all the rage and we can't make these things fast enough. We have like a 30% back like like you know waiting line, you know people are standing in queue for this stuff and we can't make these things fast enough. The wafers are too tiny, they're you know 3 in but 6 in isn't yielding well. So everybody has been thinking okay wait so optics is here right like definitely right look at the backlog look at the revenues look at the projections of all these companies everything is growing like 100% or more and uh then Nvidia comes along and says oh how about I invest 2 billion uh each for for Lumentum and another two billion for Coherent you know because we love lasers right Nvidia loves lasers too apparently and then in each of these earnings calls these companies went and said like oh yeah we like scale up is now going to go to optics because that's where all the orders are coming from and then when Nvidia invests two billion into each of these things you immediately look at it and go obviously like scaleup is now obviously going to optics right look at Nvidia's money why would they put the money down so anyway that's where the people's mind was until like a couple of days ago where two companies reported earnings like in almost consecutive days I would imagine which is I think Credo followed by Broadcom.
Credo goes, oh no, like copper is going to have a long life still. Like AECs are doing great. Like active electrical cables are doing amazing. So what's the problem here? Why do we need you know we've heard this CPO story for many years. I don't think it's going to come anytime soon. Like you know they've been saying this even in the last earnings call CPO or we've heard that song. We've heard that song into so many different names. You know, this is just another phase, whatever. We've heard this. We've been around the block. And then then Hock Tan shows up and says something like really amazing, like he's like, "Oh, wait. 200 gig? Yeah, we can do that in copper. No problem. 200 gig per per lane translates to 1.6, you know, which is even now just only ramping up." And then he also drops the bombshell saying oh by the way we can also do 400 gig sis which translates to 3.2 ter like generation 3.2t which is like years away like we are not even talking about it right now anyway. So this is how this is where we are right this is the landscape of optics first yay everybody's going to scale up all these optics companies we spoke about the company doesn't even make any money but it's like up like a thousand%. And then then comes like these companies and say yeah yeah yeah copper right so that's where we are.
Yeah, totally. Totally. And you know, I think kind of what you're pointing out is that every company has a different message and framing and no one is saying that there'll never be optics for scale up ever, but it's all about timing and if the optics and and I think that's where uh maybe a lot of investors are investing heavily if they think that the timing is like it is happening right now. Optics for scaleups like it's coming in 2027 all the I should invest in these companies and get like ride the wave. Um but then the copper companies to your point uh Credo you know Bill Brennan I think he had a quote exactly to what you were saying which was like show me the CPO deployments. Oh yeah there are none. Okay, next, you know, uh and and now to be fair, Credo's uh investing in optics for scale up as well. Um but they're they're building out a portfolio and they're staggering it in with respect to time. So it's like, you know, again, it's still like, dude, we're still just deploying like we're literally deploying 800 gig copper still right now, you know, and 1.6 will come and and these things will happen over time. And then yeah, to your point, I think Hock Tan surprised everyone and said like, hey, maybe that timeline when everyone thought uh CPO scale was like 27 or 28 and and maybe it's actually even later into 2030. And so I think the the correct thing to do when interpreting all this information is to think about the incentives in this company's business models. If you're a CPO company or uh you know, any sort of optics company, of course, you're going to say um yes, it's coming. It's immediately coming. And and to be fair, they are talking to customers and customers are probably saying like, "Yeah, we like get ready. We want to do it, right?" So, I'm sure from their point of view and from the people their set of customers they talk to, I'm sure they are working to that end. So, I I I would assume that no one's like wrong per se. And then on the other end, obviously, you know, Credo has customers who are are telling them, "Yeah, we're going to continue to deploy your active electrical cables." And so it is that tough like mental state where I don't think there is like a black and white right or wrong a certain date that everyone agrees on but there's all sorts of moving components moving pieces and really it is all about like the you know five kind of hyperscalers at the end of the day depending on how you count it in their deployments but even they are deploying different technologies in different places at different times. So you just have like all these moving parts. You kind of have to wrap your head around all of it and then zoom out and sort of get a sense for it's really sort of like again an adoption or deployment curve of like and we've been talking about this like the stacking S-curves of you know 800 gig 1.6 and eventually 3.2. So it's kind of knowing like who's deploying what where and at what scale.
Yeah. By the way, by the end of this episode, remind me if I don't say this, but I have a total 4D chess move uh hypothesis conspiracy theory for why Hock Tan is all about copper when he is actually a CPO company guy. So interesting. Okay. Yes. Well, I'll we can only explain that when we go through whatever we have to say because otherwise it will not make any sense. So I think we should first begin with why like Nvidia is investing into these companies. Kind of dig into that a little bit and then we'll hit up a little bit of the copper stuff and then I'll give you my theory.
Okay. Perfect. I just made a note on my side. Vik 4D chess at the end.
Yeah. All right. So should we stop with let's start with Nvidia in their news $4 billion on optics.
Um so you know I think what's worth maybe zooming out. So the question is why would Nvidia invest in in Lumentum and Coherent? In my head I always say light by the way Lumentum's ticker you know. Yeah. Um and I think it's worth reminding people you know if we look at Nvidia's optical supply chain today for their 800 gig and 1.6T transceiver vendors actually um if you look at like the the Grace Blackwell generation the 800 gig transceiver was uh I think my notes say like 40% of the share is actually a company called Inolite which people don't talk about much because you can't really invest in it if you live in America I'm assuming because it's a Chinese company and so um they they are a major supplier of laser for or transceivers for uh Nvidia and I think they have the first 1.6T qualification with Nvidia as well. Um, of course, Lumentum and Coherent also are vendors and suppliers for Nvidia. Um, and then I I I found a note somewhere and I need to uh understand this better, but it seems that Fabernet, the contract manufacturer we talked about uh an episode or two ago. Um, they also build Nvidia designed modules. So maybe you would call that like a white box module or something. Um, and Fabinet is the one who provided most of Nvidia's optics actually. And Fabinet's uh assembly I think happens in like Thailand I think.
Yes. Yes. Exactly. Um so I wanted to quick hit on this to set the framing that maybe something like you know 40 50% of Nvidia's transceiver volume runs through China. A lot of it, yeah, might run through Thailand and and therefore in the big picture of export controls, tariffs, geopolitical risk, that would be part of Nvidia's supply chain that is at risk to export controls.
Actually, for them, it's just like uh they want to have all these purchase commitments. They I mean they they have multi-billion dollar now purchase commitments, right? And what it gives them actually is like future capacity access for when they need it. And and interestingly both these deals are not only multi-year but they are non-exclusive. And Nvidia is kind of known to do this. You know the reason is they don't want to actually acquire this company for two reasons. One nobody wants to deal with antitrust stuff. Yeah. Totally monopoly anyways. Big enough a company for so much trouble. So the second thing is that they want to enable multiple providers like this and then have put them to a price war against each other because this is exactly Nvidia's playbook when it came to HBM because initially um SK Hynix was the dominant HBM 3E supplier and Samsung was struggling with yield. So what Nvidia actually did was they championed Micron as a third supplier. They gave them specs, validation support, all of that qualification. They did all of that stuff. Um, of course they did not invest equity like like uh they did here, but they gave them so much and encouraged them so much to make HBM that now it ended up being a three supplier ecosystem which is much more healthy and better pricing and better dual sourcing so many benefits.
Totally. Totally. Okay. Interesting. So you're saying uh besides the geopolitical angle that I raised one they just want access to capacity and so whoever can supply lasers awesome that's great. Let's talk. But then two the more suppliers to some extent maybe three is the magic number um then the more like benefit to Nvidia in terms of price competition.
Yeah. Yeah, they they are known to do one other thing and they did this with CoWoS basically chip on wafer on substrate uh packaging because when the Hopper series chips were in like high demand what they did was they went and like uh pre-bought a a lot of commitment from TSMC. So Nvidia is like this when they sense they're like hounds you know when they sense a little bottleneck somewhere they immediately go dump repayments and secure supply before anyone else can. So this is standard Nvidia playbook when it comes to this. So what it means here for optics is yeah it's a big demand uh signal.
Yes. Yes. And uh to your point I think you know whenever we see news in Asia taking pictures of Jensen at street market eating noodles or drinking beers with his vendors. It means he's over there locking up capacity. That's usually like and when he's not there eating like fried chicken uh he's always at a Denny's. I saw it on one forum saying, "Why is he always in Denny's?" That's because like Nvidia was started in a Denny's. Right.
Right. Right. Totally. Yeah. Some some good history. I need to visit the original Denny's. Um Oh, yes.
So, okay. So, back to the supply chain or the um geopolitical angle. I noticed that in these announcements, both uh Lumentum and Coherent talked about US manufacturing. Um, Lumentum in fact I think announced for the first time that they were going to use this money or maybe not directly use this money but in this announcement they said that they will be investing in a new US fab and then uh you know Coherent talked about building more US manufacturing buildout. They already have a Sherman Texas fab. Um, but it did seem at least maybe it was just for the administration to read, but it it it very much felt like Nvidia was trying to invest in its partners and they went and talked about US-based manufacturing. So, I think all of these are probably true getting, you know, price competition, supply and uh US-based manufacturing.
Yeah. Uh I would also add a couple of more things to it which is basically CPO because word on the street at least I don't have any specs and I was planning to look into this uh is that Lumentum's 200 gig EMLs or externally modulated lasers are the best in the world you know top of the line. I have no idea how they compare to Broadcom. I would imagine Broadcom's lasers are actually very competitive. I would not like put it past them at all. Except that nobody knows I publicly what those lasers look like. I suppose if they don't release it or sell it to anybody, how do you know, right?
Yeah. Yeah. You just have to buy the transceiver and rip it apart and try to reverse engineer test.
We've done it. Yeah.
But but CPO is different because you don't need that modulated laser. You need a CW or continuous wave in in in that particularly because there is no modulation involved. Uh it's actually simpler to make the difficulty comes from the high power it requires. So both Lumentum and Coherent actually have very competitive CW lasers and you know Lumentum makes it on 3 in and Coherent is I think still ramping 6 in which they will eventually get to. I mean I I don't I don't have a doubt in my mind. So this is a very good source dual source of uh CW lasers for CPO.
Right. So the second thing I the second thing that matters to Nvidia here is optical circuit switches because Lumentum has this R300 which is basically a MEMS based mirror that does the switching and uh Coherent has a slightly different technology which is based on like uh I think uh crystal liquid crystal technology.
No moving parts. Right. Uh so yeah I don't know if which is really the better approach here but anyway it gives uh Nvidia uh an option to integrate one or the other or both because they have now access rights to so many things with this investment right.
Totally okay so that's optics and the early announcements in the week should we jump to Credo.
Yeah yeah Credo Credo first uh and then we'll will hit up uh Broadcom.
Oh yeah, Broadcom. Sure. Totally. Okay, so Credo almost chronological order.
That's funny. Uh they had a monster quarter. Um you know, I think what their revenue was up 52% quarter over quarter, 200% year-over-year. And I will say all of these components companies are basically have revenue risk concentration in a revenue concentration risk in a sense where of course they have like three or four or five big customers. Um that's just part of the nature of the game right now and I think it's fine but what it can mean is like lumpy quarters like oh Meta was ramping hard this quarter so we have a ton of revenue or someone got delayed a tiny bit. But if you zoom out, you know, I think you can get away from the lumpiness looking at year-over-year. And so, you know, when you see 200% year-over-year, that that's pretty good. Uh, obviously, Copper's not dead. Um, their gross margins were good. Uh, you know, almost 70%, pretty not, again, not too shabby for a component supplier. Um, I think they got dinged by Wall Street because they were guiding gross margins down to 64 to 66% which still sounds pretty good to me, but that's kind of a mix shift that they were forecasting as their optics ramps. Um, their margins are going to come down and that's just the nature of it. Um, but I think you know probably the quote and then I'll hand it off to you. The quote that stuck out to me was uh the CEO Bill Brennan was talking about how they've grown 6x in two years and he said few companies and semiconductors have scaled at that pace. I was like yeah totally.
Yeah. And I also liked his other quote I wrote it down is there has been a bit of a signal to noise ratio issue in the market and the noise right now is dominating the signal.
Right. What a what an engineering way to say that everybody is talking trash.
Yes. I love it. I was just going to say the same thing like engineered by heart. SNR. That's great.
