The scale-up opportunity is a brand new market that didn't exist before.
All right. Hello listeners. Welcome to another Semi Doped podcast. I'm Austin Lyons from Chipstat, and with me is Vik Sekar from Vik's Newsletter. Today we're going to talk to you about the optics and networking supercycle. But first, a quick teaser. Vik, I saw you had a really interesting post today about CPUs for AI. So let's talk about that really quick.
Yeah. So the Substack post today is basically about how CPUs are a big bottleneck when it comes to AI, and it could actually limit how much GPU usage is actually going to happen depending on the kind of workload you have. So if it's highly agentic, that the CPUs need to go off and do a whole lot of things like pull databases, scrape information, that could be significantly different from just like doing something that's GPU heavy like inference. So I basically the article categorizes a whole bunch of CPUs according to their best use cases. So yeah, I had fun writing it.
Nice. Yeah, I bet because honestly lots of people are saying, oh, CPUs are a bottleneck, but I don't feel like anyone's really dove into what kind of CPU and what they're used for, the use cases, and like why they are the bottleneck. So good read. Very good read. And then I'll throw out there, I just posted today about Nvidia's autonomy business strategy. Again, I had fun like Vik really kind of getting in deep and asking a lot of questions that people have been asking me and I had been waiting to write about because I've been writing also about optics and networking, which is very hot. But with that, let's jump into it. So Vik, Silicon Photonics had a big week last week. Why are we talking about it right now?
Yeah. So it seems like optics has become a big thing now because the connectivity bottleneck is a big deal in data centers. And a few months ago, I made a tweet that got way too many likes and retweets than it should have. I said optics is the next memory. You just don't know it yet. So this is like off-the-cuff remark. It was not like highly educated or whatever, but I thought it was funny, but I think a lot of people agreed with it. And now I see a lot more articles and news outlets with that phrase, of their own accord. It's not because of my tweet obviously, but I think the main reason this is coming up to be the case is that we need faster interconnects between GPUs. Copper is running out of speed and distance and reach, and in order to do this we have to go to the optical world, for which we need pretty much everything like lasers, we need foundries to make these photonics chips, we need people to assemble these transceivers. And the problem is that there aren't so many people making these things because typically optics has only been limited to like long-haul communications and maybe a little bit of LAN and things like this. But now we are on like data center scale where you need everything on massive, massive scale, and there are only a handful of companies kind of dominating the space right now. And I'm sure we'll get into all of that, but all of these companies are signing multi-year long-term agreements with customers, getting fully booked out, and all these customers are like prepaying for all laser capacity like even one or two years into the future.
It sounds a lot like memory and HBM.
Yeah, you know, that's the reason I can see a few parallels between this and HBM is because like in HBM there are only a few concentrated suppliers who make lasers and all the optical fiber and things like that. And so structurally because only a few people do it, there is an undersupply. And what that leads to is that like a lot of people can have pricing power. If you're the only game in town, you have the pricing power. And then when you have everybody wanting what only a few people can make, you are in the same situation as HBM is.
Mhm. And I think in both races, the HBM race and like this the connectivity race, scale-up speeds, scale-out speeds, there's always these sort of like inflection opportunities too where a new technology comes out. So HBM3 to HBM4, or you know, 800 gig to 1.6T. It's almost like the foundry race too of like who can get to the next node first. There's just like this appetite where all these hyperscalers are trying to build the next data center and they want it to be on like the fastest scale-out, scale-up networks possible. And so if you're one of the three or four suppliers somewhere in the supply chain and you can be prepared for the next industry leap in speeds first, then you get a temporary advantage essentially. And kind of same in the analogy to HBM there is like yeah, whoever can have validated HBM4 first with say Nvidia where they're qualified and Nvidia feels good about it, like that's a huge advantage. And so that's something that just came to mind to me of like there's also so the question is all for these companies not just who is it but it's around kind of like the industry timing and maybe like do any companies have a lead to get there first.
Yeah. So it's interesting that you mentioned the 800 gig versus 1.6T generation shift. I think that's a good point to get into the whole optics discussion here because the way you get 800 gig optical networking is basically you have 800 gig lasers, and the way you get to 1.6T is of course you could have sixteen 100 gig lasers, but you'd rather have eight 200 gig lasers. So now you are going for these lasers that are like twice as fast, and those are all externally modulated lasers. So there is a light source and there's a modulator, and this is all done externally to the silicon photonics chip. So now what's interesting is that the 200 gig lasers can be sold for a lot more money than the 100 gig ones. And it rises the revenue of all these companies selling it. And to add to the benefit of all these optics companies making this stuff, you have both the 800 gig ramping. I mean we are not yet in the peak of the 800 gig cycle. We still, if you look at the S-curve, we are still in the rising part of the 800 gig curve. And then you have the 1.6T which is starting up sometime 2026 I would guess, and a lot of this is like what they call an overlapping S-curve. So it's not like the 800 gig generation is phased out and the 1.6T has come in newly. They kind of overlap. So these companies can not only sell into the 800 gig market which is still not topped out, but they can add additional revenue on top of this by shipping out 200 gig lasers.
Nice. Yes. So they can have a product portfolio and the newest products are all they're slowly ramping. They'll probably have the highest ASPs, but then their last generation will probably have significant volume, but lower ASPs and so on and so forth. And it actually that kind of reminds me of like TSMC for example, as they're ramping, you know, two nanometers. Well, they're still making a ton of money on five nanometers or three nanometers, right?
Totally. Okay. So let's dive in this systematically. Should we talk companies or should we talk like technology like scale across, scale out, scale up?
Let's go with the technologies part of it. We'll get to the companies a little later because once you understand the underlying technologies, I think the companies would make a lot more sense.
Yeah, sounds good. Okay, so let's start scale across. We'll start there. Scale across, talk to me about scale across optics and is it interesting?
Scale across is actually to me technologically the most interesting part of optics. Actually we don't talk too much about it. That's because you need to connect entire data centers with optics that are like long-haul optics. And so you have these things like 400ZR which specifies like a really long-range connection over I don't know kilometers, tens of kilometers, hundreds of kilometers, things like this. And this whole scale-across technology is transitioning from 400ZR to 800ZR as well. This is not even those 800 gig to 1.6T we were just talking about. So in the scale across also there is a transition happening, and the technologies to me is very fascinating because these connections can send petabytes of data across extremely long distances. Of course, now this is like power hungry, you know, these are not like co-packaged optics. This is not like that. But what they actually do is they use a whole bunch of lasers in a comb fashion. They call it a comb laser because if you look at the spectrum versus wavelength, it looks like the teeth of a comb, like closely spaced signals. And you have like 96 of them, like even more than 100. And each one of these teeth of the comb actually carries a signal, and the whole thing is like modulated and it's sent and then demodulated on the other side, and there's like a whole lot of error correction that goes on. So to me it's technologically amazing, and it is coming up now because like data centers are being built out and they do want to connect them together. And Jensen I think framed the scale-across term, and it's become the new dimension of optics. But really I don't think that's where all the excitement nowadays is lying. It's just my personal preference to look at right now.
Yeah, totally. Let me ask you two quick questions. So when you talked about the spectrum versus wavelength in the combs, are you saying that like they'll send signals using like 96 different wavelengths down the same fiber at the same time?
Yes. Yes. Absolutely. And then of course so they have to, you have to be able to do that with the laser on one end, and they have to be able to receive all of it with the photodiode, and sort of it's probably all multiplexed together and sort of demultiplex it and error correct it, and it's kind of magical that it all happens.
Exactly. The term for this is DWDM or dense wavelength division multiplexing. So you densely put multiple wavelengths, multiplex them and send them down this huge optical cable like that goes tens of kilometers, and it's like decoded on the other end. It's not like fantastically new technology, but the fact that that is also undergoing a speed transition.
Yeah. Yeah. Okay. And so when you say it's not new technology, tell me historically where was this long-haul optics used and then why is it interesting now in terms of connecting data centers full of GPUs?
So this technology has been used in basically subsea cables. Like in the telecom world you have to lay down like these giant cables under the oceans that basically take all the information from one point to another when you make calls between continents and things like that. It all goes under the ocean in optic fiber. So these kinds of things use basically technologies that are like dense wavelength division multiplexing. The reason is because on each wavelength you can send information. If you can send like hundreds of them you can send more information, and we really do want to send more information. And you have to have like amplifiers everywhere to boost the signal as it goes through the ocean and stuff like that. So all of that exists already, but today it is important because we want a lot of bandwidth between data centers. And a lot of this technology is also used in metro data centers, which means not, I mean metro, I would say metro telecom, right, within a city you have a lot of optical fiber that's going everywhere providing stuff to people's homes and collecting all the information and like feeding it to the main optical trunk. So all of this exists in technology like whether it's subsea or within a metro, but now they want to connect data centers. And yes you can reuse the metro already laid out optical fiber to do a lot of these things, but sometimes for business reasons and for exclusivity reasons data centers prefer to have their own cables drawn out between them. So that's a deep dive later.
Yeah. Yeah. I guess two last points and then we'll carry on because this is a very interesting topic. I'm sure the scale across for data centers full of XPUs communicating with each other during training. So that's the use case. This is not inference. This is just like I've got so many XPUs that I actually have to have different physical halls and data halls. And by the way, maybe they are in, you know, two separate states, you know, Tennessee and Mississippi or something because there's only so much power that I can get access to and I'm trying to ramp this up quickly. And so now these things are spread apart by, you know, 100 miles, but I want them to all communicate as if they're in the same building. I'm sure I would assume that that scale-across network is probably optimized differently than a telecom network, right? So I assume that the, and I don't know this, but I'm assuming that like the GPU use case they're trying, they're probably making different trade-offs with respect to like latency, handshakes, error correction or something. Have you looked into that at all?
Not in depth, but there is a fundamental limitation to this stuff. The coherency between GPUs across data centers will never be faster than the speed of light through a fiber. You know, there is simply nothing you can do about it. And that latency will have to be handled by the entire system put into place because these are latency-sensitive applications. So there will be some protections put into place, but I did read about it I think, but I can't like put it across to you right now exactly about that. But this technology exists and it's not just for hooking up data centers and GPUs together because a data center inherently is useless if it cannot be connected to the outside world. Now when there are so many people like putting inference requests and you know asking it to do this and that, agents are coming in and saying all of this stuff, there's a lot of data center traffic that goes in. So that connection also requires wide bandwidth connections.
Sure. Totally. Okay, last question then we'll move on. When you say that there's amplifiers on these long-haul routes and some of them are under sea and of course some are just in metros, how are those powered? Like how do they power those under the sea?
They draw electrical cables through too.
These are not like low power. You can't just like dump in batteries and call it a day. They have to be powered with electrical lines that are also drawn under the ocean. So bottom of the ocean, it's optical lines and electrical lines.
Yeah. Yeah. Because you have to amplify it every so often.
And they're terrible. They're terrible. Like there are, you know, sharks chew through them. Like the trawlers break them. You can look in the news every now and then, you know, people are cutting subsea cables all the time. So people are like, oh, now I'm, they're just like I'm going to have a backup link that goes through satellites, you know, high bandwidth free space optical link so that you know in case this gets cut, at least a conversation may drop in bit rate, but it doesn't have to drop. You know, it's a big problem.
Yeah, it's a big problem.
Yeah, that's interesting. Okay, carrying on to scale out. So let's talk about scale out and talk about how it works and where we are today with industry shifts.
