Every part of this power supply chain has a different player, has a different technology. It is not like what you can see in HBM. Oh, it's HBM. What is it like, the same three companies? What are they doing, stacking the same way, right? It's kind of a closed problem in a sense. Power is a wide, wide open problem.
Welcome to another Semi-Doped podcast. I'm Austin Lyons of Chip Strat, and with me is Vik Sekar from Vik's Newsletter. What's going on, Vik? It's been a crazy earnings week this week.
Yeah, I can't listen to all the earnings calls. I just try to listen to a few of them and try to run the transcripts through LLMs and tell me what happened. But other than that, it's too much to keep up. It's a fire hose of earnings calls this season. I can't keep up.
Yes, I know. Like in a dream world, they would just all talk to each other and schedule like one per week for the whole season. But no, instead it's like everyone in the span of a couple days.
It's not intended that people listen to all of them. I'm guessing that people have investments in company A, B, C, D, and they just listen to those, and you don't listen to everything. Because you know, even big investment firms have people who address certain sectors of the semiconductor markets. People like us try to cover optics, data centers, like memory, CPUs, power. I don't know, power. Speaking of power, that's what we're going to talk about today.
Power, we're here to talk about power. So, before we dive into power, should we address... we had lots of YouTube commenters, so thank you YouTube commenters. Should we address any of the questions?
Yeah, I think we should at least address one or two of them. One was on the memory tax episode that we did. The comment was basically that when we said that memory costs going up and all the capex being directed into memory means that that money is not available for compute. I think the overall sentiment around that was like, yeah, the money that's going into memory is more like inflation and it doesn't usefully contribute to compute or solving the memory bandwidth or making data move faster between racks. It doesn't solve any technical problem in any way other than that you're just paying memory companies more. So I think that was mostly the sentiment.
Yes, totally. So the comment was something like, you know, that we're misunderstanding and compute's not the bottleneck. Memory is a bottleneck and therefore you should spend more money on memory. But yeah, to your point, what we were saying is like, no, no, no, we're not saying don't spend more money on memory. We're just saying that like you bought a gallon of milk for $2 today and tomorrow it cost $2.50 for the same gallon of milk. So you're just spending more to get the same stuff, you know? So yeah, it's like the cost. Yeah, inflation, right? So it's like...
It's not like you get chocolate milk. Chocolate milk would be nice, but you don't. You just get regular plain old milk, but you pay chocolate milk prices.
That's not helping anybody. It doesn't increase our quality of life.
Right, right. So the purchasing power of a dollar is going down for compute. So a bunch of CFOs are going to say like, hey, we're spending more for equivalent compute. Like, you know, does this meet our internal ROIC targets? What's the projections? When will this stop? So on and so forth. But okay, so we addressed that. I think we had a lot of people asking about boardfly. Did you want to talk about it at all and how to get to seven hops?
Yeah, so the best way to go about this, even rather than me going through and explaining how this works, is that Google has published a deep dive on the TPU v5p blog. You know, it's a deep dive blog on their website. So if you go there and look at the picture there in the blog, it explains exactly how the 3D Torus approach has 16 hops and how the boardfly approach has actually seven hops. So the idea is that you have to get from the board into the group where you have, you know, so many boards in a rack kind of thing, and then you go between kind of, you know, rack to rack through the OCS switch, and then, you know, finally you follow the same process on the other side. You reach the other rack and reach the other board. The Google picture actually shows seven hops. So you can see how each hop has a different location it goes through. So the fact is that you can reduce network diameter by using their boardfly topology means that the latency equivalently comes down compared to using a 3D Torus. The latency is more than twice better.
Nice. Okay. So folks, if that's not enough, go read the original source for yourself and you should be good to go.
Yeah, it's a good explanation. They do a good job. Google blogs are nice.
Yes, totally. They spend a lot of time and effort. So, thank you Google, whoever wrote that out there. We appreciate you. So, okay, let me hit really quick last thing. I had a couple quotes from the Coherent earnings call that I thought were interesting. And we've talked about these topics before, but I'm going to hit on it again just because it's a continued theme that, you know, people should be tracking, which is ultimately there's massive demand for indium phosphide-based lasers and everyone is scrambling to create more supply. And for Coherent, the quote was from the CEO James Anderson: 'We are aggressively ramping 6-inch capacity because 6-inch wafer compared to a 3-inch wafer will produce more than four times as many chips at less than half the cost.' So this is how Coherent is trying to rapidly increase their supply is moving to 6-inch wafers. And so, again, I just wanted to unpack it in case people hadn't heard that before. This whole like, if you're like, 'How do you get to 4x more chips at half the cost?' And we've talked about this a little bit, but ultimately it's about the area of a 6-inch wafer compared to the area of a 3-inch wafer. And you know, if you... it's radius squared. And so if 6^2 is 36, 3^2 is 9, 36 by 9 is 4. So that's how you get to four times more. And then of course the cost thing is interesting and I don't think we've talked about that quite as much, but at the end of the day, if you have the same number of steps and you're just processing a bigger wafer, your costs are fairly equivalent. You have the same number of steps, you might have to pay more for a bigger substrate and you have to pay a little bit higher input cost because maybe more photoresist and chemicals and gases and things. But let's say the cost is maybe 2x higher. You could still see a world where to process a six-inch wafer, you can still see a world where you get four times as many die per wafer because it's bigger and yet it only costs you twice as much to process it. So your cost efficiency is twice as good and therefore it costs you half as much per die. So I just wanted to remind people really quick that that's Coherent's play. Now the question of course that everyone's asking is, okay, they've got six-inch wafers, but is it yielding and what are they actually making? And so James Anderson did address that as well. He said, 'Given the healthy yields we are seeing with 6-inch production, we began production of 6-inch indium phosphide at a second site in Ytterby, Sweden.' I'm not quite sure how to pronounce it. Ytterby, Sweden. 'And ramping at two sites in parallel will significantly accelerate our production capacity ramp. Additionally, we are in production on three different types of key transceiver components on 6-inch indium phosphide: EMLs, CW lasers, and photodiodes.' And somewhere else, he said, I think it was in a Q&A, he specifically said that their 6-inch yields were as good or better as their 3-inch yields, and pointing out like that's comparing to the mature 3-inch yield, not just when 3-inch was ramping. And so I think he, you know, without saying much about yields and performance, Coherent was definitely trying to signal like, hey, we're not just building the simplest component. We're photodiodes. We're building photodiodes, CW lasers, EML, and they're pretty confident in their yields. So I would just say it's something to track. Coherent is telling the story that they're progressing nicely. I think maybe the final read-through will be on their margins because of course if they are increasing supply, decreasing the cost, and yielding well, which would impact your cost, and those devices they're making on the six-inch wafers have good performance, which means you can maintain ASPs or increase ASPs, then ultimately this should flow through to their margins. So that's probably just the final takeaway is if you're trying to track it and really trying to figure out is this just a story or are customers actually buying devices that are built on this new capacity. Ultimately it's about yields and ASPs and we should be able to track that.
So one small caveat to that is that when talking about yields of CW lasers or, you know, electroabsorption modulated lasers, EMLs, which are the workhorse for 200 gig transceivers, and CW lasers are the workhorse of co-packaged optics. These are the two hotly contested areas really right now between Coherent and Lumentum and all these companies. It is very important to distinguish that CW laser, but at what power are you talking about? 50 mW, 100 mW, 400 mW? That is significant differences between what yield means on what product. Similarly, EMLs are the same way. EMLs have like two components to it: has the laser and the absorption modulator, and they are usually kind of co-designed to work together. And that is a significantly more complicated problem to solve compared to, you know, maybe CW lasers where you just have to like put out laser, right? You're not modulating anything. It's just like a flashlight that's on. You're not turning on and off the flashlight depending on a zero or one. So CW lasers are structurally... functionally, I wouldn't say structurally, functionally easier. So it really depends on what yields, which product line, what power, what output levels. So it's very easy in an earnings call to put a blanket saying, 'Oh yeah, everything is great.' But if everything is great on a 50 mW laser, that's not what we're talking about, you know? So it really depends on what exactly is yield meaning here. So that's the...
That's the wet blanket I want to put on your otherwise optimistic statement.
No. Yeah. No, totally. I definitely don't disagree with that. There's this high-level directional guidance they're giving and then to you there's the nuance and they probably won't share that level of nuance on a call just like Intel's not going to get super nuanced into like the yields on 18A and 14A and give engineering specs. Yeah. But then that's the game is how do you sort of reverse engineer or back out what you can to get a good sense of like, oh, they are actually just insinuating 50 milliwatt lasers continuously, they're not 400 milliwatt, and maybe again that shows up in revenue, like top-line revenue, or maybe it shows up in margins or something where it's like, oh, see, their margins aren't great because it's customers buying the cheap product versus like high-performance powerful product.
If they are selling high ASP products at high yield, it will show up in the money. Follow the money.
Exactly. Exactly. Totally. So, okay, cool. All right. Let's jump into power now. So, you wrote an article on power and this is a perfect forum for us to unpack it. So...
Yeah. Set the stage for us.
So, there is this growing sentiment. It's not just from me. A few people in the industry have pointed it out as well that we are coming to a point, maybe not in this generation, maybe it's the next generation of accelerators and racks, probably Rubin Ultra, because the Kyber rack is a 600 kW per rack power consumption that is much higher than what usually data centers are used to. So in the cloud era each rack used to consume like 20 kW, you know, now AI accelerator racks consume anywhere between 100 and 120 kW. Now we are talking about the Kyber era of racks where the Rubin Ultra will go in at 600 kW a rack and the future will hold one megawatt a rack. That's an enormous amount of, you know, that's a 100 times power per rack compared to what was in the data center, you know, cloud era pre-AI era which was like approximately 10 to 15 kilowatt. You're talking about one megawatt per rack and imagine how many will go into a data center and imagine, you know, how many of those data centers are going to be connected together via scale across, you know, all of that stuff. So this is a looming problem, a bigger and bigger problem with every generation of, you know, AI that comes out and chips that come out. So I wanted to address it in my Substack article to just point out that power is the next physics wall, you know, and this is a theme I want to pursue in a little bit more depth over time because as you can imagine it's an enormous problem, right? You know, you got power generation all the way at like, you know, whether you're talking turbines or nuclear power, how that's transmitted and how it reaches the substation, you know, which is converted into the data center and how it is ultimately delivered to the GPUs. We'll cover some of that in detail in this podcast. At least I'll explain how this conversion happens and where, you know, the market's directions are going to go, you know, going forward. So but you can imagine like this entire chain has so many players and it has so much going on there and all of it is important to deliver the next generation of power. So that's what I wanted to kind of touch upon in this abstract post and kind of we'll briefly cover as the next physics wall.
Nice. Nice. Okay, let's get into it. So what is the problem with a 1 megawatt rack? Why is that a problem?
Yeah. So in this one, I want to kind of start at the rack level. What is the underlying physics problem? And then we will kind of zoom back a little bit and understand how power is actually generated up to that one and delivered up to that one megawatt rack because it's important to understand how the power gets there, right? And then we can talk about what happens within the rack which is essentially what I focus on in the Substack article. So the thing is very simple, you know. The idea is why are we talking about optics now? Okay, I promise this is not a tangent, okay? This has something to do with power. Why are we talking about optics? The whole problem was with copper, right? And reach. So as the speeds got higher and we needed to connect racks over longer distances, optics was the only way forward because copper reached its physics limits. And the physics limits in copper is as you increase the speed, what is flowing within the copper cable is actually an AC signal. It's like it's a varying signal, right? Because you're transmitting, let's say, bits that go up and down like whatever. So it's like a varying signal. And what happens in copper interconnect when you have varying signals is that all the current doesn't flow through the whole copper wire. They tend to concentrate on the periphery. Only the outermost ring of copper actually holds any signal. There's nothing happening with the rest of the copper cable. This is because of the phenomenon called skin effect, right? It goes to the skin of the copper cable, not really the whole cross-section of the copper cable. So the resistance goes up because you have all this cross-section, but if you're not using it and you now have more resistance. So resistance was the fundamental bottleneck for why we needed to go from copper to optics. And now you see everybody's in the optics like indium phosphide shortages. The rest is history, right? Look at the optics market we are in. So this was a bottleneck that was physics driven and whenever something is physics driven it's easy to identify. So go ahead.
Yeah, totally, totally. No, that is what we try to do here is look for the fundamental physics constraints and then ask what is going to happen beyond this. All right. So, really quick, for maybe less technical listeners, when Vik said AC, he meant alternating current. That's the varying current. And then this skin effect thing, I mean, if you think of like, I like to think of electrons flowing through a wire as sort of like a pipe and or even you could think of it like as a subway tunnel with like lots of people trying to push through it. And if this skin effect means you can only walk like at the edges of the subway tunnel, then you try to take the same people and push them through the edges of the subway tunnel, there's going to be more resistance. You're going to bump into each other more, right, than if you could just everyone could just walk nicely with lots of space around them. So when Vik says there's an increased resistance and the skin effect becomes worse and worse at higher and higher speeds. So when we're trying to communicate more and more data at higher and higher speeds, then you get more of this skin effect resistance bumping in together. So there's just a little analogy for non-technical folks. Vik, now please carry on.
Yeah. No, thanks. That's good. I like the subway analogy. I like analogies.
