In March 2024, a company almost nobody outside of the industry knew IPOed at $36 a share. This June, it hit $499, joined the NASDAQ 100, and printed 93% year-over-year growth at 76% gross margins. Astera Labs doesn't even make GPUs. They don't make memory. They don't make optics. Astera makes the chips that fix copper. The little devices that catch a dying high-speed signal halfway down the board and relaunch it clean, unglamorous, invisible, and in every single AI server Nvidia's partners ship. Today's episode is a story of how a repair part became a franchise and how that franchise is now going after the biggest socket in the AI rack.
Hello everyone, welcome to Semi Doped. I'm Austin Lyons of Chip Strat and with me is Vik Sekar of Vik's Newsletter. So Vik, let's talk Astera Labs, but let's talk retimers. Like when I first heard about retimers, I was like, what does that even do? What does that mean? So what do you say? Should we get into that today?
Yeah. Like let's talk about the problems of copper connectivity which we have spoken about a lot before, but I think retimers is an interesting concept because we always talk about stuff that is outside let's say a server tray, right? We talk about how GPUs are connected together and scale up. We talk about how to connect racks together and scale out and scale across all of that stuff. But what we've never covered is how does a GPU connect to a CPU within a server tray or how does a GPU connect to the network interface card or the NIC? We'll keep referring to this as the NIC, network interface card. So how does that connection happen? You know, it may not seem like it's much of a connectivity problem because come on, how far are these things, right? Like the server tray is maybe this, you know, yay big with about 2 ft wide by 4 ft deep. But that is still a problem because the way you connect a GPU to a CPU or to a NIC is through the PCB that is inside the server rack, right? Like it's a host board that is holding all these chips. And so those are not like excellent conducting mediums to carry high speed signals, right? And so that is the problem we're going to talk about today and how Astera Labs solved that problem and became pretty much an integral part of what is inside the server rack, server tray today.
Awesome. Interesting. Let's get into it. Yes, because already you got my wheels spinning because you're talking about copper PCB traces and high-speed signals and we've been talking about like with Credo, you know, and active electrical cables like the copper problem. So yeah, let's get into it. When a GPU talks to a NIC and it talks over this, you know, PCB trace, like what are we talking about?
Okay, so let's go back from the beginning and like set the stage quickly but also in a complete fashion so everybody understands what's happening in this part of the AI stack. Right, fundamentally what happens outside, you know, we always talk about NVLink for scale up or you know Ethernet or you know InfiniBand, all these things, these are all like outside the server tray. What happens inside is something that has been around like very long. Right, this is in every PC that you build out. It's been in gaming PCs forever. It's called PCI Express or PCIe. Okay. So PCIe is how the GPU communicates with the CPU or the NIC for example. It's just a protocol that's been around for a very long time. So the whole idea is that these don't have to run very fast on a single lane basis. However, they have many parallel lanes. So a PCIe interface can have 16 lanes and so it's called a PCIe x16. Now the way this is dealt with or spoken about is that you have so many gigatransfers per second, right? So you can have let's say a Gen 5 PCIe, you know, has essentially let's say 32 gigatransfers per second. This was actually used in the Hopper era anyway for connectivity. So what that means is that why are we saying transfers and not like bits, right? We are talking about transfers because we are talking about transferring a symbol. A symbol can be a collection of bits. It can be one bit is one symbol which is basically this NRZ modulation, zero or one, very simple.
Or it could be a PAM4 modulation which is like four levels, right? So you could have different kinds. Like so when you do PAM4 you have two bits to a symbol. So gigatransfers means you're transferring 32 billion symbols of either one or two bits every second. So if you look at that in the time domain, like let's see, like how, so what are we talking about? How many symbols every second? So every 31.25 picoseconds you have a symbol going by. So you think of a string with like a lot of beads on it and each bead is a symbol and the beads are moving from left to right. Right. So that's how it looks. If you kind of imagine bits moving on a wire and that's exactly what's happening between the GPU and the CPU.
Did you say picoseconds, like 10 to the minus 12?
Yes. Yes, it's very less. Yeah, it's a very small amount of time and that's what it takes to transfer like a billion symbols per second. It takes picoseconds, like you know, it's a very very small period of time. And so the thing is that it's not easy to do this. Okay, it's not easy to move bits across from one place to another because it faces several problems. Copper is a problematic medium and when you try to push these bits from one place to another things go bad. Okay. So one thing is like loss, right? Like you know, you put a signal of high strength in one end and it loses all its strength by the time it reaches the other end of the wire, right? That's one problem. Then the PCB itself has problems and all that. And then these things are like high-speed signals so you can think of them like sometimes they ping-pong, so they reflect from that end. So you can think of the beads trying to come back on the wire. You don't want that, right? Those are reflections.
Right. Then crosstalk, you know, if you put two sets of beads next to each other. Imagine if one bead on this string is affecting the bead on the next string. That's a terrible outcome. But these things happen in copper. So all of these things they basically need correction. They need some way to fix this, right? But that's just the copper thing. Actually I should mention that there are two more problems we should talk about and that is inter-symbol interference is one of them. Which means the beads on a string don't actually stay like one bead on the string. Like imagine that you know they're made of sugar or something and they try to like stick to each other and try to become you know one larger ball of sugar instead of like staying distinct sugars. Like as they travel really fast you can imagine they like heat up or something and then try to stick and become one and like a blob, right? That actually happens all the time. So when one symbol travels the next symbol kind of starts to overlap with it and it starts to spread out and that's terrible, right? You don't want inter-symbol interference.
Let me jump in there for listeners. You know, so you can think of if you're familiar with like square waves or sine waves, like if you have a square wave you can imagine that as it travels down this medium it starts to spread out instead of being a square, you know, maybe it's like more of like a hill. And so you can imagine if you're sending these different like pulses in the simplest sense that as they start to spread out they could start to overlap each other to Vik's point of like the sugar sticking together. So now all of a sudden instead like a steep up to one, steep down to zero, some space, and then another steep up to one. You start to get like a ramp up to one and a ramp down, and then maybe it overlaps another one to the point where what you receive is like a 0.5 and it's slowly transitioning and it's a little bit confusing. Is this a one or is it a zero? And then, of course, to your point, as you mentioned earlier, there's this attenuation. So by the time it gets all the way down the copper, it's just like lost, you know, the peak amplitude as well. But back to the inter-symbol sort of overlap here. Yeah. There's like this smearing where you tried to send certain symbols and they're kind of overlapping each other and somehow we have to figure out what was said at the end. Yeah. What's the next one that you were going to mention?
Yeah. The other one is jitter, which means that the pulse is not where you think it is, right? The pulse you think you sent it some time ago but by the time it reaches a given location it isn't there. There are random variations in the system that cause the pulse to be further than you think it is or like earlier than you think it is. So when you go in be like, okay, at this moment I'm going to look and see if this is a one or a zero and I'm going to like lock it down, right? Like that's what you were saying, like let's say that you know the 0.5 question isn't a thing, like we are constantly monitoring this particular point on the wire and we're going to be like, okay, when the thing comes I'm going to call it a one or I'm going to call it a zero. But the thing doesn't come at all or it comes like you know late and you're like, what happened? So now you can't detect the bit because the thing wasn't there when you expect it to be there. That thing is called jitter. And that also causes problems in communication. All of these are big issues.
Gotcha. Okay. Yeah. So you're saying like with jitter, it's like if I'm thinking about like I'm on a clock, I'm at the end of the wire, I'm checking and it's like what am I reading? What am I reading? What am I reading? But nothing comes but then it comes a tiny bit later like in between readings or something, you know? It's like what's going on here? But let me ask you, Vik, between the inter-symbol interference and the jitter, like how much of that is deterministic? Like we can figure out, oh, this is copper and we know how a high frequency symbol behaves and how much of that is just like random noise or random something, randomness.
I think that inter-symbol interference is somewhat deterministic and you can kind of figure it out and there is a way to do this and I'll get to that. Jitter is more random. You really can't do much about it and all this affects how we will ultimately detect if it's a zero or a one. See that the whole problem here is to tell whether at any particular instant you have a zero or a one. If it's a two-level modulation, if it's a four-level modulation like PAM4, you have to tell which of the four levels it is.
And typically they talk about it in PAM4 as 0, 1/3, 2/3 and 1. Instead of trying to use something else they just break up the one into four levels, right? So it's up to you to tell what the level is. That's the whole problem.
Yes, okay, yeah, I could see how that's challenging especially for PAM4. Like if you're trying to read zero volts is 0, a third volt is 01, on 2/3 volts is 10, and a volt is 11 or something like that. Now all of a sudden when there's that smearing and stuff, it's not like oh is it a zero or one and we're at 0.5 and we don't know. Now it's like really crazy. It's like oh it's supposed to be zero or 0.33 but I got 0.28 volts or 0.16 volts or you know it's tighter. There's less margin.
Yeah. Yeah. There is a nice way to visualize this and you will see this mentioned a lot in this interconnect world and those are what are called eye diagrams. Okay. And I'm just going to explain quickly what that is. So imagine at this one particular time called a unit interval. That is basically the 31 picoseconds that I mentioned because you know that that is the symbol period, right? That's the period. So that thing is called a unit interval and is usually on these charts you'll see it as UI. And so you're looking in that unit interval and you're seeing all the bits that go by like you're looking at just one bead on the string but that bead is moving, right? Like you're continuously looking at the string, think of it that way. And what you're looking at is an overlapping bunch of zeros and ones for example. And always you want to have some kind of a separation between zero and one, right? So that you can clearly say anything above 0.75 is a one or anything below 0.25 is a zero, like we just round up that round down that way. And so but nothing should be there between 0.75 and 0.25 because you don't know where it goes. The uncertainty is too much, right?
So that gap is usually referred to as the eye opening. The eye of this signal must be open because if the eye closes, which means there is no space between the voltage levels, you can't tell what it is. So the eye opening determines everything.
Okay. Okay. So you basically take a bunch of snapshots at this unit interval and then plot them all like on top of each other, right? So you take like a thousand of them and in the ideal sense you would have like a rectangle. You have a bunch of ones and a bunch of zeros and that's it. But of course it takes time to get up to one and time to get back down and so that's why you kind of end up getting this eye shape because it like ramps up and ramps back down. And to your point is when you throw thousands of these measurements on top of each other, there should be some sort of gap where it's very clear, hey, there's nothing in here. So it's clearly a zero or one. But if they start to like if that gap closes that's bad news because hey some of these measurements that we took are in the middle and we won't know what to call it exactly.
So now the whole problem of why Astera Labs solved this problem is that they needed to find a way to keep this eye open when the communication is going between a GPU and a NIC or a GPU and a CPU and the distance is in the range of like about 30 cm. It's not that much, but it's still required because at the speeds that we're talking about because Gen 5 is 32 gigatransfers per second. Gen 6 is 64 gigatransfers per second and it goes to PAM4. So that makes it harder because you're adding more levels and you're going faster. So it really requires retiming. And then in the future we will go to Gen 7 which is at 128 gigatransfers per second with PAM4. So Gen 6 is already in I think in the Blackwell era of chips it's being used. So yeah we'll get to all this but so Astera Labs is basically their job is to keep the eye open within a compute tray. That's their whole purpose with retimers. So what are retimers? Right? Let's get to that portion because we need to talk about okay now we have inter-symbol interference and we have jitter whose main effect is to close the eye. Now we have to find a way to fix it, right? And there are a few tools to do this. Okay? I think we should first introduce the idea of equalization. It's a fancy way of saying look, I'm going to lose half the signal like down the line. So what if I just boost the signal twice now so that by the time it reaches that point, it'll still be like what it was here because I amplified it beforehand. And so that when it degrades later I can keep the voltage level constant. Right? So that is equalization. You're basically applying the inverse of what will happen to the actual signal. Right?
So okay let me interject really quick. So back to the eye diagram. So you're saying the problem is the eye starts to collapse especially if we just don't do anything but we're increasing the speeds. Of course it's copper and so kind of the distance at which you can effectively like keep the eye open is shrinking. And so the question is you know how do we keep the eye open and you're saying like one of the simplest ways is sort of like hey if we know that it's going to like attenuate or degrade as it goes through can we just amplify it? And it's kind of like if you and I were in a hallway and I was talking to you and then you went and went a lot farther away, I would just shout so that by the time it got to you, it would sound normal.
Yeah, exactly. That's a good analogy because you know, you can kind of see how far away that person is and there's no point talking softly because you know they're not going to hear it. So you would like adjust your volume to be good enough so that you have a good chance of them hearing it at the other end. Yeah, exactly. That's a good analogy. I like it. That's on the transmitter side though. That's what you can do by shouting in louder. But what can the receiver do? What can the person on the other end do? Right?
That's what you will often see as CTLE, is called continuous time linear equalizer. Okay. So this is like a long acronym but in concept it's relatively simple. Okay. It is just that you want to amplify the high voltages more compared to the low voltages so that you can kind of stretch the eye open. Imagine that you have remember I told you that you know 0.75 or above is a one. Now what if you can just amplify the 0.75 to make it 0.9 so that the eye is even more open and what if you can make the 0.25, you know, 0.1, you know, so now your eye is like a much nicer and much wider open. So you can amplify the highs while keeping the lows low. Okay? So that's one way the receiver can deal with this problem of all these channel imperfections that happen. This is still analog stuff, okay? There's no like digital work going on. This is just a simple amplifier. But the problem with this is that it amplifies all the high frequency noise stuff. Like if there's any noise and all that, everything gets amplified because it's just a dumb amplifier like it amplifies high signals and keeping the low signals low. That's it. That's not a good thing. And if there is any interference from the neighboring bits moving up and down, it'll amplify that as well. So it's not entirely very clever. But it's a necessary part of the system.
Yeah. It reminds me sort of like naively like if I'm the receiver, if you're at the other end and we got far away and you shouted and then I can kind of hear you, but instead I'd like put headphones on and I just hold a microphone and then the microphone sort of like does that. It like hears you and I turn up the gain or something and it's like, oh okay. What you said at one volt was down at like 0.75 by the time, but then this microphone just amplified it back up to one volt and I can hear it fine.
Yeah. Yeah. Exactly. Exactly. So now there are other ways, there's one other thing we can do with the receiver and I think it's a little bit important and this is the clever one. So what this does is remember that it's a very good question you asked me before as to which of these errors, inter-symbol interference or jitter, is like deterministic. And the fact that inter-symbol interference is deterministic actually helps because you can predict you know how this spreading of the square pulse into hills happens beforehand by running some tests on the channel and you know how it spreads and then you can like back calculate and subtract it out of the received signal because you know how much spreading has happened and so you add it back or subtract it back and you reconstruct the waveform. This is called a decision feedback equalizer. So it just knows what's going to happen in inter-symbol interference and it cancels it out.
Gotcha. Nice. That is pretty clever. So does it look at like it depends on like what I just received like oh I know that I just received 1101 and therefore I can calculate sort of in real time what kind of like noise or smearing or interference I would get from that particular pattern based on this particular channel.
Mhm. Yeah. So it has some kind of a characterization and then it reconstructs it based on how much it knows the spreading or the smearing has happened. Very clever stuff you know very complicated stuff and it is because of this that even Gen 5 PCIe works. It's a very essential component. These DFEs are very essential components right.
Does that really quick, does that have to get tuned in manufacturing like per board or something?
I think it does. Yeah I think it does.
Interesting. So it's not all that straightforward to do, but it is a system that's been around a while. So it's not like ultra new, but these are all like very clever chip design techniques. You know, there are these I have some friends who actually do this for a living and they're really good at it. They're like these analog guys. They're awesome.
Okay, so and when you go to these conferences like ISSCC and all that, you'll see all these designs of how they do this. It's quite complicated. Let's just say we're just trying to cover it at like an extremely superficial level but the reality is very very hard. Finally I think it also extracts what is called clock and data recovery. So, what that means is whenever, let's say I'll be like, Austin, whenever you see a symbol go by, just clap. Okay, just clap for me. And I'm going to sit here and kind of write down all the times that you clapped. And I'll be like, okay, I know the period of the symbol. I know the clock now. So now I can set like a metronome. You know what a metronome is? It's like that music thing that goes click click click click. So I can set the speed on my metronome and that should work out well because now I know when bits are coming just by hearing the sound. So I have recovered the clock this way. Right.
That's pretty clever and that kind of helps with that problem that we had talked about where it's like if I'm on a particular clock and stuff isn't flying by right when I expect I'm like man this isn't lining up but you're saying no no do the opposite. Just watch what's happening and then figure out the clock from that.
Yeah figure out the clock. So these are all the techniques that people use to correct it and pretty much this is all there is to it. So you can do the equalization at the transmitter thing. Remember you said like just shout louder you know or you can do at the receiver side you can do the equalizer on the receiver side which is like just put a little amplifier or some microphone and turn it up. Or you can do the decision feedback equalizer which is you know what the smearing is happening. I don't think we have a very good analogy for this but you know you can reconstruct based on the inter-symbol interference and finally you have to recover the clock too by you know basically clapping out and fitting a metronome to it or something and you figure out the clock. With all this information you can kind of tell to a high degree of accuracy what the received signal is going to be doing and you can correct for errors with these methods and that is essentially with all these build up a full you know PCIe channel in the transmitter and receiver and with this chain you can build what is called a retimer. So and I wanted to also mention the clock and data recovery process because it retimes the signal right and I'll tell you what that does in just a minute. So now I'm going to just explain what is a redriver and what is a retimer because there is a difference here. Okay. So redriver does everything that I just said but it does not do the clock and data recovery part. And it also does not do the decision feedback equalizing because I think it needs some amount of like sophistication to do that. So imagine you're just shouting louder and you're using an amplifier on the other end or a microphone and picking up the signals better. That is called a redriver. It is just an analog megaphone. That's it. And yeah, you're just driving the signal on one end to the other.
Yeah. So what are the benefits of this? Yeah. It uses lesser power and energy. It's less complicated. You know, a redriver is simpler and things are good. That's it, you know, and you don't have much latency. You can do this quickly, low energy, all that stuff. But the downside is that when you go to faster and faster signals, you got to have more and more crazy things happening to the signal. You can't recover it. So now what do you do? You have to go for the whole enchilada and go for full retiming where you get the decision feedback equalizing to remove inter-symbol interference. You recover the clock. So it's not just that you amplified it but it's the retiming process basically accumulates all these bits and sorts them nicely into the windows they are supposed to be in because you remember you have the metronome clock so you know where the bits should lie because the clock that you recovered tells you where they should lie. So you can take these bits that arrived late or that got messed up somehow and slot them into the correct time slots and now you have a beautifully recovered signal because you took the bits you received and you put it on where the clock should be. So you retime the received signal based on your recovered clock.
