The biggest computers in the world today are basically not in single boxes, right? They are in entire data centers. And the whole point about this is that all these chips are intended to work as one big computer. But the whole problem with this approach is that the compute capability has grown much faster than how you can hook up all these chips in a data center together. So today's compute capability is more restricted by how the CPUs and GPUs are interconnected in a data center rather than the performance of the silicon by itself. So the biggest problem facing data centers today is how to hook up all these chips to work as fast as possible. So networking is all the rage today and it goes from a wide spectrum of whether to use copper, whether to use optics and where to put all this stuff together. So by the end of this episode, you should be able to understand, you know, why we need networking to begin with, what are the three kinds of networks that all data centers are built up from, and why perhaps co-packaged optics is coming. That's what we hear. We don't know.
Hello everyone and welcome to another Semi Doped podcast. I'm your host Austin Lyons from Chipstrat and with me is Vik Sekar of Vik's newsletter. Hey Vik, what do you say? Should we talk about data movement interconnects?
Yeah, we should do this because it's the biggest problem that we are facing in the history of computing because in the past we've always had the discussion of what is the per core performance of a CPU or then you know multi then we went to the multi-core era like hey we have you know 16 cores in this chip or even servers server grade chips have like 256 cores and then then the GPU era came and then you could put in the accelerator cards in your gaming PC and then you got you could get graphics acceleration and we could you know if you remember there was this Nvidia SLI which is like basically that you could put two Nvidia GPUs together and you could put this local interconnect bridge I I forgot what SLI is actually called but it's basically think about it like a bridge that is like an interconnect between two GPUs so that they work as one you know that's like the simplest form of what we're talking about here Nvidia SLI was a way to make two GPUs work as one but now we need to make you know a 100,000 GPUs work as one for LLM reasons so networking is the biggest discussion today.
Totally. And you know obviously for training it's all about like yeah how can we have a data center size brain but even for inference you know this is why we're going to scale up domains of 72 GPUs and eventually beyond is even with large you know frontier models that have 10 trillion parameters and their mixture of experts. We are getting to the point where prefill and especially decode need as much memory bandwidth as possible. There's so much data movement that it's all about like how can we actually put together a bunch of chips on this scale up fabric. And we'll talk through all of these scale up, scale out, scale across, scale in, all that stuff here. But it's not even just about training but even inference is demanding lots of data movement to be able to do frontier models at high you know throughput but also obviously large batch sizes and so don't let people think that this is just about training but this is also an inference story too.
Yeah for sure. So whenever we talk about data movement right so it involves data movement at all levels that's what is interesting about this. Data movement could be restricted to information moving between memory like HBM to the GPU so that is like a really short distance over which you need to move a lot of data really quickly and that's just because of the way LLMs work you need to read a lot from memory all the time and the faster you can do it the more performant your LLM is right in inference for example in training it would mean that a faster network bandwidth between memory and you know the GPU or overall bandwidth being high implies that your training run finishes faster and that is everything because everybody wants to be in the next biggest model everybody wants to train it before the next guy gets there. So that's the whole idea. Everything gets faster, right?
Yep. Yep. So should we share some slides in this talk so it's a little bit more visual? We've heard from lots of our YouTubers who are awesome. Thank you for watching us and they've said we want more visuals. So if you're listening, of course, you know, we'll try to describe it so that you can understand it as you're you know, lifting weights or something. But for those of you who are watching, we're going to pull up some slides here and we'll talk through some of them as we go.
So the whole data center essentially has networking on three different layers. So the first one is scale up which is literally what you think it looks like. You go up a rack. So you have a rack full of GPUs and then you hook them all up going from bottom to top. So that's scale up. And then the other dimension of scaling is the scale out network which means that you can have different racks each of which has its own scale up network but now you hook them all up together. So that is scale out. And just between these two, you can already start to tell that scale out networks are kind of longer reach networks than scale up, right? Because scale up only means that you have to go within a rack. And if you've never stood next to a data center rack, you know, you can imagine that it's about the height of a tall adult human or a little bit higher. I think it'll be like maybe 7 foot 6 to 7 ft high racks. That's about the whole the length of the rack that scale up network has to cover. Scale out, it depends on how many racks you're going to put. So if you put you know one of these super pods or something you know you could travel a dozen racks across from east to west by the time you hook them all up right and then all of the racks in a data center are all scale out. But if you decide that you want to now hook up an entire second campus maybe located 20 mi away 50 mi away you need a whole new network to do that and that is called scale across right so you you can even hook up data centers to act as one big chip. So it's not just you know one data center that acts as a big big chip. You could hook up five of them or two of them whatever that's called the scale across network and this scale across network has the largest reach of the three. So it because it spans obviously you know tens of miles right so the technologies used in each of these layers are substantially different. So let's talk about these because someone listening might say like well why does scale up why do we call it something different than scale out for example why and and you hinted at it that there's different technologies that are used whether you're staying within the rack versus rack to rack versus building to building.
Now a couple things I wanted to mention so scaling up like the problem that we're trying to solve is in a dream world we would just have like the biggest compute with the most memory possible and it would just you know that would be your rack. It's just like what like you know like Cerebras for example like they've just got all these chips on one wafer and they got a ton of compute and they can all communicate really quickly on that like wafer within to each other. The the problem is the way that the industry works today. GPUs are packaged maybe it's it started out with one die then eventually made its way to two die but it's packaged with a discrete amount of high bandwidth memory on a particular chip and and you know in the future you know some of these AI ASIC startups are they're choosing you know a certain amount of HBM and maybe a certain amount of SRAM or making different memory decisions that we've talked about in other episodes but at the end of the day each GPU only has access to so much HBM and what we want is more HBM than what's possible to physically package on it. So we want to make two different GPUs let's just say on the same board for example or in the same server node forget the whole rack even within the same server node we want neighbors you know GPU A and GPU B to be able to access each other's memory as if it was their own. So scale up is really defined as how many other GPUs can I talk to at low enough latency that it feels like we're sharing memory and this is you know RDMA reaching out and you know talking to other accelerators memory that's what this is all about and so when you ask yourself like well how how big could my scale up network be it it one way to think about it conceptually is like well it can be as big as when I talk to every other GPU, it feels like I'm actually accessing my own memory because the latency is so low, you know, using NVLink or something, for example, some high bandwidth fast network. So, you can actually have a scale up domain that spans two racks. It's like technically possible so long as GPUs between the neighboring racks are still so close and connected in such a way that it feels like you're accessing each other's memory as if it's your own. So that is scale up. And then scale out is of course like now when I'm talking my rack and that rack like you know 30 m down obviously there's going to be enough latency, enough hopping through switches and things like that that it won't feel like I'm accessing their memory. So I won't I won't try that. will communicate in a different way and send information differently. I also want to just mention one other thing. So east west like everything on the same sort of like high bandwidth GPU to GPU communication fabric. My understanding is that's east west and technically when people use north south usually I think they're talking about talking to the front-end network like GPUs talking to external CPUs like a like a the chat you know I'm chatting on a chatbot and it it's gets sent in that's like the north south network.
Okay so we've got this slide from Nvidia that starts to show that scale up scale out scale across like we've hit on at a high level. And of course, now there's another new term because why stop there? On the far right side, we've got scale above. So, hey, what if you have data centers that aren't physically connected via fiber optics, maybe, you know, 50 miles apart, but are actually literally in space and and there's you know, communication, I guess, through satellite frequencies instead of fiber optics. And then on the far other end, we've also started to hear about people talking about scale in. Do you want to say anything there on that or do we plan on covering that today?
Not too much, but there is a whole networking effort to make the connections within a compute tray, you know, the thing that slides into a rack that holds maybe the two GPUs or whatever. You could talk about networking even within that. like how do you make that faster? We won't cover it too much today, but we'll briefly touch on it as we progress along. So these two are like really outside the norm, you know, of what normally is in the conversation of data center networking that is scale in and scale above, but they do exist.
Yes, totally. All right, so let's keep going on scale up. So if the goal is to make a group of accelerators behave like one large accelerator, they can access each other's memory. You obviously want the highest bandwidth possible, you want the lowest latency possible. These are going to be short reach. So we're talking potentially next to each other in the same board or within the same server node or within the same rack. So like a few meters at most. So tell us Vik you know this is the question that everyone talked about a lot and and people are still talking about is this copper is this optics?
What is scale up? It's a big question right like at this point it all comes down to what is the speed it can handle right the whole motive the whole mantra of this networking at this level is copper when you can and optics when you must so copper is cheap. It is low power. There's always worked for decades. There's no need to do anything fancy if we can avoid it. However, there is another problem is that when you do go to optics, if you go to optics, it is very very power hungry because the number of cables in a scale up network is enormous. And we have some pictures and coming up we'll show you. But it is a lot of cables and it's a lot of transceivers on either side. So you know you've got to make conversions into optics from electrical you know and then go from one point to another and convert back into electrical. Those conversions are handled by basically transmitter receivers or transceivers and there are like way too many of them, right? So nobody wants to go to something like optics which is more power hungry because of these conversions from electrical to optical and back if they can avoid it, right? And that's in this picture you see this as basically the backend network. The backend network is this really high-speed fabric. Think about InfiniBand. It could also be Ethernet scale up, right?
So these are like really fast connections which bring down the latency between these GPUs to be as low as possible so that they can act as one domain.
Yes. So scale up networking historically it's been NVLink with from Nvidia with NV switches. There's also UALink which is an industry standard that AMD is spearheading but there's lots of other people coming along to try to make like an open alternative for fast scale up networking. And then InfiniBand has historically been scale out networking. And also Ethernet so Spectrum-X Ethernet from Nvidia and Ethernet from others for scale out. But but Broadcom back to scale up has also been pioneering using Ethernet for scale up. They called it SUE I believe scale up Ethernet. So so there's definitely appetite as other merchant GPU vendors like AMD or AI ASIC companies are also trying to connect all of their accelerators together using copper talking you know very high bandwidth they need protocols so that you could buy switches from Broadcom or use a Marvell switch or whatever and so there's there's some industry standards that are being developed. So if you hear of UALink or if you hear of I think it's called EON now you'll know it's scale up Ethernet Ethernet for scale up networking. I think that's what ESON stands for.
What about the front-end network? I think it's like a worth a brief mention.
Okay. So so what is the front-end network here? So the front-end network here is all of the information coming into the data center from the outside world. So, this is connecting. Yeah, obviously you've got all of your GPUs, but like when I type when I use my cloud code locally and it needs to send all this context in, it's coming in over the front-end network and then the GPUs are all talking to each other doing their thing and then it comes back out over the front-end network.
Yeah. And then you have data center interconnect which is like the scale across thing that happens outside of the data center. Right. So, that's that's that's what it is.
Yeah. on this slide I'll that I'll pull up again you know so scaling up you know conceptually you can think of it as adding like more and more accelerators it they're added in three dimensions so you have some like within a board and then some within the rack and you might even have rack to rack connections but ultimately they all have to talk to each other and so there's a lot of thinking that goes into how do you design the topology of the network you can have leaf network switches spine network switches but obviously you can see the trade-offs. Even when you look at a picture like this, like if you in the far if you got the far left bottom GPU trying to talk to the far right bottom GPU, there's going to be hops through the network switches. And so what's actually really interesting here is if you go look at how long does it take for even an NVLink switch to send information, you can you know, let's say it's like four milliseconds or something. And then you start to look at for an LLM like like a mixture of experts LLM how how many how much infer like how many times does information need to be sent back and forth you can start to calculate like the minimum amount of time for inference to happen. So you know if you have a certain number of milliseconds and you have to make a certain number of sending information back and forth you can multiply that and you can get a number of milliseconds which then can actually tell you what your throughput will be. So you know like oh even for one user we might only be able to get you know 400 tokens per second and that is the limit that maybe a GPU could do in a particular architecture. And it's not even like compute limited but it's about data movement limited. So I just wanted to share that because I I thought it's interesting that inference it it's just like another use case of showing that inference can be limited by how quickly you can move information and how many hops you have to make and and how each piece in the chain how long it takes to respond. it can actually cap the interactivity which is ultimately like the the user experience and and and so this is why again you've got these SRAM based systems the Groqs and the Cerebras where because maybe they don't have you know in Cerebras's case everything is just on one wafer right next to each other if you can fit all the model weights in and you can keep it all onto you know one wafer or a few wafers you can skip a lot of this communication hops and ultimately get a much higher interactivity.
Awesome. Yeah, there's a lot of networking decisions that goes into, you know, how well tokens work out for you when when using it at scale, right?
Exactly. Exactly. Oh, this is this is one of the pictures that always like fascinates, right? Because you can see this rack there with like all these cables going up the rack hooking up all the cables, all the GPUs within it. It's just it's just a nice way of visualizing what kind of cabling goes on within a single rack. Now for a 72 GPU rack, you know, so because you want to have an all-to-all connection between all the 72 GPUs, which is pretty much 72 squared, you know, you'll get like a little over 5,000 cables that require to go between GPUs in a single rack. And I read somewhere that this this adds up to about 2 km of cables within a single rack. And they carry an enormous amount of information. That is just staggering, right? Because if you think about how fast each of these cables are, you are looking at in the Blackwell era, you're looking at 200 gigabits per lane. So that's really fast, you know, now you can multiply that up you know by whatever and you have like an incredible amount of data going by everything. In the Rubin era that's going to be 200 Gbps but bidirectional. So it's going to have both directions running on a single cable at that speed. So that's just goes to show that the the kind of speeds that you have on this stuff is is insane, right? Because I bet that, you know, if you go to Cat 6 cables that like run in your home, for example, they don't most people don't even run like like a 10 Gbit per second in at their homes. They don't run 10 GB networks, right? Because you need special switches for that kind of nobody even cares about that. So all the stuff that runs on your in your house, your networks, they're all like like very slow, like if if at all, it's like maybe running at one one to 5 Gbps or something. We can do a network check in your own house later on a wired network. Don't do it wirelessly on a wired network and see how that how slow that is. This is incredibly fast and you got like so many cables and that's just one rack and then you have like you have so many racks in a data center all of which are carrying this much data. Then you hook them all up together. So it's just it's just amazing how much networking there is.
Indeed. Indeed. We we'll move to the next slide. So, I remember when Elon posted this when they were standing up the XAI Colossus 2 data center in Memphis, I believe. And obviously, these are purple cables. And so, you know, Credo is synonymous with purple cables. And so, I remember when everyone saw this, they're like, "Credo, go Credo. This is amazing." Now I think interestingly now I think some customers are asking for different colored cables so that it's people aren't you know getting caught up in like what color is your cable and and does that should I invest in a particular company but again just looking at this beautiful picture of all these well-managed cables is just you know still emphasizes like how much information is getting sent back and forth just seeing seeing all these pipes. It's crazy.
