That's a lot of compute. That's the verge of science fiction in my mind.
Hello listeners, welcome to another episode of The Circuit. I am Ben Beharin.
Greetings programs. I'm Jay Goldberg.
We are joined by the famous Austin Lyons on my team here at Career Strategies who also writes under the brand Chipstrat. Welcome Austin to the show again.
Hey guys, thanks for having me. Always fun to be here.
So on last week's show, Austin, I asked Jay the entertaining question: is The Circuit becoming a networking show? Because so much of our conversations has moved to networking and that will be the topic of today. We're going to dive deep in networking. Maybe just briefly, Austin, you've written a number of pieces this year. Maybe just start with some of the topics you've covered at Chipstrat and then we'll jump into kind of just let's lay the ground. Let's do a state of networking now here end of 2025.
Sure. Totally. So at Chipstrat, I have some very basic articles walking people from the state of what networking used to be. Like when you think about the rise of cloud computing and companies like Netflix or AWS and those type of networking environments. Now when people talk about networking, it's really about AI networking or GPU-to-GPU networking. So, I have a huge training run and that used to be a thousand GPUs and then 10,000 GPUs and now hundreds of thousands of GPUs. How do I interconnect all of those specifically for AI training? And so, yeah, definitely have some intro articles that kind of cover that shift and then of course talking through, you know, what does scale up mean in this context? What does scale out mean? And, you know, recently now there's a new term, scale across.
Yes. The state of the state of all right. So, a lot's changed, right? We, as Jay and I have always joked, like when Ben got bored with networking, it was because we were talking about packet switching, which even Jay likes packet switching, but it bores me to death. But the topology is changing as a whole. You know, my argument for this whole large shift and really just where we're going with new AI infrastructure has been this is much more computationally, you know, complex workloads. This isn't necessarily just connecting a computer to a computer or an individual to a computer, right, over the internet to a mainframe or a server. Like this is really talking about new types of compute workloads. And then again like you said some definitions we'll talk about about distributing those compute workloads now across a whole facet of scale of GPUs. So.
Yeah, so I was going to say I think you're... there has been this, you're right, there's this big change in what's taken place with networking because for many years networking got very stable. Ben might say boring.
Stale or stable? Which one did you say, Jay?
Take your pick. Take your pick.
Yes. And right because we had the internet boom in the 90s and that sort of catalyzed a big shift in how we had built networks for 100 years around telephony. We got data networks that just got more advanced and more complicated, but it was still basically the same structure that we'd used for 20 years in the internet. Right. And the cloud companies all came on and were using sort of typical topologies that you'd see in a network. And it got sort of complicated. You had to do complicated things around the data center. There's a lot of focus for the last five, six, well last 10 years companies like Cisco and Juniper and Arista in particular exploring these themes of how to, you know, move, communicate from one server to the next and then out to the internet and it was very much like a pyramid, very concentrated sort of north-south traffic we say in the data center from the server up to the internet and then out to the individual users. And that made sense. And we don't talk about all the old topologies and leaf and spine and CLOS fabric and stuff like that. But the point is that that was a pretty well tested and it was getting more advanced and more complicated but just sort of scaling existing things that had been there for a long time. We're just getting them higher capacity. Something pretty important, profound is now taking place in AI because largely of how we're training AI models. And just the size of these models and how you have to do the math around, you know, matrix math requires lots and lots of compute nodes to communicate as efficiently as possible together. And that's radically shifting. It's sort of forcing companies to shift how we do networking, right? So that's where we were and now we're at this transition point.
Yep. And I'll just add on there, you know, with sort of like the cloud networking, enterprise networking, it was all about tons of concurrent users talking to lots of different servers. So, you know, high radix, huge ports, tons of traffic, and then just filtering them down to like here's our database server, here's our API server, that kind of thing. Now, with these GPU training, it's a single workload spread across thousands of servers. So not like thousands of users, thousands of workloads running on thousands of servers, but effectively sort of like one user, the ML team or the AI team training running one workload and distributing it across thousands of servers. So, it's actually a lot like HPC jobs where you're running, you know, you are a scientific research company or national lab and you're running one huge scientific computing job across thousands of servers.
HPC being high performance computing or supercomputers, massive computers.
Exactly. Right. So, a little bit of what's old is new.
Yes. Which have historically run on GPU-based supercomputers. So hey, GPU still doing those things. All right so let's talk terms. One of the things that all three of us have talked about individually and offline or collaboratively has been there are definitions used now to discuss networking within a rack, networking from rack to rack, and then networking from data center to data center. So let's level set on these terms with the caveat that everybody kind of uses these terms differently in who make networking products. But I think we will as the collective, the royal we, the three of us standardize on some terms which I'm going to let Austin explain and then also maybe Austin talk about some of the components that go into these different the three things from a networking standpoint. So launch into that.
Okay. So let's start with scale up, scale out, and scale across. Okay. So, scale up. So, historically this terminology is also called like vertical scaling. So the idea of scale up is adding more machines that are all working together on the same task. And I was trying to think of analogies for people who are like less technical and coming from an agriculture background. I've got some analogies. I'm going to try them on for size with you guys. But scale up is like instead of, it's time to harvest your corn, you've got a field, it's 160 acres. Instead of running one combine, which maybe would take you eight hours to harvest it, you add a second combine to that same field and now it's going to take you four hours because you have two combines. So you're working on the same field. You have two machines working on essentially that what is the same task and they have to coordinate that small task. We are in this field. Let's get to the end of it together. You turn first and then I'll turn around you. Right? So that's scale up. That's taking one GPU and now all of a sudden throwing eight GPUs. I've scaled up to eight GPUs, you know, and eventually maybe scaling up to 72 GPUs. So that would be scaling up is sort of like adding more GPUs on the same task.
Now scale out, that's your traditional horizontal scaling. So that would be like, okay, instead of adding another combine to this field, that would say, hey, Jay, you take that field that's 20 miles away, take your combine there. So we're each doing a task and we're doing it together. We're trying to harvest all the fields together, but you go do that field over there. I'll do this field here. That would be scale out. You know, okay, Ben, now you take a third field. That's sort of scale out. And so again, with GPUs, that would be like there's a cluster over there working on part of the job. There's a cluster here. We're not working on the very small task together, but at the end of the day, you know, you're tackling one layer of the network, I'm tackling another. Or for inference, you're handling one expert, I'm handling another. And then scale across is essentially scaling out, but it's not in the same building. So, of course, if you're XAI and you've got 200,000 GPUs and you're trying to put them all in a big building, that's awesome. You might have one rack talking to another one that's 100 meters away. But at some point you're limited by power typically and/or real estate. And so you might get in a situation where you say, "Hey, I've got a data center here and I've got another data center 20 miles away and it has access to different power." Maybe, for example, your Google has data centers. For example, one's in Nebraska and one's in Iowa and they're connected across the Missouri River. And so you might have access to power in one state, access to power another, but you have a training run that you want to all work on training the big Gemini model together, right? So you need to connect those and so again scale up is more compute working on the very same task and that's typically with first it was within the server node and then it scaled up to within the rack. Scale out is horizontally scaling just adding more workers on the job and then scale across is like wow we have to scale across buildings which could be 100 meters, 500 meters, a kilometer and so on. So I'll pause there. What do you guys think?
Oh, by the way, let me quick add the scale across farming analogy would and this happens would be like I also bought a, I live in Iowa. I have a farm here. I bought a farm down in Missouri. I've got some Missouri folks down there to work on it. It's all one operation, but it's physically separate. So, okay, carry on.
Yeah. And I think like, you know, to simplify the terms, right? Because my concern with the way that a lot of companies talk about this is that they might use these terms interchangeably. And I think if you just distill it down and the farming analogy is a good one, like what Austin said was that if what they're talking about is within the same rack, right, that's one thing. If they are talking about one rack talking to another rack that which Austin describes scale out but that's rack-to-rack and then there's data center to data center so different locations and those are three fundamental approaches to in this case a network all being solved by different things technology with interconnect network technology that takes that rack to another rack and then technology that takes that entire data center and connects it to another data center. So there's a whole host of solutions and products and companies tackling any number of those problems. But that's the high-level concept of what we're talking about. Hopefully that's not terribly confusing.
So to be clear, your definition is scale up is multiple GPUs talking to each other in essentially in one rack. Scale out is racks talking to each other across the data center.
How do we define then something like this Google node with whatever 10,000 GPUs all acting in one sort of compute domain. Does that scale up or is that scale out?
Let me add one little nuance. So technically scale up shouldn't be defined necessarily with only within a rack. That's been true so far, but really it's sort of about almost like memory coherency like when you're scaling up a GPU can access the memory on another GPU. The question is why do we even have scale up? And part of it is we actually we want more memory but there's only so much HBM like on an Nvidia GPU. So if you need access to more HBM oh why not pool them and have them talk really low latency high bandwidth so that GPU A can access the HBM on GPU B. And so technically so long as GPUs can talk to each other with essentially near zero latency like hundreds of nanoseconds of latency that is considered scale up. So technically they could be in side-by-side racks connected by AEC cable for example or optics and considered scale up.
Right. Okay. And I'm not just trying to be difficult partly but also it's an important distinction because traditionally traditional networking you think of, you know, chip to chip and then rack to rack, server to server and rack to rack. But there's actually something very different here. This concept of scale up where you are connecting a very large number of compute chips that are like there thousands of them. So that we're talking about lots and lots of racks. And from a traditional networking person, that's going to be something very different than what we're talking about, which is we're connecting all these GPUs almost point to point where they're all talking to each other in a single fabric like right and to that's that's scale up. And I think that's where people get confused because it is they're physically in different racks, but they are logically the same. They are one compute unit, right?
Exactly. And that's why I think you get it gets a little bit weird, right? Because it's both north to south and east to west, but it is one compute brain. And so to Austin's first point, if you want to be able to throw a workload, one workload at a in this case scale up rack, which is just one rack, but now that rack becomes, you know, 1,000 racks, that whole workload can use all of that compute. But that's where I think people get kind of weirded out, right? And so to again to Austin's point, distributing that workload so that five racks over here talk to five racks on the other side of the data center, there's going to be latency there for and you're probably not going to do that with if it's 100 meters away, you're probably certainly not doing that with copper for example. But that would be the distributed workload, right? I'm 10 GPUs over here and I'm 10 GPUs over there versus I'm a thousand GPUs right here in one thing. Which again is what TPUs are in Google's V7 cluster that is a one super pod and we're moving to super pods which is why we're having this conversation that's probably the most interesting part of this is we're moving to kind of these new single compute pod clusters that can function as one gigantic GPU and handle a whole lot more of a compute workload right than we ever could and we're on the precipice of these varying degrees of size. Again, limitations by networking, which we'll talk about. But that's I think why this is such an exciting conversation. Not boring like packet switching or stale. Not that the internet was boring, but you know what I mean.
I did appreciate that my web pages got faster year-over-year. I do want to throw that out, but not as interesting as a workload, I think, as we're talking about now.
Yeah. Yeah, I mean it's a pretty dramatic change in how we build these things because it really requires it's this funny thing where you're ripping out a lot of equipment. You're not necessarily seeing switches like switches used to be the sort of the workhorse of all data center networking. You'd have switches all over the place and it was just like could you how cheap could you get this switch? What kind of software was it running? Right? And now in a lot of cases we don't actually see switches anymore or the switch functionality is built into the server and that's a radical radical shift and it leads to a lot of you know very different kinds of equipment and even though we're ripping out a whole class of equipment of boxes these things aren't getting cheaper they're getting much more expensive and much more complicated.
Yeah, so go ahead Austin.
And I just wanted to throw in, you know, GPU networking today is very interesting because you've got this fine-grain coordination. If you've got scale up and you've got now let's say 72 GPUs all working as if they're one, you have to do a lot of coordination still around what is each GPU doing. But then of course that's just like you said that's part of a huge, you know, maybe there's 10,000 GPUs in total so you've got these like little scale up clusters that are all working together on a task and then there's also orchestration at a higher level across all of them. But what gets super that alone is very interesting and again back to my analogy that would be like if there's three of us combining in a field and we can see each other and point and be like you turn first versus just globally picking up the phone and calling the person 20 miles away and just saying like hey did you get done come to my field that's a little bit the difference between like scale up and scale out. But what's going to be continuing to be super interesting is with inference now when you have agentic AI and so you've got stuff running on GPUs doing inference maybe token like or like reasoning so like lots of tokens and it's running for a while but then you've got agents that are running on CPUs and starting to coordinate between different agents you're going to have like this different plane of networking that's going to get super interesting probably a conversation for another day but you got CPUs you could potentially offload it DPUs and so I think it will only get more interesting as we go.
Yes, agree. Very very gnarly problems to solve and while this has typically been a very hardware driven conversation, it boggles my mind the software orchestration that goes into this. Like I know some of it having to look at hypervisors and Kubernetes and even some dedicated chips that do it. But man, like we're talking about just the basic orchestration of GPUs with interconnect, let alone what Austin just described, which is oh, let's coordinate 30 miles away. You're going to do that workload and we're all going to turn at the same time and we're going to be like I don't know that it just the software side of this gets crazy interesting, too. Well, I know when OpenAI launched, I think it was whatever GPT-5, they introduced this concept of how do you sort of moving between different models and they call that a router and I thought that was hilarious because it's a software router, but they're like stealing a networking term, right? And I know that's kind of just marketing on their part, but to me that's the point is a lot of what's taking place here sort of moves to the software domain things that were traditionally done by purpose-built equipment.
For sure. And of course, where are those different models running? Are they on different racks or different clusters? So like maybe there is some physical routing going on.
Oh man. All right, let's talk about cables. Let's talk about copper. Now, I want to also throw this out because I haven't said that before that there is also an we talk about bottlenecks. We talk about shortages a lot. There's apparently a copper shortage coming because not only are we wiring copper like crazy and not only are we in the future going to be wrapping all forms of components in a rack in copper to liquid cool them. I just find it interesting that like you see all this data coming up that like hey the price of copper is rising the demand's coming like that's just that's interesting right so now we all know and Jay and I have talked about this many times on this podcast right copper is going to live as long as it possibly can like it's going to do everything it can to live forever and that works like you know you always talk about people and you're like hey you know where are we at in terms of like you know copper's life, right? Because again, right, it's historically been cheap. That might change some, but I don't think it's going to become insanely expensive despite the crunch. But like you constantly see like, oh, we just did another breakthrough and how far we can push copper, right, in terms of its limits. So let's talk about the different approaches. One we do like to talk about a lot is obviously active electrical, which is interesting in itself, right? Not passive electrical, but there's a lot going on in active electrical, which is actually helping extend the life of copper. But let's talk about wires. I know you have done stuff on active electrical and did a good primer on Credo for anybody who didn't see that. So let's start with kind of the cabling trends themselves for in this case we would be talking somewhat rack to rack somewhat within the rack and a little bit rack to rack because there's distance. But I'll let you expand.
Yep. So, you know, at a high level, the rule to live by is copper when you can and optics when you must. So if you think about a passive copper cable, ideally you would just connect two chips or two servers with a passive copper cable. So, just electrical wire travels right across it. It's cheap. It doesn't take any extra power. There's just signal traveling across a cable. The problem is today we want to communicate more and more information and a very effective way of doing that to increase your bandwidth is to send information faster and that's how you ultimately get right if you have cars going through you know down the interstate at 30 miles an hour you can only get so many cars through but if they're going 300 miles an hour you can get a lot more cars through right so that's what we're trying to do is always send more information along a wire well when you at higher and higher frequencies along copper, you get this thing called the skin effect where electricity essentially only like instead of being like a round conductor like at low frequency, it actually it's almost like a donut like where the electrical signal sort of travels essentially kind of along the surface. The faster and faster you go like the more and more travels along the surface and so essentially you get higher resistance which just means you get a degraded signal. So all that to say which if you go to my Credo page I have some diagrams and stuff but at the highest level the faster you send a signal down a copper wire the more degradation and essentially problems you have.
So the simple let me extend your analogy. You're sending all these cars down the road at 300 miles an hour. You're more likely to have accidents and the accidents are also much more catastrophic.
