The constraint, the constraint is no longer compute, but it is instead the networking that underlies all of compute today. That is the bottleneck that needs to be solved. And that is what Google's innovations now address.
Hello everyone, welcome to another Semi Doped podcast. I'm Austin Lyons of Chipstrat and with me is Vik Sekar from Vik's newsletter. Today we're going to talk all things Google, TPUs, networking, silicon. Yes. And Vik is going to try sharing his screen. So if you're listening to this, you might want to watch it on YouTube.
Yeah. This is our first attempt at sharing video because like there's so much developments on networking in the Google announcements recently in their whole TPU architecture and how they deal with training and inference differently. Not just from a chip side which we'll talk about but also from the networking side. It's easy to talk about chips in the sense of oh look there it has so much RAM it has so many flops. It's very difficult to talk about networking without showing a picture.
So I've like pasted a bunch of pictures on Google Slides. I'm just going to share that. It's very rough. This is not going to be seriously edited or professional or yet yet yet. Maybe we'll get there one day. But anyway, it's I think it's useful to show some pictures. So definitely watch it on YouTube if you can.
Yes. So I'm excited. I think there's a lot to learn there. I'll set the stage for listeners. So Google had at their Google Cloud Next 2026, they had their keynote yesterday. And in it, they announced the next version of the TPU. And of course, the most exciting thing that most listeners probably have heard is that there's not just one TPU. There's actually two. TPU version 8. There's 8T which is the training chip and 8i which is the inference chip. This is interesting in that historically it's for for quite some time with the TPU there's just been one chip. V1 was like an inference-only chip. Then V2 was training and serving. V3 V4 was training and serving. V5 they split out this like efficiency chip versus a performance chip. V6 was back to one chip. V7 was one chip kind of marketed very heavily as an inference chip, but here we are now. We've got a specific training chip or training system as we'll get into and a specific inference chip. Maybe I should point out the elephant in the room right away. The TPU8 has like 384 MB of SRAM, which is like three times as much as the 8, which is for training. That was like what? That's I mean it makes sense. Why not put SRAM? You know, Groq LPUs that Nvidia is now integrating into their platform is basically SRAM. And why not do that right now in the TPU8 because, you know, put in as much SRAM as you can so that you can get your low latency inferencing from SRAM right there. Get high throughput, faster tokens. So, that's like one obvious thing to do, which is put in more SRAM. So it's a good decision but you know it's explicit right now after seeing how we're going for like fast decoding.
Yes. Yes. And this aligns with when I talked to Rayner Pope from MatX, you know, he talked about how they do weights in SRAM and KV cache in HBM and these are architectural decisions to try to get both like really fast decode but also all the context you need. So we see you see see Google going this direction with three times as much SRAM but then also 288 gigs of HBM on the inference chip. And I will point out one thing that I thought was interesting which is the training chip has less HBM. It's 216 gigs. By the way listeners, people watching, Google has amazing technical blogs for these chips and they talked a lot about it. So they kind of talk about why they made these trade-offs. But I think it was interesting because right away in my head I'm seeing for training if you're going to have less HBM. You could sort of frame that as like why overpay for HBM you don't need. So for example if someone's just selling one chip like a GPU that's like hey one size-fits-all. Use it for training use it for inference. You could say well hey look what Google's doing. Maybe am I overpaying for expensive HBM on a training chip that maybe I don't actually need it.
Yeah, I was wondering about why the training chip has lower HBM and I think the inference part of it is obviously now so memory hungry that you want to put in as many fast memory tiers as and as much of it as you possibly can because the decoding and the inference gets like so much faster with much better memory throughput that the inference chip is simply you know peaked it's it's maxed out with the best memory memory you can ever put down in the current environment, the current scenario. This is what you can do. Why is HBM lower in training? I guess you could always put more chips together if you don't have enough memory. You could just like add more GPUs and your cluster total GPU memory and the cluster gets bigger. So I guess that's the solution to it. But in inference, it's it only makes sense to max out every tier. You know, remember Nvidia's like memory tier where you have like SRAM right up. They don't actually mention SRAM. It should have been there, but it should have been like tier zero and HBM was tier one and you know, DRAM was tier two. This is an effort at maximizing the fastest memory. Makes sense. Yeah.
Okay. What else was interesting? Both chips do use the ARM Axion, so Google's ARM-based custom CPU as the head node CPU. And I think there was a quote here. I'll read it. It said in the announcement blog, TPU introduces two distinct systems, TPU8 and AI. These new systems are key components of Google Cloud's AI hypercompute. So, yet another interesting marketing name of of course again though hints that it's it's a whole data center these days. You're not just thinking at the chip level. An integrated supercomputing architecture that combines hardware, software, and networking to power the full AI life cycle. While both systems share the core DNA of Google's AI stack and support the full AI life cycle, each is built to address distinct bottlenecks and optimize efficiency for critical stages of development. Additionally, by integrating ARM-based Axion CPU headers across our eighth gen TPU system, we've removed the host bottleneck caused by data preparation latency. Axion provides the compute headroom to handle complex data prep-processing and orchestration so that TPUs stay fed and don't stall. So they're really framing these as the CPUs to feed the TPUs.
Yeah, that's what is important. Never let the GPU stay idle.
That's what it so also points out that ARM is a is a very viable alternative to any x86 CPUs. now not to be ruled out at all in terms of ISA, you know, architecture. So yeah, I think they're all equal, fair game now. And like we mentioned in an earlier episode, the best CPU is the one you can lay your hands on. So just whoever can deliver most of them, they'll make it work. ARM or x86, that's the name of the game right now.
Okay, now let's talk let's talk networking. Where where should we start? I noticed that they've got this nice little table and they're showing 8T uses a 3D Torus network topology, but AI uses the boardfly topology. And so this is interesting because not only are we now starting to make different SKUs for different workloads, but we're starting to have different network topologies for different workloads. So that's super interesting. So take us take us there.
Yeah. So in terms of the chips that you know we just spoke about themselves they're cool I mean they're good chips they have HBM they have SRAM this is all like standard stuff in some sense now I think the real innovation here and the it is a tectonic shift in how networking is implemented in the future of Google data centers there is a big change and this is a change that happens once in a decade kind of a change and so I'll explain what that is but for that I need to explain what was there before. Right? So the fundamental assumption and the basic need for scaling AI today is the is to realize that the constraint the constraint is no longer compute but it is instead the networking that underlies all of compute today. That is the bottleneck that needs to be solved and that is what Google's innovations now address. Google has now reimagined the data center in a sense and as we talk about it we'll kind of break down why that is because they have different chips as we just mentioned for training and inference they have different networking architectures for training and inference as well and we'll talk about why that is and what is most interesting is that a lot of this networking is going towards optics and you will hear me talk about like OCS a lot which stands for optical circuit switching right And I want to like explain what that is right up front so that you know when I keep saying this word while talking about networking and I will say it a lot that people understand what it what this actually is. So optical circuit switching is a way to redirect light from one port in a switch to another port in a switch so that you can connect one GPU here or to another GPU here or TPUs in this case and you do it entirely in the optical domain. And the idea is very simple. It's just like holding up a mirror and reflecting the sunlight from coming in from your window. You know how you can point it to different spots in your on your wall or something? That's exactly the theory of like the concept behind OCS. Why convert optics to electronics and like do silicon packet switching like all the Tomahawk switches do. Just stay in light. You're already in the optical domain. Stay in light. So that's what the OCS so this is optical circuit switching. you can simply connect one port to another by changing how you shine light into different ports. So that's what OCS is. Okay. And so that's becoming the substrate on which Google's networking is built today. And I'll explain what parts of the network too. So the first thing we should talk about was what Google called their Virgo networking solution. Their previous network was actually called Jupiter and this is from 2015 and was at that time it was the industry's first like petascale network like nobody had ever seen anything before that at that scale. It was pretty fantastic because and you know it was built primarily for the internet era because that's what was driving everything and data centers relies you know relied on a Clos network which is basically have these racks and you can have different levels of networking switches. This is this is when I have to kind of experiment with with a picture because I actually have a picture of what a Clos network looks like. And yeah, so I hope this works.
While he's pulling that up for people listening, Clos is spelled C-L-O-S.
You might not know that when you're listening if you're trying to Google it yourself.
Yes. So Clos network. It looks something like this. So basically there are these racks and then you have the first layer called a leaf switch that is the racks hook up to and then at the next layer is what is called a spine switch then you have have another layer on top called a super spine switch and so what ends up happening is that if you have to go from like one rack to another rack or one GPU to another GPU you know between different pods or whatever you can see this picture if you're looking at it on YouTube you'll see there are two pods and basically You have to go through a lot of network hops. Think about it. You have to go from the GPU in the rack to the leaf switch. From the leaf switch, you go one higher level to the spine switch. From the spine switch, you go to the super spine switch. And then you get switched into the right super spine switch. And then you come back all the way down the hierarchy. This is too many hops. Okay. And when you have to go from one GPU at one end of the data center to the other end of the data center, it is it is too much. Okay. So that's the whole thing here and that's why we don't it doesn't work for the AI era and the reason
Right.
This this was definitely designed and made a lot of sense in like the cloud server web app era you're hitting a database over here you're hitting an API API over there but it's not that you need every leaf talking to all the other leaves and having to hop way up and way back down to do that.
Yeah, it happens sometimes and the latency is kind of you know undeterministic. It happens. Latency is what it is and the problem is training networks don't like that and like AI does not like random latencies and all that. And another reason I think this is very important that this Clos network has so many hops and you need so many layers of networking switches is because each switch does not have enough ports. In networking terminology that is called the switch radix. So when you say a switch doesn't have enough ports it's just another way of saying that is like yeah it's a low radix switch. So when you have a low radix switch you need to multiply the number of ports by adding layers on top. Okay. So that's why you need so many layers. Now the problem has changed like we have really high radix switches and we'll come into why that is. Now all of this communication was managed in a data center in the Jupiter network of Google's Jupiter network by their software layer called Orion. As this internet era grew really in the 2020 around 2022, Google introduced optical circuit switching into the data center because it actually has a large switch radix. You can I think if you see Lumentum's OCS switch and even Coherent, you'll see that has like 300 by 300 ports. It's like a lot of switches actually switch ports or high radix switch and so these were like also switched to optics now it's not doing silicon switching anymore so when they started doing like optical switching and they also do started doing wavelength division multiplexing which means they sent multiple wavelengths so all that increased the bandwidth so around 2022 it was like six petabit per second speeds or six petabits a petabit is like 1,000 tera so you're talking about like 6,000 terabits per second. That was like 2022. And then they went to like faster networking speeds like going to 400 gig networking. Made it like a staggering like 13.1 you know petabits per second. That's 13,100 terabits per second. That's a lot. remember this number like 13.1 petabits because later when I tell you about Virgo, I'm going to tell you what the number is and you'll see how much how amazing that is, right? So yeah, it's from 2015 to 2023 it's grown 13 times. That's pretty cool, right? Like and you know they've been growing it pretty quickly. So this was great. It worked great for internet, YouTube videos, web search, it's it's great. But when AI showed up, when you're training a trillion parameter model with highly synchronous traffic, you see the internet does not work that way. Internet is like I don't know, Austin is checking it in his time zone. I'm using the internet on my time zone. Maybe there are like spikes because everybody's watching a sporting event at some particular time. But those tend to be regional and you know it's it's okay. It's kind of an asynchronous thing really. Internet is an asynchronous really for the most part. AI does not like that. It's like when all of it hits AI at the same time, the latencies go crazy and it's always limited by the slowest deer in the pack. You know, the highest latency is what is going to cause problems and that is what is called like tail latency.
The highest latency is the limiting factor.
So that is the Jupiter network. That is what was the status of the world until like Google changes everything.
Yeah. And so Okay. Okay, so they had a network and it was made more for the previous era of like cloud unpredictable network. And by the way, it's not all just one big job like AI training. You're just running one job across the network. It's coordination. It's all the machines are contributing to one big job. But obviously the Jupiter network was not designed for that era. It was highly distributed. Tons of jobs running at the same time. Even if there was some pattern to the traffic, it's just highly distributed. And then to your point with training, not only is it all coordinated, but it's coordinated in such a way that there's this tail latency concept where everyone has to do work and update everyone else. And if there's a straggler, the whole bus has to wait for that one straggler.
Yes. That's exactly the word they use in their blog post, too. So stragglers are bad.
Yeah. Yes. Yeah. All right. So, let's now talk about a new network called Virgo. They call it the Virgo mega scale network. So fancy. It is actually a mega scale network and you'll see why. So Virgo is basically designed for AI. They saw the network like no this internet era stuff doesn't work anymore. We need to redesign this to actually make it work. And the biggest change here is that they reimagined each part of the network for what it is. And this is why you know need a picture again from their blog which I am going to put up if you're watching it on YouTube. It really helps but I will try to explain it without having the visuals as well. So essentially what happens is that you have three layers of their networking stack which is first one is the scale up and we'll talk about scale up a little separately but you know you got the scale up network okay you got it within a pod and you know however the pod is hooked up the Virgo fabric itself where the interconnect happens is in the scale out also called the backend network so this is where is an east west connection where you hook up all the racks in a data center these TPUs to act as one big AI hypercompute. And so that's the whole scale out network. And then you've got
A tiny bit of nuance there for people listening. Scale up. You're trying to make all the TPUs act as one and they're sharing their memory. So it's like memory coherence. So it's like they're all talking at like as fast of latency that it seems like all their HBM is shared. And then with scale out from a training perspective, we're still trying to make one massive training computer, but everything on the scale out network is is not sharing memory. Now they're they're talking as a big coordinated system, but not as tightly coupled. And I know that and everyone when it comes to GPUs, they think, oh, scale up is within the rack and scale out is many racks. And that's mostly the case, although you can have scale up in between like side-by-side racks. But TPUs are a bit of a different beast because even on their scale-up domain, they'll have lots and lots of TPUs, more than just a rack. Did I did I get that right?
Yes. So TPUs go to thousands actually. But in terms of memory coherency, there are some improvements that I will mention next and that is very it's a combination of how the latency between memory can actually be reduced as well. And that's an innovation here. Yeah, actually that actually brings me to the pretty much explaining the last part of this networking thing here which is their front-end network. The front end network is simply I don't know compute and storage or connecting to the internet. This is not like fancy networking. Okay. So you can use the Jupiter network for this. So you can use like the leaf spine topology, the Clos networking thing that we spoke about. That's fine. don't need to reinvent every part of the network stack just reinvent what is required right yeah so that's that's how this basically looks so the Virgo network this is what we should talk about now they collapse it entirely to a two-layer network because they have these high radix switches which are all OCS by the way and they have I think 300 by 300 ports if they using Lumentum's OCS the future of OCS is going to scale to 1,000 by 1,000 ports so you Imagine you can even flatten the network further. But regardless, I don't know if Coherent is an OCS provider to Google as well. Things I don't really know. But anyway, all this technology exists from these companies and a viable users in these data centers. So OCS used to be optional before, but now it has become an integral part of Google's data center networking approach. And the way you can connect with OCS because it has so many ports and you can connect it in simply two layers means you don't have all those hops.
You can get from one TPU to another TPU with like going across two network layers. That was that makes it really really fast. And not only that, because you have this high radix switches, you can connect 1,000 what what was the number I have here? 24,000 TPUs to all act as one in the data center.
That's insane. So they call it campus as a computer. It's crazy.
Yeah. Whole campus is a computer. You You're staring at a whole computer.
And you want to know what the aggregate bandwidth of this whole network now is?
What what what was it before? like 13 petabits or something. I don't you told us to remember and I don't remember.
Yes. 13.1 petabits per second.
Now it's 47 petabits per second.
That's like 4x. Okay. That's like 4x faster.
So that's one reason is it it's optics entirely. No silicon based switching really in the backend network. and they have re rearch rearchitected a bunch of things that made makes this happen. So it's really amazing. And now you know what happens when you put 134,000 chips together in a campus as a computer is that stuff breaks all the time. And so they have an enormous amount of telemetry built in so that they can continuously monitor these things and keep their goodput high. Goodput is like all you know the the stuff when it's actually working. It's not just throughput. Throughput doesn't when you say throughput, it's just like yeah, the CPU is capable of making flops, but does that actually generate enough tokens or do what it's supposed to?
The good good terminology nowadays is to be like, okay, the the working throughput.
Yes. Yes. That's good. That's actually good. Yes. It's not just like the theoretical max. It's what actually happens in practice.
Yes. Now, I should quickly hit upon that memory thing you mentioned, right? They have this thing called TPU Direct now. And what that is is basically remote direct memory access or RDMA. RDMA as a technology has been around for a while and in Nvidia land it has also been called GPU Direct. So TPU Direct is like an evolution of that concept I suppose. But the idea is like fairly simple and it's honestly this is not a like a fantastic innovation or anything. It's been around for a while. So, so not new but it's been implemented in TPUs now. So, what this is is that before if TPU1 had to access the memory of TPU2, TPU has to go through the host CPU
Neighboring thing of the neighboring TPU and then it has to the host CPU will like interact with the DRAM and like interact with the network interface card, the networking stack and all of them will like have conversations. So, what do you want to access the memory? Cool. Which access memory do you want to do? And the CPU is involved and all of this stuff. And then it'll go and tell the destination TPU, the CPU of that TPU. It's like, hey, like this guy wants to access your memory. Would you allow me to do this? And the the destination CPU would be like, yeah, yeah, fine. Let's let's do this. Let's all make it happen. And then finally, you get to the HBM of the destination TPU. So the memory coherency like you're saying is like has so many handshakes. It's like so many middlemen.
