The memory is the biggest problem. There is literally the LLM are a memory heavy method and memory is everything. So one of the biggest problems that people should solve is like how do you get the same performance without using as much memory. You know once you solve that problem a lot of things fall into place, right?
Optics was just like for long haul links, you know, metro links, undersea cables, some applications, whatever, but never has it taken a prominence like it has today. It is the technology that is required for AI and the future of the buildout and the entire world is scrambling for it because they're like, look, we were never prepared for this demand. Like this industry has never been built for this level of demand. You are asking for too much. So everybody's in short supply because of this.
What is the useful amount of work done per token? Like do I always need 2,000 tokens per second? I don't. Like you know I'm talking to you right now, but my AI could be crunching some PDFs that I want to read later or something like that. I don't have any immediate need for it.
Awesome. Cool. Well, Vik, thank you so much for joining me. Very excited. I appreciate you coming on the podcast. I've been following your work for some time and very impressed and helped me personally kind of come up to zero to one as fast as possible. So, wanted to reach out and see if I could get you on the podcast because I think you do a very good job of explaining technical concepts in ways that people can understand and having known lots of engineers, it's a tough skill. So, I appreciate all the work that you've put in to be able to teach others.
Oh, thank you. Thank you for having me on. Always happy to sit down and talk tech and how things work is basically where my head usually is at. But yeah, like I'm always happy to talk about this stuff, very even more happy to hear that you know my explanations and writings and talking on Semi Doped and my newsletter on Substack, Vik's Newsletter, has all been resonating with people who are consuming my content. So like very happy to hear that.
Yes. And for the people that aren't aware, I'll link all of Vik's Substack, podcast, everything that he's been working on, including his institutional research. Please reach out to him because I think he does a great job. And I'm going to have to reach out to you afterwards for better podcast studio setup. Maybe I can get my own Frictionless logo in the background.
Yeah, why not? It's nice.
Maybe a great place to start is obviously the industry has really gone it feels like zero to one or maybe better term zero to 100 since Opus 4.5, really Q4 of last year. We started moving off kind of both subscriptions to API based and that kind of kickstarted the whole race it seemed like. And now everybody is really talking about agents, which I think is very fascinating, just computer use, being able to do things for more normal people outside of software engineering and coding. And we kind of have a new race that gets kicked off. So maybe to like open a new question to start off the podcast is just how are you seeing the agent landscape and what are your broad thoughts?
I really like the current landscape of agents a lot, more so than I've ever liked it since the beginning of this year, which is when like agents really took off. In the beginning of the year, it was more like, oh, how do we step away from the chat interface and maybe get it to work from the phone for the first time? Like literally when OpenClaw came out, this is literally when you could like text from your phone and you could get responses, intelligent things happening, notifications, all of that stuff. But the problem with that approach, at least earlier this year, was that it was very difficult for quote unquote regular people to go and set this up. And I'm not just saying like the regular non-technical person. I tried setting up OpenClaw. I still have a home server that's running it.
Yeah, it's not straightforward, you know. It's like it's pretty hard. I spun up my own VM. There were all these security concerns about giving OpenClaw access to your whole computer, right? Like so people are like, no, no, no. Don't install it on your laptop. It's way too intrusive. Go get a different computer. So everybody went and bought a Mac Mini. You remember that whole thing?
Yeah. Yep. Yep. To me, that was super funny, just given that most of the models that people would run in my opinion were not going to be local models. Even though people went out and bought spec'd out MacBooks, Minis, trying to get as much unified RAM as possible, but then they still do API calls to data centers.
Yeah, the whole idea was that you can run some kind of a local model that at least handles your personal tasks or something like that. But again, it's like a very expensive proposition because models are getting like cheaper to use over time. But you have to make this upfront investment of like several thousand dollars hoping that the local model you will deploy on your allocation of RAM would actually do the job. It's not a very long-lasting bet. You'd rather bet on token pricing going down, being able to use frontier models for cheaper over time, right?
Yeah, fully. Maybe an interesting place to start pulling on that thread is obviously as agents are starting to get in popularity. We had OpenClaw kick this all off and then you kind of saw Hermes model come out and both were I think each iteration is probably a little bit more user friendly than the next. Then Grokbot came out, then you had Muse. What is kind of your take on what it actually takes to run these things? Because it has been interesting that people went out try to buy a bunch of Mac Minis and to your point, the capex for a normal person was high.
And that doesn't normally seem like the path forward where you ask the kind of everyday person, so to speak, to go out and spend $5,000 to $10,000 to have your personal assistant. But as these things start to scale, obviously that affects data centers and kind of the data center buildouts. What are you kind of broadly seeing under the hood when you kind of peel this back if these agents really start scaling to tens of millions, hundreds of millions and billions of people?
Yeah. So the evolution is very interesting because when people went and bought out these Mac Minis, it was important to ask the question like why were they buying them? The reason was not always because you could run local models. It's just that they wanted a separate environment where they don't have to have their personal information. So on that Mac Mini, you could just keep the stuff that agent can access. Even if you're using the cloud models, you could just keep stuff like never log into your email in that computer for example, right? Never log into your bank account. So because you don't know what this agent is going to do, nobody was speaking about guardrails like earlier this year. So you just like give it its own sandbox environment. It's like, you know, go crazy within this. Your damage is going to be limited. You're not going to blow up my life. You know, you're going to keep it contained in this box. If stuff goes really south, I just like wipe this box and everything is normal again.
But then the immediate thing people said like look, you don't have to spend hundreds of dollars to buy a Mac Mini, just go get a cloud virtual machine. So people were like okay let me go just like spin these up, you know, so all these talks about like all these like digital lotion and all these kinds of companies went up because everybody was like spinning up virtual machines in the cloud. Still like all of this is difficult to do, like first you have to buy a machine then you have to get OpenClaw installed and now you have to get a virtual machine installed and then you get OpenClaw on that virtual machine. So the next step of the evolution these companies said is like wait why don't I do all of that for you, like I'll give you the virtual machine. I'll install the agent, but I don't want to install OpenClaw. I'll install something that my company is developing.
So, Grokbot was like the first one I tried. So, as soon as it came up, I signed up for $200 and like I'm going to use it right away. So, it was fun because you could literally spin up agents by talking to it. It had these cute icons and this GUI was nice. I'm like, this is sweet. Like, I don't have to do any of that stuff I tried to do with like OpenClaw. This is amazing. So in minutes like I just downloaded it, my credit card was like already on file. So it's like bye-bye $200 and then like my agents are running. It's awesome, right? So it's become accessible now way more than it ever was. And that's where we are now.
And that has its implications on what actual hardware we would need to run all these things, right? Because now what people were going and buying, these companies like Grok now Muse too. Muse runs two CPUs and like 8 GB of RAM supposedly. This is what people are saying is running behind every agent that you sign up to use. But all of these are like kept somewhere. There's a computer running on your behalf somewhere in the world, right? Doing your stuff for you. So all this translates into like demand for CPUs, demand for RAM, demand for more inference because all these machines are now going to be running inference without you asking for anything. That's the whole point of an agent. It's going to run its own inference and tell you what matters when it matters. So all of these things are going to go up. All the real question is relative amounts, right? Like what is more, what is less. Yeah, but all of them should go up.
Yeah, it is interesting. When Grok kind of got his own virtual machine and was able to kind of start doing computer use on the back end instead of tapping into your own, it did kind of feel like a magical moment even though that you could set this up by yourself. It's just to your point a bunch of extra steps that were a little bit more cumbersome.
Yeah. And I think to me the interesting thing, and I think Elon's talked about this quite a bit, is on the computer side, is there's a bunch of software that has been invented in the world that does not have APIs or MCP connections. And if you can use it like a normal human, that unlocks a lot of functionality. And it's been amazing even more recently just with Astra and kind of the advances there on the computer use side. I've been pleasantly surprised. It's still not obviously at the human level, but it seems like the agents in each iteration of the model is getting better and better at just doing things on your behalf.
It's amazing. I have a story around this. So when there's another agent thing that we haven't mentioned yet called Instinct which is like gathering a lot of attention right now, a lot of funding too, like even like Code2 actually on their X account posted about Instinct today. It's very exciting technology and I managed to get my hands on an invite pretty early so I was like okay, you know, I'm going to try it, like I have to try it for the sake of science. Let me log in, you know, if my stuff is going to be hacked, I'll deal with it later. I'm like YOLO this whole thing now. So I like signed up. I gave it my access and then I got this registration email for a conference. Hey, you know, Open Compute Project Global Summit is open and you can register here, you know, here is the code and all of that stuff.
So I was doing something else and I told Instinct, yeah fine, do it, you do it, like sign me up, you know, just do it. And then I don't know, I forgot about it. Went back to doing what I was doing. And I came back to it. I got a message on my phone because I was running it through my iPhone and it said, oh yeah, you've been signed up. Here's the QR code. And oh, by the way, I ordered a t-shirt for you. Medium size. That should be okay, right? I'm like, how the hell did you know my t-shirt size? Like, that worked out great. Like, it did everything and even got my t-shirt size right. So, that might have been a lucky guess, but I'm like, this is, I'm in. If something can go and do all that for me, I'm in.
Yeah, I think obviously the zero to one with the software engineering side, I mean that was kind of a slow iteration over time, but I am excited now that more people can interact with AI outside of just the chat bots and LLMs. And it does seem like agents and kind of thinking about it in the limit, hopefully in the good sense of like Her the movie, where it's like the agent just like knows everything about you. You can do things, maybe you have a personal relationship with your agent, but definitely seems like that is the direction that we're heading.
Now, how to actually make that work under the hood from both a kind of how many gigawatts and how large of a model do you need versus like how much memory do you need and do you need kind of one CPU, do you need many? I think is also an interesting question for people kind of following the space and trying to go deeper on some of the harder side of things.
Yeah, I have a take on this at least to at the very least. I don't know if I'm right, but I'll say it. So, for most people's tasks, I think day-to-day, right? We don't need frontier intelligence. We really don't need to run, you know, the heavy models with like extra reasoning set and all that. So in general I think a lot of these models that run personal assistance can do with I don't know Sonnet level models or even if you go open source, I don't know, the cost could be even lower to run these things.
And from the inference wise, I think the token count is going to go up but it's not like everybody's going to be running the frontier model. So that has its own consequences, right. So maybe we can accommodate more users running like Sonnet level all the time compared to everybody trying to run frontier models all the time. There's a big difference on the infrastructure needs between these two cases. And I think personal agents mostly don't need frontier model. I mean you're not going to ask your personal agent like solve this gigantic math problem. I mean that's not like a personal agent kind of a job, right? Those kinds of things will continue to happen on this side. So it's going to be like mid-level models which will do just fine.
So the other thing is that the whole CPU allocation story, it would be wrong to assume that everybody who runs an agent is going to have their own CPU all the time, right? It's not like buying a Mac Mini but in the cloud. It's not that way because whenever your agent is not using it, that piece of hardware will be serving other people. So it's a shared resource across everybody. So the real question is how much sharing will happen and you know how often do you need it, like do you need your agents to be working all the time for you and doing stuff all the time for you. Or you know maybe in every hour or so it could come to you and tell you hey this is what has happened.
I already have my ChatGPT doing that for me. Every hour I get a notification on ChatGPT that says hey if you miss these important emails this one needs your attention immediately but the rest of it you can do later in the evening so I already have that set up. So it's about an hour every day, like I mean I have other stuff to do which I don't always need the agent running for me. So when it comes to hardware demand, right, like I think there are a lot of people who will get on this train now because it's easy to use, you can text, everybody knows how to text somebody, right, so you just have to text like an agent. That's all you have to do. So a lot of people can do this.
But I think the hardware demands itself is not going to be like cutting edge frontier stuff all the time. You know, it's going to be shared CPUs, lower level of inference. Memory will be like shared as well. Like maybe they will decommission your memory when you're not using it and spin up your computer just in time to use it, you know. So a lot of this is going to be shared. So a lot of the sentiment that everything is going to explode because of agents may not actually happen.
And is that because in your belief it's more shared resources and that the frontier intelligence is not necessarily required for every day-to-day task. So that you could stay constant and just onboard additional users?
Yeah, no, you don't have to stay constant. I think it'll go up still. I think we will need more compute. We always need more compute even for training. We, you know, the scaling laws haven't ended. So we can make bigger and bigger clusters, put in more and more GPUs. These things keep getting smarter which is amazing. So we're going to keep building bigger GPUs and CPU, that path has never changed.
But the incremental demand for like agents from a much broader population who would like to use these things, of course that depends on cost because right now the ChatGPT thing is going to be only for the pro, right, so I mean adoption is a questionable thing at that point. I think Muse is much much more inexpensive to use so adoption is a question but assuming a lot of people adopt it, they're going to use more compute than we did in the past.
But we have to be careful to like limit our expectations that it's not going to like completely blow up and like we are in a whole new wave. This is good because now this AI has become useful to the regular person who doing regular things like you know you could keep your kid's school calendar in check and find out you know when their trips are, when the soccer lessons are, any rescheduling is happening, that kind of stuff. Everybody can use this thing. So I think it'll be much much more useful than it has been in the past. And that's a good thing for the AI buildout, right?
100%. Yeah, to me what has been really interesting is really trying to understand what are the things that I would say the broad adoption curve because I think everybody is trying to underwrite at least in my point of view like how many new gigawatts are coming online. Is that 10 this year, is that 20, is it 30 next year, is it 40, just so whether that's pre-training to build larger and larger models or just for inference as kind of potentially more of the workload shifts to the inference side maybe that's more memory bound than GPU bound and trying to put all the pieces together.
One thing that I personally have been trying to track at a high level is just the amount of gigawatts just because I feel like that's kind of downstream of everything else or upstream so to speak and everything else kind of falls back from that. What are maybe if you feel like the space on the agent side is obviously increasing general adoption but maybe it's not a straight up line so to speak. What are the broad things that you are really following as like a broad trend because I think people are either in like one of two camps right now. It's like AI's a bubble. The buildout is insane. Like this is not sustainable. Or on the flip side, it's like you're completely AGI pilled. You're like, of course, all these gigawatts are getting built out. Inference demand is insatiable. We'll never be able to get it. And then everything else in between is a little bit tough.
So I guess within this maybe not exact spectrum, but within kind of how you're seeing the next year or even six months, where are you kind of in line there?
Yeah, that's a good question. I think I have a long-term view and a shorter term view. The long-term view for me is much clearer. I think that I would not consider myself completely AGI pilled or anything like that. But I do believe AI is a genuinely useful technology that has useful implications for the long run. So in a 10-year time span regardless of what happens, it's going to be up and to the right. You know, when the internet and the dot-com era showed up, yeah we had a crash of course and a lot of stuff happened but you know it was fundamental foundational technology that exists to this day and is a durable technology that's going to last.
Now we could argue whether this is what LLMs are actually capable of, AGI, I don't know. People say like no, this probabilistic statistical approach of intelligence really is not true intelligence and something else must come by maybe, I don't know, there are like far smarter people than me who know that stuff better. But given what AI can do today, I think it's a useful tool. Like I use it for a lot of day-to-day stuff that has unlocked so many things that is not possible otherwise, right? Otherwise I would even for what I do now between the Substack and the podcast and the institutional stuff I'm getting off the ground really is too much for like one person to do. So I do have ways in which I can streamline so many things, meetings, you know, reschedulings, all the administrative work is going away. So long term I see that this is like a very useful technology that's here to stay.
Shorter term, do I believe that we are overdoing it? That's a difficult question to answer because every six months or so something new comes up. And you know last year if you had asked me this question like December or November last year I'd be like, like what really? Like coding agents were a good thing, right? Like you know, and so yeah coding, is that all we're going to do with this thing? Okay, what about this chatbot? This chat box was giving you like half answers some time ago. It wasn't, you would never have any faith in whether it was saying the right thing or hallucinating stuff. I think a lot of those concerns have gone away today.
So like it's so even in a six-month period it's very hard to predict where the trajectory of AI is going. Then we had agents come up. Now we have agents in our phones, in our pockets, right? And now the next question is what do those phones have to do? Like is this a game for like, is edge AI finally going to be useful? Because although your device is doing some of this inference, like you're texting Instinct or Muse with your phone, but why do you have to go to the cloud every time? What if you can do some stuff on your phone, right? That's the whole next step of like what do we do with AGI.
So in the short term if I look at the financial side, it makes me very nervous. I'm like oh my god, look at the amount of buildout, look at Anthropic's S-1 and see how much commitment they have to capital deployment and to build out compute. It's incredible, the numbers like above 500 billion I believe is like insane, like they have commitments to build out that much. It's scary on the financial side but purely from a technical side I expect that, you know, I think in a year we'll be in a much better place compared to now even with this technology because we haven't even started on the adoption curve of like amazing yet. I think there's a lot more good stuff to come so I'm optimistic on the technology side, but don't ask me the financial stuff, it's like a little too much.
Are people over-ordering memory? Is everybody like, you know, buying too much and are we going to overstock and get into a glut and crash? All those things might happen. Okay, I'm not ruling it out at all. But in the meantime, I think we're still going, we still have a long way to go. So that's a long answer to your short question.
No, that's a great answer. In terms of kind of the buildout, so to speak, what things have kind of been interesting most to you? Because it does feel like obviously the leading labs are doing or investing the most and kind of the broad hyperscalers into capex. I think north of a trillion dollars at this point. And obviously they're building smarter and smarter models. At least from my standpoint it looks like as those models scale up the clusters need to be coherent in the sense that they're co-located and we've seen some attempts to do decentralized training and in large part I don't think that has played out and so continuing to focus on kind of coherent clusters that have extremely high throughput.
And going, I think Elon has even talked about building a Memphis data center end of this year. I think they're targeting 2.5 gigawatt, like 8 to 10 next year. But it does seem like as the models generally grow in terms of like total parameter counts that over time, and I think Jensen has said this on Brad Gerstner's podcast, who's like inference is going to like a million x or a billion x, and that does seem true just given that that is the source of generally all the revenue and training is really a loss leader for inference. So how are you kind of like seeing things on the ground floor because I know you're very deep on the technical side of the different buildouts both on inference and in training and kind of how that shifts the workload a little bit from more GPU heavy to more memory side which obviously the memory stocks have done fairly well this year.
Yeah. So the difference in training and inference, I think training is going to continue. We are still trying to build larger and larger world sizes. We want to put, you know, 144 GPUs together in scale up. We want to put 576 in scale up in, you know, 1,152 GPUs in scale up. All of that.
What's the max right now? 72.
Yeah. Today it's 72 in a rack. But we want to go beyond that. And we want to put 144 in a rack. But it gets really really difficult because even a 72 GPU rack today, the Blackwell rack is like a 200 kW rack. Now you want to double that, you're going to increase a lot of the networking as well. So even realistically if you just double the GPUs and you say you have north of 400 kW in a very simple calculation, that's a lot of power for a single rack and cooling is a concern, interconnects is a concern, you know, so a lot of things happen when you go into that number of GPUs in a single rack.
Now imagine putting 576 in a single rack or 1152 in a single rack, it's just not possible. Right? So now what you want to do is you want to put them in multiple racks, right? And now that's opening up so many other problems like how do you hook them up because it's not one or two feet, one or two meters anymore to cover like eight racks. You need nearly 10 m of reach and copper can't do that at the current like generation of speed at 200 Gbits per second. Copper can't do 10 m of reach at all, like no way. So that's the problem we are faced with now.
So we will continue to go to larger and larger domains and scale up because training is going to continue and is very important as we push the frontier because we have still not seen any signs of scaling laws slowing down. But yeah, inference is the real big I think money maker because everybody needs inference, right? Like we all need a model but do we all need a frontier model? Maybe, you know, in the large course of things it's a good thing to have. It's a longer project, like everybody needs to run research projects that give you the next best thing, right? Like otherwise what's the progress here? Progress has to continue so that will happen.
But inference is going to unlock like so much more useful stuff, right? Agents is just one use case. And the chips and the architectures are continuously diverging from what chips used to be used for training. Like now even in the Google TPU version 8 you have an inference chip and a training chip. Now they've been doing this since version 4 actually, but this is the first time I think they released them together and they have different chip architectures.
So you could even look at the data movements in inference and the data movements in training, they're completely different. So in training you have all these weights, you know, I like to think of them like waves, you know, in an ocean, like they come in sets and they go in sets, you know, that's how data flows. You know, you have all these operations that complete and then there's this thing called all reduce, so the entire wave of data comes back and then the next iteration completes and then it's a wave of data comes back. You know, it's a very known data flow movement.
But in inference it's like completely chaotic, like it's chaos. It's like looking at instead of the beach you're looking at like the rapids, you know, the Grand Rapids or something, you're looking at the water and going, what the hell is happening here? Because you have the mixture of experts model and what you're doing is you're not pulling all those trillion parameters or five trillion parameters, whatever. You're just pulling a few tens of billions and saying that's my expert. Now, you're going to use a different expert from me.
So, you think about it from a data center perspective, right? All of these things are like random. It's chaotic. Like weights are being pulled in all kinds of places. The right weights you need might live in the cluster over there, but then your next query might require the weights all the way in this end of the room. You know, literally that's what it is like, the weights can be stored in different ends of the room that you are asking for. So when you look at the data movement patterns, it's chaotic, right? So we will see all these things like diverging.
So inference now unlocks, because of the nature of the problem, it unlocks so many more architectures in chips that can be better suited to that, right? It's not for training, that's why you don't see too many companies out there trying to make training chips because those big GPU chips are great for training, right? Like from AMD and Nvidia and all that. But you know when you go to the inference, it's the wild west. You've got Cerebras, you've got Grok, you've got like so many approaches.
Like if you go to Hot Chips for example, you'll see like so many varieties of stuff like Microsoft's inference accelerator, Microsoft Maia is totally different. Meta's MTIA is a different like way of doing it. Of course, OpenAI's Jalapeno itself was like completely different from all of these other things. So, so many different things are happening in inference. It's like amazing. So clearly there's a reason why everybody's focused on that because there are a lot of dollar value to be unlocked there as we go out into the future.
Yeah, it's super fun. I would say in kind of my world currently within blockchains, really as if you boil it down to kind of one key thing, it's historically been pretty throughput bound. And I think the interesting thing within the AI data centers, you're generally talking about, as you put it, the weakest link across racks as like 200 gigabytes. When you look at some of the things on the crypto side, you'll laugh. It's like measured in megabytes. And I'm like, this is so cool. Like we're pushing the frontier of really what's possible across like the physical limits of these different materials and then obviously like they're scaling to terabytes and beyond which is super interesting and I'm hoping crypto and kind of blockchains on the back end will get to gigabytes and we'll keep scaling but thus far it's taken a little bit longer than I would like.
Yeah, it's insane, right? The speeds are insane. Like most people don't even have gigabit fiber to their home, you know, like your whole internet. Most people don't have gigabit fiber. Like some you could get it. I guess that's about it. Like think about the speeds in a data center. It's like insane.
It is crazy. Yeah, the amount of throughput is very high. I mean I think even like normal data centers, they have like 10 gigabyte lines and then they scale to 100. I know like maybe slight tangent, I think blockchains are mostly the next evolution of finance as like fintech and it's really just connecting all these heterogeneous databases and syncing information at like a bare level.
And so why it's a throughput issue is because you have these different, I call them banks or exchanges, that have relatively low throughput and the thing that you need to do is synchronize the data between them. So, we've been scaling from really kilobytes, which was terrible to me, and now hopefully gigabytes. And I think if you push it to the extreme, it kind of looks more like high frequency trading where you get like 100 gigabit internet connect lines. You take a bunch of data in, you do some like data parsing, and then some like interesting stuff with the models on what you want to do.
But the reason why I've been super interested in the data center side of things and why I feel like at least trying to learn as fast as I can on zero to one, and I again appreciate your help here, has been because a lot of it's like different throughput issues. And I think one thing that I wanted to touch upon in terms of what you mentioned, just like going to the scale up and then also want to touch upon the inference side of the things is I think from a rack perspective it's also very interesting.
It's like one server in a rack has a certain amount of throughput on the chip itself. And then as you go to the rack, the rack has a certain amount of throughput with things like NVLink. And then as you go from rack to rack, the throughput is much different kind of across those racks. And so maybe in more layman's terms, can you kind of explain like the scale up there and why it's difficult from like a throughput issue and even scaling up past the 72 GPUs to 144 etc. And why that has kind of been more difficult.
Yeah. So it's always easy to push bits really fast if the distance is really close. Like if you look at the bandwidth between HBM and the GPU, right? You are at something like 22 terabytes per
Second. Okay, that's like extraordinary. Very fast because whenever you push bits through copper, you know, the longer it stays in that copper line, the worse it becomes. You want to get from point A to point B and get rid of it because the longer you stay in copper, the worse the signal gets. So the same speed will not work once you go from, you know, that level of GPU to HBM, which is very, very close by, right? But then when you go between two GPUs in a rack which are spaced, I don't know, let's say a few feet apart, now you have to travel, instead of a few millimeters, you have to travel a few feet. That's pretty bad. That's a big, big dimension jump. So you can't push the same speeds. There's no way, you know. So you have to drop the speed and it just gets slower there compared to the die-to-die link. But to push the speeds anyway, you have to do all kinds of crazy design tricks where you are like, oh, I'll make sure I compensate for what happens in the copper on either end so that sometimes I know like, oh, the copper is going to lose this much energy by the time the signal reaches there. So what about if I boost the energy to begin with so that by the time it reaches there, I compensate? All these kinds of tricks, you know, or they'll say on the other side like, I'm just going to receive the bits and then I'm going to count them and see, like, make sure I can see if some kind of error calculation, like how many of these 10 bits are wrong and can I tell which of these 10 bits are wrong and how to correct them, right? So that's why you need a digital signal processor, a DSP on the other end sometimes, but you know, just so that you can compensate for all the crappy stuff that happens on copper lines. All this is a problem because it takes time for the DSP to count the bits and be like, oh, bit 7 is wrong, let's change it from zero to one, or it takes energy because you have to blow silicon and energy on that chip that is doing all these functions. So it becomes a problem. Now copper, whatever you can do, like now we're reaching 200 Gbit per second, maybe next generation is 400 Gbit per second, it's run out of reach. The physics has reached its limits, like the loss of that bit. But when you put a single electrical signal on one end of a copper line, it's almost indistinguishable, you know, like 5 m away at this speed, it's no use. You can't even tell what's happening. So that's why now the industry is like, okay, look, copper has reached its limit at 400 Gbit. Absolutely, 200 Gbit is still running. The Rubin racks still have 200 Gbit and copper, it's fine, but at the 400 Gbit it's totally dead. And even in 200 Gbit per lane, within one rack it's okay, but if you've got to go to the next rack which is like two to three, I say 2 m away this side, it doesn't have enough reach. So now everybody is like, wait, so I can't put more GPUs in a rack, there's no space, or I have to put 400 kilowatt in a rack, like that's ridiculous. It's already, by the way, in the cloud era the data center per rack power consumption was like 20, okay, 20 kilowatt. Now an AI rack consumes 10 times that and now I want to double that, just like it's very hard, right? Cooling gets hard, power delivery gets hard. So they're like, okay, the best interim solution while we figure out all this crazy stuff is let's just put separate racks next to each other and then hook them up. Think of it like a giant rack but made up of eight smaller racks, right? So that requires interconnects that reach eight racks worth of distance. Copper is not going to do it. So that is why the entire industry is like optics, optics, optics. We need to go faster. How do we hook up racks? How do we get the reach copper can't get? And so they're like, wait, I can't use traditional copper, like optical pluggable modules because those are just like plugs. You can see them online. It's just like plugs that hook in. They are energy inefficient. Like Nvidia never wanted those things or they would have gone long ago to optics. Why would you struggle with copper? And the solution exists, but it's too much power. So the solution in the industry is like, let's move this electro-optical converter module right next to the GPU or the switch and that's what is like co-packaged optics, right? Like that saves the energy problem but also solves the reach problem but introduces other problems like reliability, like when it fails, how do you change it? It's next to the GPU, you know, so many other concerns. It's never been done before, so can we deploy it at scale? Can we put 100,000 GPUs hooked up with this technology? These are all real questions, right?
Yeah, it is. It is super interesting. And I really love the physics of it all because again, you're pushing what is possible from these different materials. And even as you mentioned with copper having kind of a physical limit with just how far you can actually send the signal to further distances and potentially needing repeaters or other things there. But it's also much more energy efficient than something like fiber and the optical side. But optical now has much higher throughput and potentially uses or does use more energy. It's like, okay, how do we figure out the optical side because we want throughput to be high but we don't want the increased energy usage.
We don't want usage, we want reach. Copper was essentially free, like you just put energy through it, it goes through the copper wire, simple. Oh, optics is like so much harder, right? Like you've got to convert electrical to optical and do all of this stuff. So it is harder but it's become a necessity now and that's why you see so much optical content in data centers. It's only going to get more. Like optics is not... the industry has never seen anything like this in the optics world. In the past optics was just like for long haul links, you know, metro links, undersea cables, some applications, whatever, but never has it taken a prominence like it has today. Today it is the technology that is required for AI and the future of the buildout and the entire world is scrambling for it because they're like, look, we were never prepared for this demand. Like this industry has never been built for this level of demand. You are asking for too much. So everybody's in short supply because of this.
So you're telling me, Vic, I'm going to soon have a 1 Tbps internet connection to my house. Thanks.
No, they never give it to you. You don't need that. Only the data centers need it. You don't need it.
Unless you're running. Unless you're running training at home.
Maybe. Maybe. That's funny. Yeah, it is super interesting. Yeah, I again, I think the coolest thing that I have found is that because the demand is so high and people's willingness to push these has really gotten to the physical limits of the material. It's very exciting because then you have to come up with clever and new different engineering solutions to make these things all work.
Yeah. Yeah. So maybe touching upon the training thing a little bit more on the maybe interconnects and then we can jump gears to the inference side of things because I do think that's super interesting as well. As you're starting to scale up these interconnects, it seems like people want to almost have like one giant GPU if possible and the...
Obviously if you couldn't have the throughput issues, that would be ideal, but there's...
Kind of you're only as fast as your slowest link within the data center, but we are starting to scale and have to one gigawatt clusters and now 2 gigawatts and again Elon I think is pushing 10 now, TBD how much of that's going to be interconnected or not versus kind of the old H100s versus new VR300s etc. The direction seems clear. So as you start to scale up more and more GPUs and more racks, what is the thing that you've kind of been following there? Is it the interconnects across racks, things like LPO, coherent etc., or are there other things that you think are more interesting to kind of follow along as the scale up continues?
