I guess there's all this talk about like agentic coding with very fast Codex, right? Like as Sam Altman put it, but do they have a way to handle long context storage with this whole Cerebras system?
Welcome to another Semi Doped podcast. I'm Austin Lyons of Chipstrat, and with me is Vik Sekar from Vik's newsletter. Hey, what's up, Vik?
Hey Austin, how's it going? It's going. I shouldn't call you Austin. I guess you're now Tom. Did you see on Twitter how somebody said you look like Tom Holland? They're like, wait, why is Tom Holland speaking semiconductor stuff? Like, what's going on? Like, is everybody an AI head now? And then I'm like, and then I looked up Tom Holland. I'm like, holy, you look like him. That's awesome.
Yes. I've never gotten that. I too had to look him up and I was like, oh yeah, it's the Spider-Man guy. Cool. I'll take it. I haven't gotten it, but famous movie star. I'm sure he's like 10 years younger than me, so let's go.
All right. Your friendly neighborhood Spider-Man talking to you semi stuff today. You know, I liked it. I guess I need a teacher. Let's do it.
All right. So, Vik, you wrote a post about context, memory, storage, and I learned a ton. It was really good, very in-depth, and I actually liked you had a lot of really cool illustrations. But let's start there with our audience. So why don't you take it from the top? Like, what should we know about context memory storage?
Yeah. So the whole thing got started because Nvidia at CES mentioned the influence of context memory storage. We mentioned it in our podcast about Nvidia CES also. And the whole thing is all about having just a bunch of NVMe flash storage in a rack and then hooking it up with like really fast networking to your main GPUs and just have it store stuff. And it seems like this idea is not new. You know, people have had these flash disk arrays in a network storage for a very long time. But the very fact that Jensen mentioned it in this big forum immediately has everybody looking up NAND and then the prices went up for all these NAND companies. It's the usual thing.
Yeah. The Midas touch, the Jensen touch. Yeah, he does this and you know, everybody talks about this stuff. So I think it's important to talk about it a little bit more than we did last time because it unlocks a whole new era of things we can do in AI inferencing that just wasn't possible before right now. So for anybody who's actually used like these tools, and I'm guessing it's a lot of people at this point, you know, I in Claude especially, the more I keep talking to Claude, especially in the Opus mode which consumes a lot more tokens, pretty soon it says like that's it, this chat has now run out of like, it's full or you're done with this chat, continue your conversation in a new chat. You know, if you ever see that, or more recently it has started saying, hold on, I'm just compressing our chat so we can continue talking. So what is actually happening behind that is the context memory is filling up. Like there are only so many tokens that can be stored and after some time something has to be done with those tokens to continue the conversation, right? Or they have to be discarded, in which case, you know, AI forgets the entire conversation you've been having with it all this time, which is extremely frustrating because now you're like, so what do I have to tell you all that stuff I just told you all this stuff now you want me to say all that stuff again? It's terrible, right? So then you copy your original prompt and begin. It's very frustrating. So the compacting idea was good except that in my own usage of Claude, for example, it kind of forgets stuff. It gets amnesia and it's like tells me stuff that I've already told it before and it's like, did you forget that? Or it tells me something I never said to it before. Anyway, it's just messy. And all of this is because context is running out. Okay. So what this context memory storage solves is exactly that. You can actually have really long conversations if you can store more context, right? And this is getting more and more important especially when we are in the age of agentic AI and agentic workloads, especially like AI-based coding. You know, there was like a piece of news weeks ago where the Cursor CEO said that they built an entire web browser just with like ChatGPT 5.2. They just like prompted it and then let it run for like a whole week.
It came up with a browser. It's just kind of fantastic. This is wild.
Yeah. It used to take like a couple of years before and a whole team of people building browsers, right? You didn't see browsers come out every day. Like when Chrome came out, that was like a big deal. And then before that was, I don't even, like you and I probably remember Netscape Navigator. Like you know, that was like way before even Internet Explorer. So you don't have browsers coming out every day. But if AI can make a whole browser with all the support possible in like a week, that's fantastic. Now what does it take to run a whole AI job for a week to do something like this? You know, this is the kind of stuff that context memory unlocks because it takes lots and lots of tokens. It probably has like a big code base that it's generating and at the end of the day it can't store all this in HBM, right? So it needs to store this context somewhere.
Yes, we should talk about that, unpack that a little bit, right? So tokens are our currency of AI in a sense because all the words we put into the chat box on LLMs are converted to numbers and that process, you know, the resulting number is called a token. It's kind of broken up in some way. Now the question is, what does it mean that we're running out of, we can't store tokens? What does that even mean? Like are we storing everything? Like there is some nuance to what it is that we exactly have to store, right? So one of the big things that is a selling point for a lot of LLMs is the context window. So when you see like Gemini has a 1 million token context window, what that means is that it can take 1 million tokens together and form relationships between all those tokens. So it knows the first token and the millionth token and the relationship between them. So that is what we want to do. Now in the process of generating output tokens from the input tokens we give it through a prompt, there's like a whole bunch of calculations that happen here. And this is essentially what the LLM transformer is doing. And within that functionality is something called attention. Like this is the fundamental revolutionary concept that Google introduced in their 2017 paper saying attention is all you need. Attention is a mathematical formula. And the whole idea is that you calculate a bunch of matrices, right? So it involves a bunch of matrices called the key and the value and the query matrix. And then of course there are the weight matrices. Now for every token that comes in there, there is a key value cache, sorry, there is a key and value pair that is calculated based on the model weights. So the model weights are what you get from training and using an input token you can make a key value pair and that is stored in memory. Then the next token comes by, you make another key value pair and you add it to the first one. And you keep, for every input token, you keep adding key value pairs to this original matrix. And so as the context length grows, your key value matrix grows linearly with it. It's not a quadratic or cubic or anything. It's a linear increase of the KV matrix and that is what is called the KV cache. This KV cache has to be stored somewhere. Now imagine if you have a 100 million token input window or a billion tokens. I don't know how many agents would need. If you have a billion token window, you need to have a massive KV cache stored somewhere and it can't be in HBM. So this is why, this is where we land at what you said. We don't just have enough HBM for this, right?
Makes sense. Yes. So, okay, we calculate all these key value pairs as part of the attention, paying attention to all the different words and the relationships between them. And the longer the context we have, like the more useful the AI is because we're giving it all this reference information, PDFs, all you know, whatever, what have you, code bases, entire code bases. But at some point there's too much and this can't all fit on the HBM on the GPU. So where does it go?
So HBM on the Rubin is 288GB. Okay, half of that is taken up by the model weights, let's say, it depends on the model. And so let's say you're left with like, I don't know, 150GB of HBM. That's not enough to store 100 billion tokens. It doesn't matter how small you make it. People have thought about ways like with multi-headed latent attention and grouped query attention. Smart people have come up with very smart ways to reduce the size of the key value cache you can store, but you just can't store everything in HBM. So what happens next is that whatever spills over from HBM is stored in the CPU DRAM. So we mentioned in the CES episode that the Vera Rubin platform supports 1 to 1.5TB depending on what kind of RAM sticks you throw in there of DRAM in the CPU. That is also intentionally designed for longer context storage because in the Blackwell series you only had 480GB of DRAM associated with the Grace CPU. In the Vera CPU you have over 1TB. So you can store a lot more context on this. So whatever spills over from HBM goes to DRAM. And DRAM is good. It's not as good as HBM. It's still pretty fast. So you can get this back pretty quickly when you need it. Now, if it doesn't fit on DRAM of the, you know, on the CPU, it goes out to the next level of storage, which is maybe the local SSD. The local SSD is all good, but you know, when you're serving 100 million people, all of whom are running agentic AI workloads, that is not going to fit on a local SSD. So now you have to go at scale. You need petabytes of storage in a rack server somewhere and that is where all your KV cache is stored. Gotcha. So we're talking about memory hierarchies here.
Yes, totally. And just like any other memory hierarchy, right, you have a latency and a throughput associated with each one. HBM is the fastest. So if you can get your KV cache quickly when you need it, that's very good because you can compute your output token quickly and then you have a very low time to first token and a high throughput of tokens because you can keep getting KV cache quickly. Now the farther down this memory hierarchy you go all the way to like network storage, this gets pretty difficult to do quickly and the time to first token and all those metrics, throughput drops. Now there are ways that things have, people like Weka with their augmented memory grid or even the ICMS system, the context memory storage system that Nvidia is planning, they're all finding ways to make this access of this data from this large storage pool really quick and that is where it all lies. Like so you have to be able to store a lot of information but also get it quickly.
Yes. So this reminds me if anyone has studied CPUs and like L1, L2, L3 cache, there's like that nice pyramid where at the top it's SRAM and it's like the smallest and it's the fastest but it's the most expensive. Then you go down a layer and it's bigger and it's cheaper but it takes longer to access. You go down further all the way, you know, SRAM through DRAM all the way down to SSDs and even like cold storage. I saw that Nvidia put out a diagram very similar. Instead of L1, L2, L3, they called it like G1, G2, G3, you know, probably for GPUs. Same concept. There's always a trade-off. More storage and cheaper, but it's slower to access. Now, I thought it was interesting that they put G1 was HBM when technically maybe there's a G0, which is SRAM. And I just say that because of course they're only, you know, in Nvidia GPUs the SRAM is for like local computations like registers, but of course there's other AI accelerators who are using SRAM to actually store the weights and the activations and presumably the KV cache as well. But anyway, I just thought it was interesting. But carrying on. So you're talking about essentially Weka, Nvidia, others are trying to ask how can we take like that lower tier of storage on the pyramid and have the benefits of cost and capacity but speed it up. So kind of almost like move it up the pyramid a little bit. Or you have a nice diagram where you showed this as well. So talk a little bit more about how to like get the benefits of the storage but trying to speed it up so it acts as if it was like a level higher.
So there are just these data processing units that really reduce the latency and increase the throughput from all these storage devices directly to the GPU. So you can take a lot of information and bypass all the middlemen, so to speak, like PCIe controllers and all these network switches and then directly take it straight to the GPU using something called GPU Direct Storage, so GDS. So you can just like move large amounts of data and this will help get it to the destination quicker. So they're making some modifications to the whole process to make sure that GPUs get the information they need.
Nice. So tell me about the benefit of this with regards to token economics.
Yeah, so I was looking this up recently. If you look at like Claude and go into their API docs and look at how they charge for tokens, they have a token system based on where the token is in the inference process and so to speak. So for so many input tokens, you have a certain price. I think it's like, I don't know, like let's say $10 for a million tokens or something like that. And then for output tokens it's actually much more. It's more like $25 per million tokens. And then for cache hits you have something like $5 per million tokens. So what that means is you're looking for that KV cache, but if you have a cache hit or a cache miss, it charges you differently. So if you want to write it to cache, you have to pay a certain amount for a million tokens. So for example, if you want to write a million tokens to cache and hold it there for like five minutes, it's probably going to cost you like $5. Okay? These numbers are not exact, I don't, but it's on the API you can check it out and they change too all the time. So if you want to hold the same tokens for like, I don't know, let's say an hour instead of 5 minutes, it costs more. So you have to kind of plan what kind of tokens you want to keep in cache and for how long. You know, this arbitrage is very important for you to keep the cost of inferencing down. Now I mentioned about the cache hit versus cache miss. So you've made some decisions on what you want to store in the cache and for how long, right? If you were right about that and you hit the cache and it's a hit, then your cost of inferencing is one-tenth what it would be if it's a cache miss.
It makes a ton of sense. It feels super complicated to expose this in your pricing to say like, think about what you're storing in a cache and if you get a cache hit it's way cheaper but if you get a miss it's way more expensive. On the other hand, and I'll let you carry on, but I just want to point out it goes to show that clearly this adds up.
