A New Era of Context Memory with Val Bercovici from WEKA

100,000 tokens, one megabyte of data translates to 50 gigabytes of KV cache working memory. And that's because you're vectorizing. You're adding 10 to 20,000 dimensions of intelligence to this 100,000 tokens, this one megabyte of data. So if one megabyte equals 50 gigabytes, and that's just at the beginning of one session for one user.

Really good. Excited to be here. Very enthusiastic and new fan of the newsletter.

Absolutely. My favorite topics. We should go. Let's go.

Yes and yes and yes. You know, it feels like deja vu because I remember I left NetApp, actually joined a small cloud-native storage startup called SolidFire out of Boulder, Colorado, and then got reacquired back basically into NetApp surprisingly enough. And by then we were doing cloud storage and that was after VMware, kind of a golden era of enterprise storage and these NAS and SAN acronyms of the past. And I figured, you know, Kubernetes was really cool. I actually got involved with the Borg team at Kubernetes, Craig McLuckie and Sarah Novotny, Joe Beda and others, and we created the Cloud Native Compute Foundation under the Linux Foundation together. You know, I got on the board, the first governing board of the CNCF. So I thought around 2017 there was nothing left to do in storage and I left it. But it was a very disruptive time back then, you know, 2012 to 2015. It's hard to remember that now, but cloud was hyper controversial, right? Kind of as controversial as AI eating software today. Back then it was cloud software eating the world. So it's deja vu in that, you know, it's just big transitions all over again. And at Weka I joined again on the promise of actually not joining another storage company. And what I mean by that is that, you know, we were using the term data platform last year. And fundamentally that confuses some people because it's an overloaded term. There's so many things up and down the stack which we'll get into that could be called platforms. But fundamentally what Weka provides is high-speed storage and memory for AI infrastructure, which ultimately is the key bottleneck that we'll be diving into right now.

Exactly. Exactly.

Yeah. Let's get into that straight away then because for those who missed that announcement, it's at CES 2026. Jensen announced that they are introducing an inference context memory storage platform as a first-class citizen into the whole Rubin platform and this is something that's going to be here for the future and is here to stay because we need a lot of context going forward especially in the agentic AI era. But maybe we should just start with what context memory is and why we need it so badly today.

Yeah. What we've learned obviously in the just breathless evolution since three years ago, that ChatGPT moment three years ago at about two or three months now is November of 2023 I guess, no, 2022, right. What we've learned is that prompt engineering came and went as sort of an important focus. You know, kind of instructing the models was important at one time when the models really didn't understand how to process large attachments, if you will, large memory. Then we reached like the era where yeah, you know, RAG, retrieve augmented generation, where we figured out ways to attach just the right snippet of large attachments into the limited memory and be able to instruct the models to give us very clever answers. So chatting with your PDF, you know, was one of the most interesting things you could do, particularly when it was a large PDF and you only had to chat with a subset of it automatically. We're very much in the era of agents right now. So, of course, the viral agent of this week is Claudebot and Moldbot and the amazing things people are doing with it. What we are seeing right now is that agents, I like to say, are not really a singular thing. You either have an agent swarm on your hands with dozens and hundreds of sub-agent tasks and sessions or you're not really doing an agent at all. And at that point now you basically have hundreds of parallel instructions to a model with hundreds of parallel attachments so to speak, whether that's a codebase, whether that's documentation, whether it's a video stream, telemetry logs, etc. And there's still, even though the models have gotten smarter about understanding larger and larger context windows, there's never enough, right? So context memory now, a way to extend the memory windows, extend the interactivity of these models is the hottest field and there's lots of different ways to create memory for LLMs and for even diffusion models. And I always like to say if you think you know how to do this from your experience even two to three years ago, it's almost easier to just unlearn what you know about memory and storage and relearn from scratch because it really is very different this time.

The biggest change really is the fact that there's never enough memory. If you just do the math, and the math is not complicated. It's not calculus, but it is a multivariable formula. When you do the math of, for example, loading the weights of the models, which are very big, we're in the era of trillion parameter models right now. The hyper popular Opus 4.5 is a trillion parameter model. The latest DeepSeeks are trillion parameters. Kimi K2.5 just came out yesterday, the thinking version. That's a high-end trillion parameter model. Merely loading the weights of these models into memory is more than a big GPU server can handle. It's literally more than like, you know, an H100 can handle. And that's still a very, very popular GPU server. You need an H200. Very often you need multiple Hoppers or even more than one Blackwell server which is eight GPUs per server just to handle the weights of these models. And then you and I enter the picture and we open up a prompt session, right, and we start chatting with the model, we start feeding attachments, we start feeding context into the model. And very quickly the math gets quite brutal because the round numbers about 100,000 tokens which is roughly a megabyte of actual data and that could be the system prompts that can be at the model level, at the agent level, at our own preference level. Then you've got the prompt itself. Then you've got the data in question, the real context that we want to focus on, those attachments, so to speak. 100,000 tokens, one megabyte of data translates to 50 gigabytes of KV cache working memory. And that's because you're vectorizing. You're adding 10 to 20,000 dimensions of intelligence to this 100,000 tokens, this one megabyte of data. So if one megabyte equals 50 gigabytes, and that's just at the beginning of one session for one user, and we have multiple users kicking off agent swarms, so hundreds of subtasks per user, you see that quickly, you know, you're very much out of the working memory, the high bandwidth sort of primary memory tier that's on the GPU package itself. And last year saw a really nice evolution of the engineering of realizing we have these old concepts, right? Virtual memory concepts of being able to use other storage tiers or other tiers, if you will, of memory, and basically be able to, you know, flush data, old memory pages out, you know, bring fresh memory pages in. So we've applied these algorithms and concepts to, you know, context memory right now which means, and Nvidia has a very nice, I think you included it in your newsletter, a very nice hierarchy now of essentially the first level is a high bandwidth memory on the GPU package itself and that's memory, if you're a gamer you know this, right, it's memory on the GPU card itself so to speak, it's independent of the motherboard's memory. Every GPU on a server, there's eight GPUs, each one has its own dedicated, some people call it VRAM for video RAM if you're kind of come from in the gaming background. Data center people call it HBM, high bandwidth memory. So there's dedicated memory per GPU, and that's where most of the work really happens, but there's never enough. By the time you load the model weights and a few of our own prompts, you're out of that memory. And now the software layers. So we have very popular inference servers, vLLM, SGLang, Triton, you know, TRT-LLM from Nvidia that understand now with these KV cache managers how to tier the memory, how to basically, you know, tier down to other memory. And now we go to the motherboard, the shared memory across all the GPUs on the motherboard. So there's about one terabyte of shared memory and sliced up on each GPU on the HBM there's almost one to two terabytes of dedicated memory and aggregate split across eight GPUs. So let's say you've got about four terabytes of actual working memory again with today's latest models, with today's very context-rich user sessions and agents, not enough. So context memory now and context memory engineering is all about where do we go next? Unfortunately, there are these 10-year-old standards called the NVMe, non-volatile memory express was the original, you know, term or definition of NVMe. I like to call it non-volatile memory extension because I like to joke there's no S, there's no storage in NVMe. It is designed to treat NAND flash as a memory extension using more native memory semantics. Flash memory is memory. It's just slower memory than DRAM. And now there's ways to engineer storage solutions based on this NVMe protocol into this tiering hierarchy. And the art of it, you know, is it is obviously still code. So it is science but it seems a bit like alchemy at the moment is how do you make this lower performance, higher capacity, lower cost tier of memory, NVMe, behave just like regular memory to the model so that you and I, especially in a voice chat, don't notice awkward latency with a voice agent and things like that.

That's the ideal goal. And there is again some complicated formulas, but we can maybe walk through those to understand how that can actually work if you piece these puzzles together correctly.

That's exactly it. They've been around for decades. And when those solutions started, NFS and Lustre in particular, they're very, very popular in the AI world, were literally built around the era of hard disk drives. We have to remind ourselves, this is now history. It's funny because I started my career with these things, but it's history now. These are antiques where essentially you have rotating media, these rotating platters, many, many platters and you have this head, an actuator which goes across all these platters and you have to wait for a platter to spin before you can read data off of it. Then the actuator has to move the head to where that, you know, which sector, which track, like if you remember for people that like, you know, vinyl records again, which part of the record, you know, the data is on and then finally retrieve it. It really is like a record player and very much like an antique. And NFS and Lustre were excellent protocols for that because they made assumptions around the latencies of accessing metadata. You know, basically like finding out exactly where the index card is to find the ultimate data, you know, looking at the glossary of the book and so forth. That was the way that these protocols worked. Fast forward to flash drives. NAND flash drives first started out as these SCSI, small computer systems interface devices underneath these network protocols. Then you had Fibre Channel. You had serial access, sequential access, sequential access, SCSI, SAS, all these other protocols that came out of the spinning rust, spinning disk era. Only when the NVMe standard came around were we finally able to treat NAND flash as a true memory extension. And even then you really didn't have the latency sensitivity that you needed to make flash behave more like memory where NVMe devices, and this is going to be very important for your audience and your readers going forward, are also going to go through an upcoming evolution called high bandwidth flash. And so these flash devices speaking the NVMe protocol are nothing more than layers and layers of NAND flash with these like ARM-style microcontrollers in front of them. And each of these microcontrollers has a work queue and you have multiple queues per device. You have multiple devices per NVMe fabric. So the art to this is a modern protocol. We call it Neural Mesh. And it's a modern protocol that actually understands that there are thousands of queues across tens of thousands of devices in a fabric. And you have to understand natively the depth of each queue. So you understand at a global level for every read and write across the entire system what, you know, that latency is going to be across the entire system by understanding all the individual components. That's what we call Neural Mesh. That's the magic of Weka and it's because Weka has the advantage of timing. We were designed from the era where there was no spinning disk. We tier to that but we don't use it directly. You know, there isn't really a need for NFS or Lustre. It was just a brand new era with NVMe. And another key thing here is when Weka was born, the networks, the high-speed networks at supercomputers were faster than the motherboard. And even today, I have to keep reminding people that everyone just assumes the memory hierarchies are static. The motherboard is always the fastest way to move data around and everything on the network is the next compromise down. And it's inverted now. And Nvidia by acquiring Mellanox and building now, as Jensen will say, they don't ship chips, they ship systems. And the systems are these chipsets worked together to form these amazing AI factories, these supercomputers. They all depend on the fact that the network is actually faster than a motherboard. And that's how the GPUs are not kept waiting, are actually able to collaborate together and behave like a giant GPU.

Yeah. So, a very important question, a vital question. You know, the motherboard math is again non-intuitive because you have to factor in the software and the hardware into the final latency equation for all of these pieces. Latency particularly because we have two kinds of kernels. We have the kernel that boots the server, the Linux kernel that runs on the CPU. Then we have the kernel that actually runs the inference server. That's the GPU kernel. We hear about CUDA all the time, right? CUDA kernels in the Nvidia world. And so those are two different kernels that have to communicate on the same server and the inference server, you know, that runs our DeepSeek, that gives us our tokens from DeepSeek or Kimi K2 or GLM or all these wonderful new models, MiniMax, etc. OpenAI of course and Anthropic included. These kernels are communicating between each other. So when an LLM is processing your request and it runs out of memory, it still communicates before it runs out of memory at nanosecond speeds and with a high bandwidth memory and very, very low nanoseconds. We aren't even talking about this little thing called SRAM which we should come back and talk about because that's really what the whole Groq with the Q acquisition was about.

So with the G1 layer, that's nanoseconds and that's the GPU kernel. As we know now, we're basically tiering and including G2 in this hierarchy very commonly today in every modern LLM inference environment. And that crosses the motherboard now because you're going from the GPU kernel across something called a bounce buffer. Historically sometimes now a chip-to-chip interconnect between the GPU and the CPU and the memory that the CPU addresses and on paper the DRAM, the DIMMs, the DRAM DIMMs on the server motherboard that the CPU controls on paper it's also nanoseconds but not in reality. By the time that communication between the GPU kernel and the CPU kernel happens we're into the microseconds of latency so already kind of, you know, three orders of magnitude between the HBM and the effective of DRAM latency and that's not great but it's functional, right? The inference servers definitely do enough parallelism and enough asynchronous memory copies and things like that that it doesn't feel like a big step down. However, the big step down now comes from various legacy protocols entering this equation by tiering the G2 memory, the CPU DRAM to storage. They introduce enough overheads because of the way they go between user mode and kernel mode and the way they deal with metadata that you're not just going another three orders of magnitude from, you know, microseconds to milliseconds. You're going to many, many milliseconds, two, three, four, five milliseconds, thousands and thousands of microseconds as you make that transition and that's where you hit a wall, the kind of the memory wall because that's where it's very noticeable additional latency to the model and then to you the user and an agent really notices that because if the agent is spinning up hundreds of subtasks they can either complete in minutes or they can complete in hours with this additional latency. So the key design goal for Weka is to keep this real world effective transition from high bandwidth memory in nanoseconds to DRAM in microseconds to flat microsecond latencies at DRAM performance levels but with the benefit of flash capacity. So it is a best of both worlds scenario when you understand the engineering and the math. And instead of capping out at four terabytes of memory per server, you can extend that G2 layer now to hundreds of terabytes easily to petabytes. And in some extreme cases, we've done the math. Some of the biggest, you know, LLM apps like Cursor of the world and the various models they use are processing tens of trillions of tokens a day. And the KV cache numbers that you refer to, especially that great Manus blog from last summer. I call it the two billion dollar blog because of course they're part of Meta now for two billion dollars. That requires an exabyte of KV cache to service trillions of tokens a day effectively. So, we're going to be seeing this. This is why Jensen called it the biggest future storage market of the world because when you're dealing with an exabyte of memory, you have to think about some kind of storage semantics at least and even the industry now referring specifically to context memory is referring to it in the context of cache writes and cache reads. Sort of like a database or a file system, but remembering it's at these crazy memory speeds, hundreds and hundreds of gigabytes per second, ultimately terabytes per second, at these crazy capacities, at these microsecond rigid, you know, tight latencies.