Welcome to another Semi Doped podcast. I'm Vik from Vik's Newsletter, and with me is Val Bercovici, Chief AI Officer from Weka. Weka is an AI data and memory infrastructure company. Val has been on Semi Doped before, and a few months ago we spoke about context memory storage right after Nvidia announced their platform. So if you want to check out that whole conversation, it's right in the back catalog, and we'll try to link it up here as well. But yeah, today's conversation is more about more recent developments in the world of memory and storage and where things are going in AI inference. Val, thanks for being on the podcast.
So much fun to be back, Vik. And as we predicted, the only constant is change. So, lots of updates to talk about since I was last on.
Awesome. Isn't that so exciting? We have things that are constantly changing, and since we spoke, like what was it, a few months ago? Seems like everything has changed, and you were mentioning before the show, seems like 10 years have gone by in AI land.
Exactly. In fact, I don't know if I made this prediction last time. I expected a Mythos Fable moment at the end of the year, and it happened in like what, May or April, I forget exactly when. So things are definitely happening faster than we thought.
That's amazing. Yeah. It's really picking up now, and memory has reached an all-time high. It's become extremely expensive. I think a lot of people who are deploying AI hardware are finding ways now to make the best of what they've got. Increase utilization rates, pool stuff, offload to different tiers, whether it's other kinds of RAM, not just HBM, or go to NAND flash. So there are so many techniques that have sprouted up in a matter of mere months because we have to do something about the memory situation.
Exactly. I would say even one of the, I wouldn't say it's the simplest, but one of the most obvious and coarse-grained techniques in my mind has been model routing. If you're not always using the most memory-hungry model for every prompt, especially in an agent swarm with hundreds or thousands of turns, and you're intelligently routing requests to medium and small models, implicitly there you've got lower memory requirements. So that's kind of an easy button, if you will, for reducing memory in aggregate. But then we get obviously much more specific here with regards to each model's memory requirements, working memory, KV cache, and so forth.
Yeah, exactly. So that's what we were talking about in some sense last time, because the KV cache started getting so big when the agents arrived on the scene that you couldn't keep it on expensive HBM because it's a very limited resource and it has to be preserved and used judiciously. So there was a lot of talk about offloading to DRAM. And then as agents started working more and more and having greater context length, they started talking about offloading to context memory storage, which is what was this whole rack which Nvidia calls, what was that, STX, right?
Yeah, the new storage platform is STX. The biggest use case of STX is going to be context memory storage, CMX, context memory extensions in Nvidia terminology. And that is, yeah, within the STX hardware and software framework.
Awesome. So yeah, I know Weka has hardware, we spoke about this last time too, where you have got Weka's AMG, which provides really a lot of storage directly to the GPU via a very fast network connection, and the speed that you get out of it is really good, comparable to something like faster than NAND but slower than DRAM, right? So that's where we were.
Let's double click on that because it's actually different than that even. So the standard positioning is you can be faster than NAND with higher bandwidth connections, you know, faster than local storage effectively. But in a standard hierarchy, you could slot in as slower than DRAM. However, depending on your architecture and whether you can take advantage of the full line rate network performance or something like NVLink the way Weka does, you can be faster than DRAM. So if you actually take a look at the number of, you know, PCIe lanes that we have in these wonderful scale-up domains with NVLink on Nvidia servers in particular, there are far more PCIe lanes, like 128 I think in NVLink, than there are PCIe lanes from the CPU to DRAM, which is only 32 on the motherboard. So there's more. This is one of the cool architectural, I would guess, leverage points of Weka is we recognized at company founding inception about 13, 14 years ago for high performance computing, which is what AI factories are, there is more bandwidth on the network than the motherboard, and PCIe is a bottleneck, it's not an accelerant. And this is still something that I find a lot of systems architects and developers struggle with. They assume nothing is faster than what I can access on a motherboard, and they design entire database hierarchies, entire application latency profiles based on that invalid assumption today because the network is faster than a motherboard in a GPU network and a high performance network. And so yes, I think the memory tiering can have whatever you connect on the other side of that high bandwidth network, an NVLink style network, you know, east-west compute memory network, NIC network, it goes by many names. Whatever you connect on the other side of that can be faster than DRAM. And the way Weka connects, optionally we don't require that network, but we certainly leverage it maximally. But the way Weka connects, we're faster than DRAM on a raw bandwidth basis. Another confusing part is the way CPU and GPU memory transfers, KV transfers for example and others, operate on most of these servers, GPU servers, depending on a generation. Hopper has something called a bounce buffer between the GPU and the CPU. Blackwell introduced, especially with Grace Blackwell, more sophisticated chip-to-chip interconnects that are faster between GPU and CPU transfers. Vera Rubin yet again continues to advance that with higher bandwidth, you know, far more scalable memory transfers between GPUs and CPUs. But you can still be faster than DRAM on these servers because of NVLink. There's a ratio in terms of NVLink bandwidth relative to memory bandwidth relative to front-end network bandwidth. And as long as you can be line rate on NVLink, you can be faster than DRAM, which is counterintuitive, but the math is math. It proves out.
That's awesome. Okay, that's great because just to recap, what we are doing is that the networking on modern hardware is so fast that when you hook up even storage using that, you can get bandwidths that are faster than DRAM because motherboards aren't the fastest, and that is an assumption that a lot of people are making right now. But instead, if you view the NVLink as like a very high bandwidth interconnect, you can actually bring storage to operate with much higher capacity but at speeds that are better than DRAM speeds.
Yeah. And these networks are really critical right now. We've seen a rise in interest in networks, not just with co-packaged optics, CPO, but what Google does with their Taurus network, AMD obviously with Pensando and that line of networking that they're going to continue to improve with Helios. But just what Mellanox has done with NVLink has been wonderful. And it's not just NVLink, right? Mellanox offers CX7, CX8, CX9 network adapters, BlueField 3s and 4s. There's multiple varieties of BlueField 4s, and so we're not picky, at least at Weka, with regards to where the bandwidth comes from. As long as systems architects configure the bandwidth they want for the memory tiering they want, Weka can support it and continue to deliver these radical economic benefits in terms of more tokens per second, more concurrent users per second, which is really hitting the market sweet spot right now of agent swarms.
Awesome. Yeah. So since we last spoke, like Weka right now shouldn't be viewed so much as like a storage company anymore because in the AI era, because of what solutions Weka provides, I think it's far more than that. We were speaking a little bit about it before the show. Do you want to double click a little bit more into why Weka should be seen more as a memory company and a company that does AI data infrastructure instead?
Yeah, very much. You know, two of sort of our industry partners, if you will. One is Groq, with a Q, that Nvidia acquired last December for $20 billion in a non-exclusive license to their technology and most of their founders and engineers. And then of course a very popular and successful Cerebras IPO were clear indicators that the inference market, the inference infrastructure market is very distinct, you know, very different than the training market that Nvidia dominates with GPUs. And even Google themselves, for me, for some reason this is like the clearest example, Google has had seven generations of TPUs that were applied both to training and inference, right, in radically different markets. Back then when, you know, TPU 1, 2, 3, etc. through 7 was released, with TPU8 which they announced just a few months ago, they were very explicitly saying there is now a TPU8 for training, but there's a completely different TPU8 for inference. So that's just another example that, you know, inference is different, inference infrastructure is different. It's great when you can leverage, you know, the same infrastructure for training and inference. There's nothing wrong with that. But understand that, you know, whether it's really cool companies coming back out of stealth like Etched, you know, or Cerebras or all sorts of other LPU-based, you know, ASIC-based, SRAM-based companies, inference will continue to diversify in terms of infrastructure and diversify away from training. So Weka is very much following in that trend. Weka has a great legacy of high performance computing, HPC storage. But the new tagline for the company, data and memory infrastructure, reflects the fact that at its first principles raw basis, inference is not storage-centric. Inference is a little bit compute-centric, as we know, for prefill, and extremely memory-bound for decode and feed-forward and so forth. And that's really where the Weka Augmented Memory Grid product line fits in, is it's very much, you can use it as an inference-only product without any Weka storage whatsoever. It just happens to be backed by NAND at the bandwidth of memory, and as we just discussed, when that particular context memory network happens to be large, then it's actually more bandwidth than DRAM.
Yeah, that's awesome. So depending on the kind of hardware that is running now, you mentioned that Cerebras is one, and the whole inference landscape is like really seems like there's no one right way to do inference. You could run it all on SRAM, you could run it with like LPUs which is also SRAM, then you could run it with a combination of HBM and SRAM, or if you look at SambaNova, they use all three. So it's like really breaking up all over the place. And so how do all these compare like in terms of performance, or where does storage play a role into this whole thing?
So there's a memory hierarchy, and it's interesting now because when we last talked about this memory hierarchy, it's funny, we talk about it as if it's been around forever, but it's really only about a year old. But a fairly well-established four-tier hierarchy that's going to have to change now. And Nvidia's Dynamo team has done a really good job of documenting this. They even label it G1, the top tier in the memory hierarchy is for high bandwidth memory. G2 is for CPU DRAM, but it also goes by LPDDR and SoC memory, but it's, you know, for the uninitiated, it's all the same thing. And then, so that's G2. G3 is what's called local storage, and sometimes it's called like the rack SSD that's in the servers. And then the G4 tier is remote storage, often it's NFS or S3 type storage layers. And the one thing to really not maybe confuse about this memory hierarchy is there's no smooth graduation from HBM to DRAM to storage, local or remote storage. These are very, very rough sort of jagged transitions, almost a Grand Canyon sometimes between them, because we're talking about orders of magnitude higher latency between those memory tiers, orders of magnitude different bandwidth between those memory tiers, and you very quickly reach these cliffs where, you know, if you're okay providing one token or 10 tokens per second to a user and you're okay with latencies, time to first token of like 10 seconds and end-to-end latency of hours where they should be minutes, then you don't have a problem. But in the real world, no one tolerates, you know, five or 10 tokens per second output. Kind of the human eye depends, human attention span demands about 35 to 50 tokens per second of output minimum, and agents of course at machine speed will take thousands of tokens per second of output in a multi-turn agent swarm. So, you know, SLOs, service level objectives matter. And that's where we quickly find right now that even though there's a lot of vendors talking about memory tiering, including NAND and storage in the KV cache hierarchy, it's kind of irrelevant. Most of the benchmarks we see out there today that include storage in the KV cache tiering hierarchy are workloads that you could just run on a DGX Spark or a Mac Studio today. You don't even need a big server to run some of these smaller models with small context windows and, you know, single-turn chat sessions or two to three-turn agent sessions that are not representative of reality when you're running real-world workloads today. And SemiAnalysis is actually working on updates to InferenceX that they've nicknamed AgentX after my suggestion actually. So AgentX, when it comes out soon, I'm not going to steal thunder. But when that comes out soon, that's going to reflect something called an input sequence length, an ISL, which is the context window of much larger than 8K, which has been the limit in the past. Hundreds of K. You know, they themselves have published analysis of the fact that I think the median traces for coding agents now are 200, 300K tokens because of course we're in the era since we last spoke of million, you know, token context windows instead of 100K or 200K token context window limits. So when you reflect the reality of large models, trillion-parameter models or models in that class, you know, MiniMax is in that class even though it's not quite a trillion parameters, but Kimi is, and GLM 5.2, and the new hotness is, and so forth. When you reflect large models with large multi-turn sessions with high concurrency, the way, you know, Claude Code, OpenCode, Codex, Pi, you know, Hermes, OpenClaw, actual agents work, then you end up with a radically different workload profile. And on the one hand, the models have gotten so much better in terms of KV cache consumption individually, right? I use a number I think I remember of for every 100K tokens without optimized compression like TurboQuant, without optimized compression like sliding window attention and others from DeepSeek V4, you are up to 50 gigabytes of KV cache usage for that 100K tokens. That's been reduced on a unit basis right now with Turbo, with optimizations from DeepSeek and others, to about five, five gig instead of 50 gig. So that 90% reduction is great. But what's happened on the other side? Context windows 10x'd, and then agent swarms 10x'd, you know, or 100x'd the number of concurrent sessions. And on and on and on, we've gone multimodal. And so the net effect we always seem to see is that for pretty much every 100x reduction in unit KV cache size, there's a 10,000x increase in consumption of KV cache.
Yeah. So we're always looking at about that 100x more token volume, and it's reflected in the pricing pushback we're seeing by enterprises today. Maybe they're not paying 100x more than they expected, but they're paying a lot more than they expected because of that token volume.
Yeah. Yeah. So this is great, right? Because clearly KV cache optimizations are on the way. Like TurboQuant was one which scared the daylights out of the market. Everybody thought like memory is dead, HBM is dead, everything's dead. But it turns out that the quantization is a good thing because we only need, like you said, one-tenth or 90% lower KV cache storage for 100K tokens. So 100K tokens was taking 50 GB. Now it takes 5 GB. But this is great because which means with our existing models, we can run far more agents and do a lot more work. And so the demand really never went away. In fact, it got more, right? It increased. Yeah. In fact, you can almost make a meme out of this that 50 gig that went down to 5 gig went right back up to 50 gig because the context window size went from 100K to a million. So there's give and take here, and the net result is just always more memory demand.
So just because KV cache is being compressed like this, do you see that it's going to continue to be so? Do you think that we'll go to like maybe one gig per 100K tokens in the future? And what does that mean for KV cache? Because if you say like we dropped usage by quantization techniques and then we brought it back up because of like agents, I guess we don't have a net increase. Is that something that the memory and the storage industry should worry about? Because I don't know if the increase in the usage is exceeding the benefit coming in from the quantization.
That cliche, this time it's different, applies here too, right? Every time there's this massive new innovation in KV cache compression, it's not different. It just Jevons paradox keeps kicking in over and over again, and we're seeing that. Oh, you know, that higher increase in demand every time there's a reduction in unit, you know, consumption. And so for me, let's look at context windows. We're not stopping at 1 million. It won't be long before we're at two and five and 10 million, maybe by the end of this year. Everybody wants more context. And we're going again longer horizon in our agents. We're giving agents far more ambitious goals, being encouraged to by the agent and model providers. The goals now span hours and days. Soon I think some agents will run weeks quite regularly. In fact, for cybersecurity, which we can get into, they run forever, right? You've got to run the security operation center persistent blue agent swarms forever now. So we're seeing longer multi-turn horizons. Of course, we're seeing more parallelism, more concurrent subtasks. Cloud workflows was one sort of feature introduction that became very popular to just solve a problem in parallel 10 times faster. And then again, we're going multimodal. So it's not just language, but we're inserting more video and audio frames and image frames into these agents. And it all just results in a continuous explosion of overall token demand and overall KV cache consumption.
Yeah, that's awesome. Okay. So now that we've established that this is only going to get more from here, so NAND and DRAM or whatever form you would want to look at it as, pool, not pulled, we're going to need more of this because we're only going to run more of these things. So we're going to need more. So that doesn't spell any decline or leveling off in the near term because context windows are going to get longer, agents are going to run longer, and more agents are going to run per task because you can have parallel ones run. But I think one interesting thing that you mentioned earlier was like DeepSeek's optimizations because when DeepSeek V4 came out, there was a lot of emphasis on SSD-first approach to inferencing, which I think really helped in terms of their token pricing, especially when you end up hitting cache. And they showed that for some workloads you could hit like 95% cache hits. And what happens then is that as long as you keep hitting KV cache, you already have the tokens stored in a high bandwidth network connection to an SSD, which means now you can basically get like free, it's free now because their token pricing is so low for cache hits. It's insane. Like how is this playing out into like other models, or are other models also hitting cache hits at this level, or what do you think of that?
Yeah, so on the one hand I'm glad you noticed that because we really want to emphasize this key point now. The majority of token volume is agent swarms by far, 80, 90%, it's no longer chat sessions. So chat sessions are almost, you know, irrelevant to the conversation from an infrastructure perspective. And agents, it's interesting, even the Sonnet 5 announcement just the other day, right, that everyone looked, even Anthropic, interestingly enough, said here's the new pricing for input tokens and here's a new pricing for output tokens, and it's a promotional price. And the reason why I don't even quote the numbers, it's really irrelevant because of all the tokens used by agents, most of them are cache read tokens. And some of them, with Anthropic's particular, you know, unique pricing models, you have to pay for cache writes as well to benefit from cache reads. But we should only be talking about cache read pricing. The other pricing is a rounding error. And DeepSeek's pricing reflects that, right? They're dramatically, you know, inserting two zeros, right, basically after the decimal point for price per million cache tokens. Really is a shock to the industry, not so much the memory industry, it's a shock to the other models and the other inference providers. But the very important asterisk there is that pricing is only available out of China, right? You have to use DeepSeek hosted. You know, the rumor is I think a Mongolian data center with very, very low energy costs, but also some secret sauce that DeepSeek has with regards to how they've integrated version 4 Flash and Pro with the, you know, 3FS file system, parts of which is open source. They famously open sourced that last year and wrote some wonderful papers and blogs about it. Curiously, unless I've missed something, they haven't updated those blogs with what definitely are some new innovations and improvements that factor into that dramatically low pricing. The net of it though, I think VentureBeat did some math, which I was quoted on earlier on, is it's about 87 times lower cache read pricing from China than the same model, V4 Pro or V4 Flash, hosted in Singapore or hosted in the US or Europe. And so the real conversation is, yes, DeepSeek has completely set the bar for global pricing for people that are able to inference out of China, but for people that can't inference out of China, it's still by far the best pricing. But there's a massive margin opportunity for open weights inference providers to differentiate on cache read pricing because you may not be able to match DeepSeek's subsidized pricing, but you can still leverage their innovations, their sliding window attention, hybrid compressor attention, HCA and so forth. You can still leverage those innovations and offer very important reductions in token cache read pricing and really drive effective, you know, aggregate agent pricing down because that is the one pricing metric that dominates, you know, the cost of agents.
Yeah. So I have like a couple of questions on this one. The first one is why is it, it's news to me that that pricing comes only out of China. So if I run a DeepSeek model out of, I don't know, OpenRouter, I don't have that pricing.
OpenRouter is wonderful because it shows you the difference, right? In fact, for some reason, I don't even know exactly what this signifies, but there's a little slider button you have to slide to say show ignored. And I'm not exactly sure why they label the button that way, but when you click on that, it does expose the DeepSeek pricing out of China. And then you can compare, in fact, your agent, of course, your Hermes, your OpenClaw agent can compare side by side in real time the pricing for the exact same model, the exact same input pricing, the exact same output pricing, the exact same cache read pricing. And I should double check because I tend to stand that OpenRouter site quite a bit daily. I should check whether there's more and more cache read pricing beginning to appear because I know for some models it has, and it escapes me whether it's appeared for DeepSeek providers as well. But you're going to see some providers now differentiate not just on cache read pricing, which is a major point of differentiation, but also on cache write pricing.
Okay. So in terms of like cache read, in what the real workloads that we're running today, it could be like agent workloads, right? Because like you were saying, any other kind of workloads like chatting and typing in, you know, on the chat window is a negligible portion of the inference market today. So let's just talk agent workloads. How does somebody make sure that they have a cache hit rate of, I don't know, 95% because that will drive the pricing down enormously and create meaningful differentiation like you were saying. And where is the industry right now in terms of cache hit rates?
Yeah, you know what, we could spend a whole pod just on this question because it's very opaque and a little bit confusing. So let me go through this because this comes up a lot in other pods as well. When you look at your agent dashboard, your Claude Code or just your Claude dashboard if you have both Code and Cowork and other Claude products or OpenClaw, pick your dashboard, right? It'll have a cache hit rate that is a logical cache hit rate that reflects the cachability of the tokens in your agent swarms. And that is often very, very high. Like every agent pretty much has a cachability of 95%, right? Because you are reusing a lot of context for agents, especially agent swarms and so forth. However, the providers, you know, the actual cache hit rate from the provider is not one-to-one the cachability of your tokens, right? The effective cache hit rate is very much a function of the memory tiers you have. And everyone or by and large, you know, most people in the first quarter of this year and some in the second quarter have had very finite fixed memory tiers. There's only so much HBM that comes packaged on your GPUs. There's only so much of the URM that's on the GPU servers. And as I mentioned before, you can try and add storage tiers and KV offloading to storage, but they ruin your SLOs. So it's really not that common in production for the popular models and popular token consumption. And so the way to really understand where people actually have true effective cache hit rates that are as high, maybe not as high as the logical cache rates, but close, is in the pricing. So OpenRouter again is a great proxy for that because it's the only way to introduce transparency to the real-world cache hit rates. And even OpenRouter themselves, they publish actual by provider, by model, by provider, they publish some cache hit rates. And I like them because if you look every few minutes, they do change, right, based on the actual token traffic of the moment. But even with some of the numbers I see there, I don't think there's a real effective infrastructure, hardware, memory cache rates. There's some blend, but nevertheless, they're much closer than what your own agent dashboard reflects. And yes, for me, I like to joke now we're seeing a lot of benchmaxing in a lot of the models right now. You know, for example, like SWE-bench everyone sort of trains for. So it's no longer that relevant to benchmark, but Deep is still a good benchmark. We see the same thing obviously when vendors sort of benchmark their KV offloading solutions. There's not a lot of truth left in benchmarking. So my personal belief is that profit and loss, right, pricing, transparent pricing is the ultimate benchmark. And you're going to see Weka be more explicit in that space to do that, right? We want to continue to prove out our own advantages and we think that real-world metrics reflected by pricing is probably the best way to actually benchmark a solution now as opposed to something in a lab.
You also mentioned that DeepSeek's sliding window attention is something really to look into. What's unique about that sliding window attention in DeepSeek?
It's actually, if you've ever paid attention to where compression of files in general has evolved from simple like PKZIP compression or simple block-based deduplication towards the similarity hashing algorithms and a combination of global similarity and global hashing and local hashing. The same thing is happening now with KV cache. The same concepts are being introduced at the token, at the attention level. And sliding window is just that. It's a way to take a look at just the more recent tokens and compress them as much as possible and just assume the tokens that your particular attention head hasn't paid much attention to recently aren't as relevant, aren't as compressible. And it's just cumulative, right? I think if you count it, there's formally about five different attention mechanisms in DeepSeek V4 now, all simultaneously applied. And each one specializes in short-term compression, long-term compression, context-relevant compression and so forth. And the net result is that impressive 90% real-world reduction in KV cache usage per request. But again, the requests now are just coming faster and bigger. So it's kind of necessary to support. In fact, I think the reason why they continue to innovate so aggressively on KV cache consumption is they want to support million-token context windows and soon two and five and 10. And the only way to do that effectively is to keep being more and more efficient, intelligent, current, if you will, on how you consume KV cache.
Okay. Yeah. Yeah. So, DeepSeek, what it does — the sliding window attention idea — is that you just keep the window of attention to what is most relevant right now and keep discarding what stuff isn't relevant anymore, and that saves you...
KV cache almost assumes what's been attended to before that's not being attended to now has already been compressed to some extent. So now let's really be aggressive on what we're attending to right now and see where the opportunities to compress token memory, you know, is token attention in KV caches.
Okay. In terms of just like NAND flash storage even, I wrote an article recently about what it really takes for an SSD to be like AI-ready. Like SSDs in the past were meant for a different use case really than what they're being used for now. And one theory I have, which I want to run by you, is that NAND itself is now breaking up into several tiers. Like we always in the past wanted to get more capacity out of NAND. So the industry was like trying to go from single-level cells, which has lower capacity but, you know, more endurance and faster write speeds, to going to QLC, or, you know, quad-level cells on the other hand, where you can store, you know, basically four bits or 16 different states in just like one cell. So it really gives you a lot more capacity, but endurance is a concern, and I'd say it's harder to write to because you have to make sure you can differentiate between 16 states. How do you see these NAND tiers working out? Are single-level cells making a comeback? How is that working?
Yeah, it is making a comeback. And so just like we talked about earlier, you know, training infrastructure used capacity NAND because you wrote a lot of training data infrequently and you read it very, very frequently. So that was ideal for QLC-type approaches. Whereas inference is just fundamentally different, right? It's really fundamentally memory-centric. So you're really trying to make NAND appear more like memory and a lot less like storage. And memory doesn't care whether you write or read at the same rate because the assumption is there's no endurance issue, right? There's a power issue, but there's no endurance issue, a persistence issue, but no endurance issue with memory, with DRAM in particular, or HBM. And that's not true. So this is where NAND flash struggles to act more like true memory from a performance and a consumption perspective as opposed to just a capacity perspective. Again, this is something that Weka anticipated a long time ago. And so you're seeing that largely because of inference alone. Intel had this really interesting technology with Micron called Optane or 3D XPoint.
It's a real shame. Oh my god. Yeah. You know, today inference is the killer app for Optane, but it's gone now, right, based on different materials, phase change memory and so forth. And what is trying to replace it is SLC flash, right?
If you can't optimize your writes, your KV cache writes well enough, then you have to buffer that endurance issue, that drive-wear issue, with a more endurance form of NAND flash, which is a single-level cell, SLC tier, and sometimes that can be complemented by QLC. So you buffer a lot of writes in SLC and then you destage them down to QLC later. Again, that adds complexity, that adds latency. That's no free lunch there. Another thing you can do, and this is something that Weka does, is kind of anticipate that you're going to write to NAND flash a lot, you're going to read to it a lot. Basically use NAND flash as opposed to just keep it for capacity.
Yeah. And in that case, yes. What Weka has done is we've always advertised writes across a whole fabric of NVMe drives. My joke is there's no S, there's no storage in NVMe. It's non-volatile memory express or memory extensions. And when you treat it like a true memory protocol and you amortize the writes across a whole fabric of NVMe devices, you don't need SLC. We don't have SLC, for example, never have in Weka. We literally can use TLC for a very high write endurance workload like KV cache. And of course, in the future, as we're able to influence the rest of the inference stack, so the scheduling and the routing, not just the KV transfer storage layer, we will be adding support for QLC drives as well. But again, you got to be careful about that because you don't want to break those drives by doing unintelligent things on scheduling writes or unintelligent things on just routing tokens to somewhere that needs a quick write but is really kind of not worth it.
So you can actually optimize the controller layer and make sure that you don't clobber the drive, whether it's a TLC or QLC drive, and intelligently write to the whole TLC/QLC array and still keep endurance.
Exactly. This is something, it's a category of storage technology called shared everything, as opposed to the Hadoop-style shared nothing, which you still see in parallel file systems like GPFS or Lustre, etc. And with a shared everything approach, the client, if you will, has global awareness using a RDMA protocol of the queue status, the work queue status. There's multiple queues in every NVMe device, every SSD controller basically. So when every client has global visibility into the queue depth of every NVMe device in a fabric, tens of thousands, hundreds of thousands of queues on tens of thousands of drives in a fabric, then you can treat NVMe like a cache line for DRAM, and you can be very intelligent about where you shard the writes, right, where you load balance the writes very, very intelligently. And at that point, you don't need expensive SLC tiers to buffer writes. You're not being, you know, suboptimal, you're not being unintelligent or treating devices as blunt storage devices. You're really digging into the devices and you're making sure that you're optimizing the actual underlying NAND flash well and cooperating with the controllers versus offloading work to the controllers.
So you don't really see SLC as coming in on its own tier. Maybe it's just going to be used as a buffer.
Well, let me disclose here that I'm talking about how Weka has decided to optimize flash tiers. I think the industry needs SLC, right? The industry doesn't have our patents, doesn't have, you know, our implementation by and large, except for, you know, the server partners that we partner with, but generally that technology is not available outside of Weka. So you do need SLC. There will be a rise, you know, NVIDIA is forecasting this themselves even for CMX, there will be a rise in the need for SLC to buffer the writes when you can't amortize them over TLC. But the upside of that is that if you buffer correctly to SLC, you can still use QLC. At Weka, we're skeptical, right? Because again, these are not capacity workloads where you have the luxury of time to destage SLC to QLC. These are very bursty, very intensive workloads that are attempting to emulate memory. And so we think that ultimately one tier, whether it's SLC or TLC, is the only safe and fast way to offload KV offload to storage. But, you know, the industry, there's a lot of smart engineers across the industry and many vendors, you know, the market will prove what works over time. We just know what works for us right now.
