Aloha listeners, Austin Lyons here with Chipstrat, and with me is Durga Malladi, a fellow which is pretty cool, and EVP and GM of tech planning for edge and data center. So we're here to talk high bandwidth compute. But first, before we get into that, which is a great topic, I wanted to know more about — like, tell me what your job title means. Tell me what you do at Qualcomm.
So I run all the IP and the technology planning for what we need to do across all the products, ranging from data center all the way down to edge solutions, including devices. So that means the IP, if you kind of think about it, is what should we be doing in our CPU portfolio in, let's say, three, four years downstream, in '29, in '30, in data center, versus what do we need to do in devices, in handsets, in new agentic devices. And then rinse and repeat the same exercise for what do we need done GPU, what about NPU, what about modem, what about sensing hub, and so on. So these are all the different IPs. So that's the planning exercise, which kind of runs the roadmap behind the scenes, but it is across all the business units.
Nice. That's pretty cool. I like that you see across all the different technologies, so that it's not like CPU running not together with GPU or not together with the different products. Yeah, that's pretty cool. Okay, so let's talk high bandwidth compute, HBC. So the way I see this is, obviously Qualcomm is trying to enter and is in a competitive space with inference solutions in the data center. You've got GPUs, there's XPUs, and Qualcomm has the Dragonfly lineup — AI 200, 250, 300. It feels to me like HBC is Qualcomm's big technical bet to differentiate and bring something different to market as compared to what everyone else is doing. So, you know, of course, I want your thoughts on that, but also then tell us what is this HBC and how is it different from what others are doing?
Absolutely. Yeah. So when we reentered the data center business last year, and I actually ran it for one year, like last year, one of the things that we really focused on is — you're absolutely right. There's a lot of players already in the market, and they have some pretty incredible solutions already. So it wasn't one of those moments where we said, okay, let's also jump into the fray. It's not like a me-too moment or anything of that sort, but it was like, what is it that is going to be differentiating us from the rest of the ecosystem there, and what is a really good, tough problem that we really want to solve? Because that actually gives us the benefit of, okay, that is different, let's take a look at Qualcomm. And we latched on to the fact that — and we had an R&D project that was running internally for a while on what is called near-memory computing, and the principles of that are very simple. But before that, what is the problem that we are trying to solve? And it turns out that if you kind of go back in time, maybe over the last six years or so, you will see that the compute capability — like if you just take any of the GPUs and say generation over generation, how much is it improving? It improves quite a lot, actually. It's like at least a factor of two, sometimes a factor of three, but the memory bandwidth wasn't actually improving at the same league. And so the natural next question to ask is, so what does that do? Well, you know, if you kind of go back to some of the basic principles of how do you do compute — well, you have all your data sitting somewhere, and you have to fetch that data to do the compute processing. But at some point in time, what happens is that your compute is so good that you've kind of tilted the scales quite a bit. So you might be just idling, you're waiting for the data to actually show up. So memory bandwidth becomes a bottleneck, and it's called the memory wall. And you see that a lot more with AI inference, especially in the decode portion of it.
And so what we did is our HBC architecture is based upon what is called near-memory compute. Instead of treating memory and compute as two distinct concepts, what if we actually do a lot of the compute right next to memory, like literally right next to it? It does two things. One, the latency comes down quite a bit, and second, we can reduce the power consumption quite a lot. So today in data centers, HBM — you know, the bandwidth is very high, but it's expensive and it's a power hog. So we actually wanted to make sure that we can solve it in a different way, and one thing led to another. And as we started talking about this with a lot of the hyperscalers and a lot of other partners, they loved the idea. And as we announced it, we actually announced a couple of names in terms of some of the partners that we have. So it's been a great journey on HBC. But as we briefly mentioned in our investor day in June and today at Snapdragon Summit — in fact, Cristiano left a teaser out there. But let me just tell you, you know, what is the point of this. HBC for us is a technology, but it's not just for data center. It actually applies to everything, all the way from data center to servers and accelerator cards and automotive and PCs and XR, and yes, in smartphones as well. But it actually works in a different way in some of these things, but the principle remains the same, which is you're doing a lot of computation right next to memory.
That's good, that's a great explanation, and I like how you mentioned that HBC is something that can be applied across from the data center all the way to the edge, and that's a good organizational design, as you were talking earlier about how your job is to see IP across all that. So that's pretty cool. Okay. So what I heard from you was the problem is compute is increasing quickly, memory bandwidth is not increasing fast enough, so all this extra compute can't get fed quickly enough. And maybe instead of trying to just increase the bandwidth of the memory we use today, you guys stopped and said, 'Well, wait a minute. Why have to shuttle all this data? Why shuttle even more, like wider buses or even faster speeds? Why not just bring it really close to the compute?'
That's correct. Exactly. And keep in mind that it's a principle that, you know, as you start bringing down the power consumption quite a lot — just going one step further on devices, power consumption matters quite a lot in devices, especially as you think of all the agentic AI-based devices that you see these days. And they're still evolving, by the way. There's a lot of new players who are coming into the market. In fact, quite a few of them have very little background in manufacturing devices or making devices. They come in from an AI-first mentality, and they're coming out with these devices. And the job of agentic AI devices is to be kind of constantly on, because they're working all the time. So that power consumption benefit that we talked about is going to be key, we believe, in unlocking the full potential of agentic AI devices.
Yeah, that makes sense. So it's not just the memory bandwidth problem, but also the added benefit of because you're not shuttling this data so far horizontally now, it's just going vertically, which we'll get into. There's also big power savings. And of course we know that power is the big constraint. Okay, so then, you know, to someone who's listening to this the first time, they've heard of high bandwidth compute — are you sticking memory on top of an accelerator, or tell us more about like the stacking and the separation? What is being sent kind of across the wire horizontally versus vertically?
Yeah. By the way, there's a really cool animation that we showed at investor day. I wish we had that, but I'll just use my hands as props. Okay. So the way it works is that the memory is like — you know, you have these individual — you know, the DRAM, it's called this stack DRAM. You're basically stacking DRAM one on top of the other, and there's a lot of wafer-on-wafer bonding techniques that are used on that. So this is a hardcore process technology, by the way. This is in fact, it has nothing to do with compute at this point in time. But then right underneath that stack DRAM is what is called the logic die. That's where you put in all the compute. And depending on the use case, you might actually say, I want some specific logic to actually reside over there. There might be yet another processor that is still sitting which is away from it. But you've brought in a lot of the heavy computational needs, like the ones which really need that memory bandwidth. You put all of that right next to memory in that base. And that HBC therefore is a combination of both: the stack DRAM on top plus the logic die that is underneath, or the compute die that is underneath. And now you've kind of done a bulk of the work just with that alone. There might still be some residual compute that you want to do, and that could occur in a separate processor over here. In some sense, HBC kind of is like a bit of a co-processor, because it's doing a lot of the compute, but it's not the only thing. You might still have one of the main processors that's running out there, but it removes that complexity quite a bit. So, in short, it is literally right underneath that stack DRAM that we're talking about. That's where all the compute's been.
Okay. Okay. That's helpful. So anyone who's familiar with HBM will say, well, this sounds a lot like HBM4E with a custom base die and stacked DRAM on top. So tell us the differences. First of all, is it a different class of DRAM that's being stacked? And then how should we think about the difference between the accelerator at the bottom versus a custom die?
Yeah, that's — in fact, this is exactly one of the things when we first came out with the concept and when we talked about it. You know, I went to Korea and we — and some of the other memory vendors in the US and in other parts of the world. And to be honest, I half expected people to say, 'Now wait a minute, now we're doing exactly the same thing, I don't know what you guys are talking about, you know, we have a pretty robust roadmap coming.' Actually, what I heard is the diametric opposite of that, which is, 'How can we work with you closer on this?' Because yes, this is indeed the right way of actually doing it. Because one thing that happens is there might still be ways of — you know, the basic principles and the concepts look similar, but that wafer-on-wafer bonding that we are talking of, and we still use a lot of LPDDR-based interfaces towards the main processor which is sitting out there, which means we still keep the power consumption extremely low. There are evolving things that are coming up in HBM as we move forwards, but at this point in time we see that if you're a memory vendor, even if you are working on HBM and you're looking at customizing HBM a little bit further going towards 4E and beyond, I think stack DRAM is here to stay. And you might have noticed a couple of announcements that have been made even by the memory vendors. They have different terminology for that; it is their equivalent of HBC, but actually they are partners of ours. So we expect that these two technologies will co-evolve for a period of time, and we'll see what happens after that.
Interesting. Interesting. Okay. So then — which, by the way, have you guys disclosed anything about the layers, number of layers that you're stacking?
We have not yet provided all the numbers. In fact, stay tuned on that front, because, you know, you haven't asked the question how's the silicon coming along and so on, because we've talked a lot about it in terms of PowerPoint presentations, and, you know, most people are actually taking our numbers for what they're worth. But we will be talking — we'll be showing some proof points over the next quarter or two as the silicon is back in the lab, and it's coming along very well. So stay tuned.
