Qualcomm's main point is that they're diversifying their businesses and the majority of their business in the long run will be non-communications. And I thought this was interesting because the name of the company and the legacy of the company is communications, quality communications, Qualcomm. But the crazy thing is the data center business could become so big and it could inflect so much that what Qualcomm makes from their new businesses, data center and automotive, could ultimately dwarf what they make and have made in the aggregate of the company's history as a communications player. So that's something big to wrap your head around, which is just Qualcomm being a communications player could just be the start of their story.
All right. Hello. Welcome, listeners. Welcome to another Semi Doped podcast. I'm Austin Lyons with Chipstrat, and with me is Vik from Vik's Newsletter. Hey Vik, what's up, man? It's been a while since we chatted.
Yeah, it's been a while. I've been traveling, you've been traveling. Finally, we're back at the home studio. So, it's time to do something now.
Totally. Totally. So, you were in the UK and it was hot.
It was crazy. I thought I ran away from India to get away from the heat and feel some cool air in the UK because everybody says, 'Oh, it rains all the time.' So, I was hoping to get rained upon a little bit. But anybody listening from the UK would be like, 'No, no, you never wish for that stuff. Don't do it.' You know, we like a little bit of sun once in a while. I know it's a lot of sun right now in Europe, but it's okay, baby, for a week or two.
Yeah, totally. Yeah, I saw it was like 110 degrees Fahrenheit or something in France. Just like when you said it was hot, I was like, oh yeah, it's like UK hot. But it's like, no, dude, it was hot hot. It was hot hot. And you know, the other thing is in the UK they're not just equipped to deal with this stuff because in India, like either there's AC in some places, if not, if you go by buses for example, there are like windows, sometimes there are no windows, okay? So you always feel the breeze. It's not cool breeze, but at least there's movement of air.
Sure.
In the buses in the UK, it's just like sealed.
So, there's no air movement. The windows don't open out or slide out or anything like that. And you're just trapped and you just have a bus full of people just like sweating. It's a sweat lodge.
Just suffocating. Yes. Brutal.
Well, I was in New York City and Qualcomm flew me out for their investor day event and the weather was really nice in New York City. It was very beautiful. I woke up early. Obviously, the sun's out for a long time right now. I ran in Central Park. I ran one day to the west side along the Hudson River. I saw crazy huge yachts. I have never seen such big boats in my life. I don't know if it's like someone who lives there's yacht or if it had to do with the World Cup because there's a bunch of people in town for the World Cup, but it was like I was like, 'Dude, you need a whole staff to run this thing. Like, this is crazy.' And here it is. And I'm just right by... I was going to take a selfie in front of it, but I thought like, I don't know.
That seems kind of touristy.
I mean, all the investment bankers live there, right? Like half the people who listen to our show must be on their yachts. Maybe they'll let us know.
That's you saw our yacht.
Live Semi Doped podcast from your yacht. You send it to me.
[email protected]. Send me an email and we can chat.
Okay. Okay. So, wait, wait, you were in New York for the Qualcomm investor day then?
Yes. Yes. Qualcomm investor day. I was there obviously. You know, they host it in New York City because the audience is the sell-side analysts and the financial community, but they brought out industry analysts as well because they had a lot to talk about. So really the story that we'll walk through, and I want to talk through the technical bits and hear your take on some of it. The story that Cristiano Amon and Qualcomm were trying to communicate was all about their diversification away from handsets, which ultimately Qualcomm has always been, you know, a communications company, big smartphone business. Everyone is seeing the writing on the wall with smartphones, which is just we're at the top of the S-curve, everyone has a smartphone, right? There's lots of competition in the space and so on and so forth. Qualcomm has been moving into automotive and that business has been growing and it's a proof point that hey, they can use M&A to get into a new business and then build the technology, use their channel, use their manufacturing scale to grow that business and take it to market. And the whole point of this was Qualcomm trying to say we are going to do that with data center as well and here's our plan. And so ultimately I think there was, we'll include this, the image, but like they had this sort of one money slide. I think it was from the CFO's presentation where there was like three circles. It was like a donut ring slide and they were showing like okay, in fiscal year 25 handsets were two-thirds of our business. Automotive and IoT is one-third. Our goal for 2027 is like roughly half and half, half handsets, half other, which now starts to include data center. And then fiscal year 29 the goal is actually two-thirds of the business is automotive, IoT and data center and only one-third is handset. So they're trying to move sort of beyond their legacy communications only business and move into these other new businesses. Which, last point and then I'll let you chime in. Obviously communications is part of all of these businesses. So it's there, the IP and the technology is still important to the other businesses. It's just that they're going to have to build more and do more than just handsets.
So I saw the whole investor day talk. I wasn't there but I saw it on YouTube. It was interesting because Cristiano Amon said that for the 40th anniversary of Qualcomm, you know, they had the founder over, Irwin Jacobs, and during that conversation or the talk that he gave, he mentioned that he made one mistake in naming the company, quality communications. He's like, I should have just stopped with one M in the com, you know, because now it could be quality compute, but now it's a problem. So they could... hilarious.
In my opinion, they should rebrand it and drop one M.
Yeah. Totally. That's... Oh, yeah. If only he had the foresight to think 40 years ahead, right? No, that's so funny. That's so true.
Yeah. So that's the big deal. What you say is actually a big deal because this company from the get-go has always been about communications. They have a rich history. It started with like satellite communications and when I joined there in 2018, they had a tour of the museum within the campus that showed the entire history of all their devices that have been made from the beginning, you know. So it's really nice as part of the orientation is that they take you to the museum of their... it's a very inspiring way to start working there. But that's the legacy because they've been through 3G and 4G and 5G, they are also on the 6G train, we'll talk a little bit about that. But now they are saying that only one-third of that business is going to be handset based, which is a way of saying that's communications business. Two-thirds will be IoT and data centers. It's a big shift.
Indeed. It is a big shift. So, let's start with maybe the confidence question, which is, do you think Qualcomm can get into new businesses? Maybe we'll start at the top there, kind of strategically high level. Do you think they can build a real data center business? I'm going to let you have your take and then maybe I'll provide some color of what they said as well.
They can because there is no reason a company like Qualcomm cannot get into data center business because they have compute, right? They have the CPU business, they have NPUs, which they use graphics acceleration on their SoCs, mobile handsets anyway. They have all kinds of IP now through the Alphawave acquisition. I was looking at what all they got through that and it's quite a lot actually because they have SerDes IP in copper and optical. They have PCIe Gen 6, CXL for like servers and storage stuff. Then they have Ethernet IP, 800 gig, 1.6T for like switches, routers, DPUs, NICs, and they have like HBM and DRAM IP for GPUs, which is very, very important. We'll talk about that because that's one of their key technological innovations in the era of AI and the way they are doing inferencing chips. And they also have chiplet IP through the Alphawave acquisition through like UCIe or bunch of wires or other standards like this. So they are very well positioned as a company who has also shipped like billions of devices into handsets world over. So it's not like they are new to doing this. You know, I could argue that if OpenAI says like I'm now a chip company, Anthropic wants to be a chip company, they have never been a chip company. It takes experience to be one, which Qualcomm has. So, yeah, they are well positioned to do so. Although, one could argue that they are a bit late to the scene.
Yes. Yes. Okay. So let's get into that. You know, now in the grand arc of things, late and timing is interesting because obviously this has been going on since late 2022, but it's only 2026 and this is the type of technology that will be here for the next, you know, 30, 40 years, whatever, forever. And so, you know, being late now feels like a big deal. In the grand arc of time it won't, it'll just be a little blip. But the question is if you're going, what does it take to succeed? Which I think you hit on nicely, which is you need to have IP, you need to have talent, you need to have manufacturing prowess and you need to have technological differentiation. It can't just be a me-too fast follower like, oh great, a company did this, I'll build the same thing and sell it into a different market or something, right? Like that's not going to work. Like there has to be a reason for a customer to choose you. And then to the point of being late, you especially have to differentiate on some vector. I think of like when I think of Nvidia's, you know, Hopper era GPUs and then eventually AMD came in with Instinct. And what was nice was they made a different decision about the amount of HBM capacity that would be on a single chip. And that just, that small decision on one vector opened up some particular workloads that could run on like eight Instinct 300s, I think, or maybe 350, I can't remember off the top of my head. But it took 16 Nvidia GPUs because they just didn't have enough memory capacity on each GPU. And that illustrated for me that like even if you come in late, if you make some sort of different technological bet, it might unlock a certain set of customers which can help you get your foot in the door and grow from there. And so I think to your point and as we'll get into, Qualcomm checks a lot of the boxes as far as like they're a big company, they have experience, they have IP. Oh, and by the way, they were able to use M&A to get IP that they didn't have, get talent that they didn't have and kind of fill those gaps, which is something we've seen them do before in their automotive business. We could talk about that, but that's beside the point a little bit. But there still needs to be something different so that at the end of the day, a customer like a hyperscaler or an AI lab will know why they want to choose Qualcomm. And it's not just because it's cheaper or it's not just because everyone's out of supply and Qualcomm happens to have some. None of those are sustainable. Those are temporary. But there has to be a technological differentiation that really impacts ultimately like TCO for a particular workload. And this is even why we saw like the Groqs and the Cerebras have success is because they made design decisions and there was lots of sort of negative consequences, but for a particular workload, high interactivity, it was much better than GPUs could do. And so obviously Qualcomm had to come out swinging and say we think we can do something different here.
Yeah, their key differentiator here, apart from their memory architecture which we will discuss, is basically the fact that inference now is getting disaggregated. This was their key theme that went through the whole talk because they kept saying this is now disaggregated, which means that you can actually develop hardware to do a particular task. So you could put in a Qualcomm rack along with some other racks. It doesn't mean that you know you have to fill Nvidia racks with, you know, all these Blackwell GPUs along with their Vera CPUs. You don't have to do that. Now you can put a rack full of CPUs separately. You can put a rack full of low latency decode, like you were saying, the Cerebras or the Groq LPUs, and now you can put maybe a rack of Qualcomm inference chips just to do that one portion of it. If they have a high memory bandwidth, you could just do decode on that if you want to and still continue to use maybe Nvidia GPUs for the pre-fill part. So the disaggregation across the whole inference space, which is becoming more and more a theme because one size does not fit all. You've spoken about this in your Substack as well, like there is a right-sized hardware for the right-sized workload. So that is increasingly becoming a theme, which is why you can run different kinds of hardware along a common software platform, which Qualcomm also wants to have, you know, open and developer friendly for everybody to work with.
Yes. You're totally right, which is disaggregation has really opened the door to everyone. Once we started to break the workload down for inference, you know, today LLM inference is obviously the defining workload. The value is actually all being created by agentic inference workload. So, you know, reasoning models. But in breaking up that workload into pre-fill and decode, then people could say, okay, I'm late to the market, I'm just going to focus on decode. I'm going to do something very decode specific and it's going to be very different and that will help me get my foot in the door. Again, worked for the AI ASIC startups and this will have to be the mindset that Qualcomm has because ultimately, yes, would they like to sell data centers full of Qualcomm Dragonwing, which by the way, that's what they branded it for anyone who's not watching. Dragonwing is like their data center brand. Would they ultimately like to sell... and the branding was white and gold by the way, so it looks very nice. So would they like to have a data center full of white racks so that whenever on FinTwit you see a picture of, you know, someone saying, look, I built a data center and all the racks are white, everyone can go, whoa, Qualcomm to the moon, you know. Yes, they want to do that. But the reality is, which I think that they're very fully aware of, is they need to sell like a few racks into the existing data center that's full of Nvidia GPUs. And so trying to focus in on decode, for example, would be one way to get there. But okay, take us further. Let's talk... do you want to talk about the memory, the HBC, that was sort of like the big star of the show?
Yeah, let's go to HBC because that is an underlying technology that is very useful regardless of whether they want to use it in data center hardware or IoT or automotive or robotics even, because this is going to be the platform that lays the foundation for all of their businesses going forward. In my opinion, it seems like that way at least from what I heard. So high bandwidth compute is their way of getting a better memory bandwidth hardware packaged right next to an XPU so that you can get a really fast inference bandwidth like you would get with, you know, like Cerebras or Groq LPUs because those things can do like SRAM bandwidth is in the range of 100 terabytes per second and right now HBM bandwidth is about a tenth of that, it's about 8 terabytes per second, right? So they want to break through this 8 terabyte barrier which currently involves an HBM stack sitting next to a GPU or XPU and connected with maybe 2,000 lanes of interconnect between the two, memory and the GPU chips, and it is packaged with an advanced packaging substrate like TSMC CoWoS, right? So their idea with high bandwidth compute is that you take the memory and you put it on top of the XPU die and instead of connecting the memory through the side using, you know, on the shoreline density as it's called, on one edge of the GPU die you can only put so many lanes before you run into advanced packaging limits. If you put the memory on top of the XPU, you expose the entire face of the chip between the XPU compute and the memory to have interconnects. So now you can put instead of 2,000 lanes going through between these chips, you can have, I don't know, tens of thousands or even 100,000 different lanes. So what that does is if you increase your bandwidth, number of lanes by 100x, you know, you'll get 100x more bandwidth kind of thing. So this is their main idea of doing what they call high bandwidth compute.
Yes. Yes. Okay. And so there's so much to unpack here. So I'm going to walk it through again for people at a high level in case they didn't catch it the first time because I will be honest. One, the naming might not do it justice, although I don't have a better name because HBC, HBM, it feels very similar, but there's actually a lot of differences going on. And then two, there's actually some nuances that again weren't obvious to me like the very first time it came across, but as I looked back at it again as we prepared for this episode, there's some things that stood out. So I'll talk about it. So one of the problems, you know, obviously the way everyone knows the way GPUs and HBM work today is you've got the HBM, it sits next to GPU and you've got things like weights or activations, whatever, KV cache stuff sitting in this HBM and during decode especially that stuff just has to fly back and forth. It's like if you separated like your kitchen and you put your fridge in the garage and you just had to walk to the garage and get something, bring it back, chop it up, walk to the garage, get the next thing, bring it back, chop it up, walk to the garage. You know, obviously it's like, dude, what are you doing? Why are you spending all this time walking back and forth to your garage? Just put your freaking ingredients right next to your chopping block, you know? So because of course there's lots of power loss, lots of latency time, that kind of thing. And to your point, you can't... the analogy breaks down, but you can't get as many lanes and as high of bandwidth. So one point is ultimately like how do we bring the compute as close to the memory as possible? And other people have called this like near memory compute, processing in memory, stuff like this. There's lots of startups doing this. I think is D-Matrix doing something like this?
Yeah. Yeah. This is D-Matrix approach, what they called in-memory compute. In-memory. D-Matrix also puts memory on top of logic. So there is a question of really, is the Qualcomm innovation all that different is another question to ask too.
Totally, totally. So okay, so that's the first problem is how can we bring the compute as close to the memory as possible? And so what you said is like, oh well, what if you actually... the closest thing is to stack the memory on top of the compute and just literally have it as close as possible. Now there's obviously thermal concerns there. These GPUs, XPUs are like crazy hot. So can you actually put memory on top of the XPU? And as I looked closer in it, and we'll try to include some pictures as well, my understanding is they are not actually stacking the memory on the XPU, but they are putting a logic chip under it and they're putting it close to the XPU, but they're actually offloading some of the workload to live on that logic chip. I don't even think it's necessarily like a programmable logic chip. I have a feeling that it's almost more of a true accelerator in the sense of accelerator. And so for example, like my guess would be like maybe you have during inference some sort of primitive, some sort of function that's run all the time like softmax or something in attention. Can you actually take that off the XPU, put it in the logic that's right under the memory? And the idea is like fetch what you need from the memory, do those primitives right there, and then send the result back. And that would obviously help you solve like the thermal issue of you're not actually putting the memory on top of this crazy hot XPU, but you're still using advanced node logic and putting the memory on top of it and then just accelerating an even finer grain piece of the workload. And so again, and I'll let you jump in because I know you'll have thoughts and reactions on if I'm interpreting that right. That would be... this would be the name that we've always used before AI, which is like an accelerator. So like a camera, like an image sensing processor or any other little like acceleration that we've done in the past like with graphics, which is you're like, dude, we just keep doing this particular loop over and over on a CPU. Why not build a little, you know, Qualcomm does this all the time in their Snapdragon SoCs, like build a little thing to do some image processing that's just dedicated for that? And in my... it felt conceptually like they're saying that take a little tiny piece of the inference sort of primitives, move it off, but then also it's very important that it's very close to memory because it's very sort of memory bandwidth intensive. So I'll pause there. What is your reaction?
So when I saw this being announced on the talk, it was very confusing the way it was drawn out because in the first example where they show how the current solution works, they actually had a GPU and they had an HBM there and there was like this gold colored like back and forth happening. It was very beautiful to look at and you could see that data was like this gold data was going up and down the HBM stack and was like going around into the GPU. So that's how it works today. But then when they put in their solution, which is basically their high bandwidth compute, they still... they showed a stack of LPDDR and then they said something about XPU. Okay. And then sitting next to it was an SoC, right? And then they were showing the same gold thing which presumably is data going between the SoC and the LPDDR stack. So for somebody who's looking at this, it looks very similar to the HBM one. Like so I'm like, what's different? And then they go on to mention that no, you don't need advanced packaging anymore and that you could do standard packaging because you don't need that bandwidth anymore. So my interpretation of this is that it's not really... the logic die under this memory is not really any simplistic logic die of any kind. It is a full, full-up XPU, right? Because that's the only way that you will not need advanced packaging to make this work. You have to do the full thing there and that's what D-Matrix is doing. They have a... they do all their compute right under memory.
Okay, so when you say XPU, what is your definition of XPU then in this particular use case?
This is the thing that does matrix multiplications. That's what I call an XPU.
Yeah, so do you think they will do any matrix multiplication application anywhere in the workload right there under the LPDDR?
Yes, I think so. It's going to... to me it looks like it's very similar to what D-Matrix is doing because D-Matrix, digital in-memory compute, and their Raptor, which is the next generation of 3D RAM which they call 3D DRAM, those slides are out there. They previously announced this stuff so it's nothing that's unannounced or anything. They actually have a full-up XPU chip that does all the compute under the DRAM die and that's it. That's their Raptor chip. So this has to be something like that. This SoC sitting next to it is doing something else, right? It could be a mobile SoC, like if you're using AI on a phone, it could be a mobile SoC. If you are using AI in a car, it could be an automotive SoC. But it's not doing matrix multiplications. It's doing SoC stuff.
Sure. Sure. I mean, so I agree with you. I like conceptually the idea of essentially like a tensor core coming and moving over and doing all the matrix multiplication. I totally agree that that's what makes sense. That's what you would want to do is get all of this multi-dimensional data, put it right down into some sort of like tensor core type thing and have it do all the mat muls. It would be nice to understand the clarity of what else is in this SoC. Obviously, right there there's I/O communication. There's orchestration of the whole workload. There's communication with CPUs. So there's all these other sort of like blocks, if you will, that don't need to live right there. But I do agree with you that it seems like conceptually anything in the workload that's related to the specific matrix multiplication stuff would live under the memory.
And you mentioned earlier that this technology could actually come to the edge, come to phones, come to auto, come to wherever. And I think that's also the key point here is if people are thinking like... if people think like GPU with LPDDR stacked on top, then that might be a little confusing because it's like, oh, are we putting big GPUs into a car now? But the point here is that the technology, the concept of the technology, which is if your car is going to run neural nets, wouldn't it be nice to have the matrix multiplication under a big stack of not HBM but, you know, LPDDR, something that maybe like lower power, cheaper, higher capacity, but that would still let you do essentially generative AI at the edge, presumably with even bigger models for example.
Yeah, I want to talk about this memory stacking part first because we will definitely talk about the automotive, IoT and the robotics aspects which they really want to get into in the future and I think they're well positioned to do it. But there are like several issues with the way this memory works to begin with. First of all, you already mentioned the thermal aspect. It is pretty challenging to stack logic on top of GPUs or XPUs like this. The second thing is that there is a fundamental size mismatch, right? These XPUs that sit under any form of memory, like even when D-Matrix does their Raptor, they are pretty big dies because they are reticle size GPUs that sit there, right? Because Nvidia even like packages multiple GPUs in a single like, you know, Rubin or whatever nowadays. So it's easy to assume and it's okay to assume that the XPU will be a reticle sized compute unit, which means it's like 850 square mm or anyway it's above 800 mm squared of chip. Now if you have to stack DRAM on top of that, can you imagine how big of a DRAM die you need to put on this chip? And can you imagine the planarity requirements of actually hooking this up and keeping it planar in a thermal environment? Like stuff is heating up and the planarity between the top die and the bottom die needs to be maintained across without having the bumps rip off each other. This is a challenging problem.
