So today I think we are going to discuss one of the most critical issues in the AI space, what we call the memory wall, and why memory is a bottleneck right now in the development of AI. And I'm very glad today to be joined on stage first by someone who doesn't need an introduction, Pat Gelsinger. As you might know, he is a current partner at Playground Global, a VC firm, but before he was the Intel CEO from 2021 to 2024. So please applaud him.
Pat doesn't need any introduction, but I will give you a few biographical points about Pat for you to grasp how deep his experience in the AI space and the semiconductor space is. So Pat joined Intel when he was 18 years old in 1979. He got promoted at Intel, got involved in the x86 processor architecture, which is the dominating architecture for desktop. All desktops have x86 processors inside them. Then Pat was promoted as the CTO of Intel in 2001 and left in 2009, went to EMC, and was called back by Intel to head the company in 2021. So enough biographical points. I think the goal today is to help people understand why memory is an issue in the AI space. So Pat, if maybe you could explain to us why it's such an issue, and please speak to me like if I were a 5-year-old child or Gordon Ramsay.
Okay, so let's dive in. So why is memory a problem today? You have, I call it the threefold problem today. First, AI demand is somewhat insatiable. So we just have so much demand for AI. The token economics, the value of tokens is essentially unlimited as they become more and more intelligent. So insatiable demand. Second, we're now making memory requirements for that computation larger and larger. Context windows, model sizes. So I want more memory. But then the last thing is, when you type in a prompt, how long do you want to wait? You want first token latency to be dramatically low. So you want it to be large and fast. So you have insatiable demand. You want larger and larger memory windows, and you want faster and faster memory. And that trio has led us to this period of time where memory has never had it this good. We used to joke in the semiconductor industry that if you were a memory company, you had one good year out of four; the other three sucked. That defined the memory industry for essentially 50 years. We're now in five good becoming great years for memory. And now with HBM, a memory should never have 90% margin. It's just insanely good that we're seeing this demand that we're in today for memory just because AI demand is so insatiable.
Okay. So, yeah, typically, when I enter a prompt, the latency that we have behind it fast, and you want it to be large. Because I wanted to know everything about everything I might query it on and I want it to have context on me and everything about me as well to make it most appropriate. So that just makes memory demand and memory bandwidth extraordinary.
Maybe I don't want to preempt what we are going to discuss next, but maybe if you can actually explain to our audience in very simple terms the solution that the industry has put forward to solve that memory issue in the industry.
Yeah, so obviously we've heard about HBM, high bandwidth memory. And HBM is a lousy memory. I know my memory friends are about to come on stage and calling their baby ugly is not maybe the best way to make friends and influence people, but it's a lousy memory. You're stacking DRAM. DRAM doesn't like to be stacked because essentially you're creating a thermal sandwich, that's bad. It's shoreline bandwidth limited. Essentially I'm trying to get bandwidth through this point-to-point connection between the memory and the GPU. So I have bandwidth limitations associated with it and it is bitwise inefficient. For every bit that I'm creating for HBM, I've given up four bits of memory nominally. So essentially it's bit inefficient, it's power inefficient, and it's bandwidth inefficient. It is just the best one we have right now. So HBM itself is a bad memory, it's just the best one we have to satisfy the near-term demands of AI, but fundamentally we need much better memory architectures if we're going to realize the full benefits that we would hope AI will enable for the future. So it may be the tallest we have for this requirement right now. So we have to do better in memory over time if we're going to fulfill our dreams for AI.
And since you talk about better solution than HBM and that you're now in the VC space, can you maybe give us a hint of the innovation that is happening in the sector to make it even better or to solve this HBM kind of imperfection?
You know, obviously we now have three trillion-dollar market cap memory companies. It's just extraordinary what's going on. And with that we're seeing enormous amount of R&D investment going on. But fundamentally, the idea that memory sits horizontal to the compute requirements. I expect that the majority of memory needs to be integrated into or on top of the compute requirements. Different memory architectures for the future. A number of companies are working in this space. People like Cerebras, part of what Groq was acquired by Nvidia because they were solving the memory challenge. We have companies like our company D Matrix, NextSilicon that are doing stacked memory on top of the compute layer as a much more efficient way because then you're not shoreline limited. You now have the full XY, either hybrid bonded or micro bumped onto the GPU as much more effective ways to create memory bandwidth. So these will be breakthroughs that we'll see before the end of the decade really starting to change the memory industry and solve the AI memory problem.
But I understand from what you said that until the end of the decade, HBM will stay the best technology to process AI, right?
Yeah, supply chains for semiconductors are long. They're expensive, they're capital-intensive. A great innovation today may take three, four years to become volume in the industry. So I fully expect that we're supply limited on HBM for several more years, irrespective of how much capital gets invested because that's what the designs are. And we're going to see continued demand essentially unconstrained. That said, these innovative architectures will solve capacity and bandwidth in fundamentally better ways that are more thermally efficient, bitwise efficient, and ultimately deliver bandwidth and latency improvements at scale. So memory will get a lot better because fundamentally AI is a memory problem. The computational, once we've solved models like transformers and are able to take advantage of it, everything else is about doing those functions over and over again on large memory sets, on large context windows. So fundamentally, memory innovation, it's never been a greater time to be a memory person than for the age of AI.
Great. Thanks, Pat. I think we need to be joined on stage by the other speakers. This was actually the easy discussion, and now it's going to be a debate. So not everybody is expected to agree. So let's bring our other speakers on stage.
Yeah, I think we only have like four mics, so we'll have to share them. It's pretty unusual to be as many people on the same stage. So guys, we just had Pat giving his perspective. It was super interesting. Maybe if you could each introduce yourself very quickly so that people can understand how complementary you guys are to the discussion, and then we will discuss the many topics that we want to touch upon.
Wait, I'm going to say or...
We see s'il te plaît. Avec plaisir. You speak French, right?
Yeah, we, we, we. Okay, sorry. All right. We thought that was going to be an opening joke for this audience and then Louis forgot to deliver it. We were trying to surprise our other presenters here to tell them that we were going to do this whole presentation in French, but okay.
It didn't land as well as I'd hoped.
Anyways, and Roman, I'm managing partner of Maverick Silicon. We are a private focused semiconductor and AI infrastructure fund based in Santa Clara.
Hi, I'm Vijay Shilpikondala. I'm operator turned investor. I've been doing this for the last 11 years in the public markets. Before that I was a hard tech engineering researcher and operator. This is a disclaimer. None of this constitutes investment advice. You should understand that my employer has positions in this space. So you know where my bias sits. You can read the full disclaimer here. Happy to discuss this. I have a focus on systems co-design. My PhD's in mechatronics and systems co-design. Systems co-design is how you integrate hardware and software systems from the get-go. It's been a lot of fun investing with that theme, especially because it's a foundational aspect of a lot of AI systems.
Hello, my name is Hoshik Kim. I'm senior vice president of SK Hynix. I do system architecture research and design for memory solution, memory subsystem. My job is actually unique at SK Hynix because right now memory is so critical component in overall AI computing infrastructure. We now have to understand the whole system: AI model, software, hardware accelerator, and memory. So that's what I do. So we approach this memory bottleneck, memory problem from a system perspective. Thank you.
Thank you guys. So maybe a quick summary for those who have just joined. So every time you type a request on ChatGPT, the brain, the GPU, is waiting for the memory to give the data that it requires to write the proper answer. And so that's the issue we have in memory. And the solution that the industry has invented is called high bandwidth memory, HBM. And it's basically increasing the amount of information that is actually flowing to the brain, to the GPU. But I'm going to open a provocative question since Pat, you actually said that HBM was an imperfect solution to a very complex problem. Maybe Oshik, I will ask you what you think of this statement?
No, actually I agree. I kind of agree with Pat. So memory HBM is not the answer.
So SK Hynix is admitting that HBM is a bad memory.
Oh no, nothing is bad memory. It's a good memory, but it's not the final answer. Actually HBM will not solve the memory wall problem. The memory wall problem is an inherent AI problem. So we cannot avoid. So we try our best to provide higher bandwidth memory with HBM technology, even though it has some limitation as Pat already mentioned: thermal, IOs, cost, a lot of stuff. But that's the only thing we have right now to help ease this memory wall problem, memory bottleneck problem. But at the same time, we are working in different directions, multiple directions to solve this memory wall problem with different solutions.
Okay, great. So maybe turning to Andrew and VJ because HBM is believed by many people in the community to not be a commodity anymore. So for many years memory was considered to be a commodity, and now the view of many people has changed and they say that HBM is not a commodity anymore. Maybe can you explain why so many people actually have this belief?
Maybe I'll kick it off and then pass it over to VJ. So there is a fundamental change happening just in terms of the co-optimization between the logic, which could be a GPU in most cases or another AI accelerator, and the memory itself, where there's a much tighter integration that frankly didn't exist with the memory that you'd find in your iPhone for example. And as a result of that, it's much more difficult to swap and replace one supplier with another. So you are making these design decisions years in advance, working closely between the memory partner and the compute partner, which by definition just creates a much tighter relationship. It's just harder to disintermediate. That makes it much less of a commodity. But I don't know if you'd add anything.
Yeah, I agree with what Andrew said. I think it's important to frame this based on the historic context that we have here. So I'm building on Pat's comments before. We have inherited a von Neumann architecture way back from 80 years ago. So what we have is a processor which sits far from memory and data being moved back and forth. So imagine a factory which makes goods very good, but then it cannot store much; most of the raw materials are stored in a warehouse far away and there's a narrow highway connecting the factory with the warehouse. So we are in this von Neumann architecture and the importance of that becomes very clear if you look at some of the leaders of the industry. If you consider Elon Musk, who one could argue is one of the best engineers in our times, what he is focusing on right now, I think Gavin Baker in the recent all-in podcast said that memory is the biggest bottleneck out there, and I want to expand on that a little bit. It's important to throw scale at a lot of problems. Elon does that with whether it's gigafactories or data centers in space. He calls it the Kardashev type one and type two sort of Kardashev climb. But at each rung of this Kardashev climb, you need to pay this memory bandwidth tax. And the question that you asked, Louie, when it comes to HBM4 going forward, it looks like we're moving towards more custom-built designs than commoditized designs. The reason for that is the base die is likely to be a logic foundry die, which basically means that you now have a better traffic control system with a logic die if it is based on a more advanced logic node. So this I think is one of the most long-standing, most challenging problems that we have in the industry, and now we have the greatest capex cycle that we have seen in our time. So we're at this juncture where the memory wall and memory bandwidth have become very important problems.
Thanks. So I think something that is important for the audience to know is that 40 years ago you had almost 20 memory players in the space. Now you only have three: SK Hynix, Samsung, and Micron. So you have had a massive consolidation. So it shows how complex and how difficult this space is and how cyclical. So you guys are really survivors, let's put it that way.
You almost died? Yes, several times. Okay. Congrats on being here today.
