Welcome, listeners. Thanks for listening to the very first episode of the Semi Doped podcast. I'm Austin Lyons. I write a Substack called Chip Strat, and with me is Vik, who writes Vik's Newsletter.
Hey Austin. Yeah, I really love the name for this podcast. I hope we continue to love it like five episodes in. By the way, for listeners who are not familiar, my name is Vikram Sekar, and I write a Substack called Vik's Newsletter. We write so much semiconductor content between Austin and myself that we decided we should just talk about it on a podcast, and so here we are.
All right. So we should talk about what's on everybody's mind right now. Nvidia decided to drop a whole lot of coin to buy a company called Groq, and pretty much everybody online lost their mind about the sheer size of this deal and what it means really for Nvidia. Like there was so much narrative about, oh, HBM is going to be affected. Oh, GPUs are dead. This is the new thing. We are in the age of inferencing. SRAM is what it's all about. Anyway, there was so much noise that I spent so much time reading all these threads and figured, you know, I think we should kind of sit down and have a little chat on what's going on here and kind of break down some of the more technical and business aspects of what's going on here and maybe give some more context. What do you think?
Yep, that sounds great. Let's talk Groq. So, for listeners, this is Groq with a Q, the chip company. And yes, the news dropped. It was Christmas Eve, and I felt like that was strategic. Like it was surprising, and Groq had an announcement, and Nvidia didn't really say anything, and then everyone signed off for the holidays. And so I think that also helped lead to a lot of speculation, you know, amongst us who track the industry closely. And so yeah, it was a what we read at first. I actually read it as it was an acquisition, and then quickly people came and pointed out and said, no, no, no, this is a $20 billion deal that Nvidia is doing where they are acquiring a non-exclusive licensing agreement of Groq's assets for $20 billion. And I think Groq's announcement said, you know, CEO Jonathan Ross and the CTO Sunil Madhav or whatever his title is will be going to Nvidia, as well as many key employees. And so I think right away, I was first a little bit like, oh, is this just like the leadership going over? But then I think later there were reports that actually it's like most of the company is going. But right off the bat, it was a little surprising in that like, oh, this is not an acquisition. This is more of this like new-fangled pseudo acquisition that we've seen with AI labs. Did you have a take when you saw that news?
Yeah. So I was actually disappointed when I saw that it was kind of an acquire kind of deal initially, or at least what I thought it was, because I think it sets a really bad precedent for startups in the Bay Area or anywhere, really. Because if they only were to hire the key executives or the, you know, key brains behind the project and not really take the whole team with them and just pick the brains and the technology, then there is really no incentive for anybody to work at startups anymore, because everybody knows the payout is going to go to the top, I don't know, two, three employees, and nobody else is going to get anything out of it. But I'm happy, like you mentioned later, even I found out that it's not like that, and they did right by the employees of Groq. So this is a good precedent. I'm actually happy with the way things turned out in a sense.
Yeah, totally. It did have a happy ending, and I have seen now there's talk where lots of people are saying, hey, we should make sure that this gets codified somehow, that companies can't come in and sort of like raid the executives and the IP, and the employees are left with options that they were promised. And that's part of the comp package for startup employees is this idea that your options could be worth a billion dollars someday, and if that can't manifest, then it does sort of break the whole premise of startups in the first place. So yes, it was happy to read that Jensen did right by the Groq employees.
Why do you think that the whole narrative came about after the acquisition, or rather acquire or whatever it is, that GPUs are dead? Why did we get there?
I think that the, you know, zooming out, looking back in 2025, the narrative like coming into 2025 was really about like GPUs versus XPUs and is Broadcom and their hyperscalers going to eat Nvidia's lunch. And so there's a lot of pushback about like, no, no, GPUs are really flexible and XPUs like, you know, where are they? You know, Google's got a roadmap and they've shipped seven, you know, generations, but at the time we had barely seen Trainium. And I don't know, no Maya from Microsoft, that just seems like a design on paper. And so it was a lot, you know, the pendulum was in GPUs' favor and a lot of people pushing back on XPUs. I think throughout 2025 then there was started to be a lot of talk about TPUs, especially with like Google potentially selling TPUs to others, and the pendulum started swinging a lot. There was a lot of narrative and conversation around TPUs, and so it felt like sort of a GPUs versus TPUs dance. And meanwhile, at AWS re:Invent, there was talk about Trainium 3, and so again kind of TPUs and XPUs were a thing. Now, of course, all of these use HBM, and so then it is a little interesting that here at the tail end, all of a sudden LPUs slid in, and everyone's like, HBM is dead.
But the thing with LPUs, which is Groq's solution, and why everybody declared HBM dead when this acquisition news showed up, is because it uses SRAM instead of HBM. And SRAM is actually so much faster than HBM is, even though it's actually high bandwidth memory. SRAM is actually faster. The only problem with SRAM is that its capacity is highly limited. So the question immediately comes up is GPUs are great, and HBM is highly supply constrained, and it requires chip-on-wafer-on-substrate packaging from TSMC, which is very hard to get and is also highly supply constrained. And I think that I read in the news that several Google executives were affected by the fact that they could not, you know, get sufficient HBM procurements on time. So it's like a serious matter, this thing. So the SRAM solution from Groq provides an entirely different alternative universe because you don't have to deal with HBM, because HBM is only made by three companies: SK Hynix, Samsung, and Micron. Did I get that right?
And SRAM is actually made on logic chips. So TSMC could make it. There's no need of all this fancy packaging. And you don't have to package it, you know, try to stack DRAM die or package it on a co-interposer or any such thing. So it like takes out so many capacity bottlenecks from using HBM. And so that's why it seemed to everybody that, oh, wow, this is a solution that, you know, breaks all barriers and it could take over. The main problem lies in the fact that it does not have capacity. So if you were to run a Kim K2 model, you're going to need 1 TB of memory to run that model, even for inferencing. And that is simply not possible with SRAM-based chips like Groq LPUs, because you're going to need a lot of them, because each Groq chip has 230 megabytes of SRAM. So you can imagine how many chips you would need to run 1 TB requirements of memory. So that introduces a whole dimension of complexity, because now you will be bottlenecked by how the networking between so many GPUs work and how the interconnect bandwidth and limitations start to show up in the big array. So it's not as simple that just because you have SRAM basis that you can just substitute HBM and take over everything.
Right. Totally. Totally. And you make a good point, and I think the framing that leads to all this is not HBM versus SRAM, because I think people who aren't quite as close forget that every chip uses SRAM for local storage, for cache, right? CPUs, GPUs, LPUs, TPUs, whatever. But it is HBM versus no HBM. And so yeah, Groq's LPUs don't have HBM. And so to your point, that's like the key insight, which is no HBM means no capacity constraints like co-ass and HBM constraints totally. But of course, you know, you point out, yes, but there's a trade-off. SRAM, it takes up a lot of die area and thus is expensive. It's very fast, it's very local to the compute, very fast, but with only, to your point, 230 megabytes per chip, you're talking like tying together thousands of chips to run a really large model, and even, you know, several hundred, 500, 600 to run a smaller or midsize model. And so then that's definitely the trade-off. Now, if you've got hundreds of chips, you have to ask, how are we tying all these together, shuttling all the data around? And of course, what is the power cost of all of those chips? What is the real estate cost of all those chips, right? And so of course the economic cost might be a lot less because by the way, you know, one might ask, well, why did they only have 230 megabytes of essentially SRAM on these chips? And you know, you have to remember that this chip was designed many years ago, I think 2020, on like 14 nanometer technology, and this essentially was designed prior to ChatGPT exploding. I mean, transformers were a thing, but I don't think that they were probably necessarily designing this chip with these really large language models in mind, which is how we end up in a place where you have to tie hundreds of them together to run today's large language models. Now, back to real quick, going back again in all the narrative, you know, some of the nice things, not only do you not have the HBM constraints and the cost constraints, it's on 14 nanometer technology, so surely these chips are cheap. So there is the whole like, well, yeah, you need hundreds of them, but they're probably way cheaper than buying a GPU. But yeah, let me pause there and ask you. So going back to LPUs, so we know they're cheap, but we know the trade-off is they don't have much memory. Do they have any other benefits using LPUs?
Yeah, LPUs are an interesting architecture actually. It is substantially different that we should talk about this for just a minute because GPUs are, you know, graphics processing is what they've been used for, but they're very general purpose. You can do a lot of stuff with GPUs. You can, you know, do ray tracing or whatever. You can do AI with it. It's all about how you program this thing and it does whatever you want it to. LPUs are actually quite different because they are like a finely tuned machine. An analogy I really like is that they're like a factory where every worker's movement is coordinated to the last millisecond before their shift starts. So that whenever the product has to run through that factory, there is almost no wasted time, motion, energy that is used to push the product through the factory line. So that's kind of the basis behind what LPUs actually do. So they fundamentally work based on what is called a very large instruction word or VLIW, and this is so unique that actually I should take that back. It's not, it has been around a long time, but its context in inferencing is important to understand because typically how processors work is that when you give it an instruction and say, hey, go and compute this for me, what it does is it takes the instruction and the hardware decides, okay, what instructions need to be executed first? Do I need to make the multiplications first? Then do I need to add them? So that means wait, the multiplication should be done before I can add them together. So the sequence has to go this way, and then I can probably take this and load this into memory, and so it decides all this at runtime. So this is how general processing works, you know, in GPUs and things like that. And the very long instruction word approach by Groq kind of flips the whole argument. So nothing is done at the hardware level, everything is decided beforehand. So what that means is that you have to take a bunch of these operations. So if you're going to do the addition, if you're going to do the multiplication, if you're going to load it in memory or store it somewhere else, everything has to be defined beforehand, and you have to specify it exactly, and the hardware is just dumb. It just does exactly what you tell it to do, right? So this is very difficult. So think about that. Not only do you have to orchestrate hundreds or thousands of these Groq chips, but you also have to be very specific about how the instructions are executed on this thing. So that makes it very, very difficult for people to write code to it. And it is actually a compiler nightmare because it is so complex because everything has to be decided beforehand. You know, one analogy to this is, actually before that, I should explain another thing. There's another benefit to LPUs, and that is that it is very deterministic, and it kind of falls out of the process of predetermining what has to be done well in advance, right? Because if a compiler, if the instructions have to be decoded, what actually happens is it has to determine when it's going to access the cache, if the data is available in the cache or it misses the cache. So there's a lot of variability in how general processing works. But with Groq's VLIW approach, it is fixed because the whole orchestration, the choreographing of all the instructions are done beforehand, right? So you can exactly count to how many clock cycles certain instructions will take. So that makes it very deterministic.
I'm going to give one last analogy to explain how determinism works here. Because let's say I'm giving you, Austin, instructions to go to a restaurant in, let's say, San Francisco. So I'm going to tell you, hey, Austin, just go down to Union Square. You know, go four blocks east, you'll find a gas station, and diagonally opposite from the gas station is going to be this restaurant I'm talking about. So what that requires you to do is first you need to think about how do I get to Union Square, and then wait, then I have to go what, how many blocks east, and what's, and then you need to find the gas station, and then you're going to find the destination, right? The Groq approach to doing that would be I'm going to give you the GPS coordinates to Union Square, and then I'm going to tell you exactly how many meters you're going to go from there in a given direction, and at what angle from that destination point the restaurant is at, right? So this makes it very difficult to program things like this. So although it has its benefits in terms of very low latency because nothing is determined on the fly, it's all predetermined, and you know exactly how long it's going to take. So this has important use cases which we should talk about actually.
Yeah, definitely. Definitely. Let me summarize it for listeners. So Groq's LPU, architecturally, it takes the complexity and moves it from hardware into software. So the hardware is very simple. No scheduling like a CPU, for example. Anyone who took a CPU architecture course would remember like out-of-order scheduling and execution, right? You know, the hardware might just look at the instructions and schedule them differently to try to optimize on the fly. And what you're saying is that with an LPU, the hardware is very simple. All of that scheduling is done in software by the compiler. It takes a look at the program, decides how to order everything, and then gives precise instructions such that when anything is running on the LPU, everything is known. And that's why it's called deterministic. Like we know exactly where the data is and how long it's going to take to flow through and when it's going to come out the other side. And so an LPU has ultra-low latency. It's deterministic, and the determinism comes from how the architecture is built and designed. And then the ultra-low latency is just like everything's running. There's computation, and then there's accessing memory. And the memory is SRAM. It's fast and it's close to the compute. And then the determinism is just, you know, where everything is going to be. Now GPUs, you know, are more general. Now, I will say with data center kind of cloud GPUs, obviously they're not being used for graphics anymore, and some of that's stripped out, and they're definitely tuned for AI. You can think of them as AI ASICs in a sense. Even like with every new generation of Nvidia GPU, they tune it specifically for AI workloads. For example, spending more compute area and die area on lower precision like, you know, FP16 and not spending as much on FP64, right? So they are making architectural decisions that are tuned for AI, but at the end of the day, it's still general enough that if you want to run all sorts of programs and with different quantization, like it can handle it. And so there's some trade-offs just on the way they're designed. LPU, deterministic, ultra-low latency versus GPU, flexibility, dev-friendly, of course. And, you know, we can get into some of the trade-offs of the LPU, but let me hand it back over to you to keep going.
Yeah. So now we have established that LPUs cannot hold large models because it is SRAM-based, and SRAM does not have that capacity. SRAM has stopped scaling quite a few generations ago. Maybe we'll see small improvements as we have gate-all-around, we have a little bit improvement from there. Maybe as we go to complementary FETs, we'll see more. But essentially SRAM doesn't get any denser, and therefore it's going to be expensive to begin with. So you just can't stack SRAM the way you do HBM. There are many reasons for that. Maybe we shouldn't, you know, get into all those reasons right now, but fundamentally it's limited in capacity. You cannot run big models, whereas you can run really big models with HBM, and that has implications on what kind of use cases HBM-based inferencing can do and what SRAM-based inferencing can do.
Right. Yes. So what, so I'm going to ask you, because I really like your article on Substack where you refer to hyperspeed inferencing using Groq and SRAM, which is a very important use case that we do need. It's not always about serving the large mixture of expert models that we use day-to-day nowadays. There are so many applications that require just small models. So I'll let you explain some of these awesome applications you had in your article.
