Each wafer consumes about 23 kilowatts of power. That's like enormous. Like if you think about a one-volt supply that is feeding these GPUs, you're talking about something in the tens of thousands of amps of current that have to flow into a single wafer.
Hello listeners, welcome to another Semi Doped podcast. I'm Austin Lyons of Chipstrat, and with me is Vik Sekar from Vik's newsletter. So Vik, we're recording this. It's the morning of May 14th, Cerebras IPO day. What do you say? Should we talk about Cerebras?
Yes, it's all about Cerebras today. That's it. Nothing else. Single focus topic. No deviations.
Totally. I love that. Which, by the way, I don't even know how to pronounce it. Cerebras.
Cerebr. Cerebras. Yeah, whatever.
How do you say cerebral, right? Yeah. Cerebral. Cerebras. Yeah, totally. So, listeners, you know who we're talking about. The big wafer company.
Yes, those guys.
All right, so let's see. Okay, so they priced their IPO at 185 bucks and they raised 5.5 billion, I believe. I was looking at it this morning to see like if the stock price is immediately like bouncing around and stuff, but it was just pegged at 185 as far as I could see. So, I don't know if there's some price discovery thing that happens early in the morning or what.
Yeah, this was an insane IPO because it was oversubscribed massively, and they used like an eBay bidding style approach where they were like, okay, how many shares do you want and at what price are you willing to go up maximum, pretty much like eBay does. And then people were like putting in those orders. It's kind of insane. Then what happens is, like Bloomberg reported that ARM and SoftBank came in last minute and tried to buy it at the 11th hour. Typically, like an eBay snipe, you know, you come in last minute with your bid, the eBay snipe. Didn't work anyway. Didn't get bought. But I've been tracking it all week, and initially it was like, oh, it's going to be priced at 135, and I saw it was going to be priced at 150 to 160. Finally, it came out to be 185. I believe they were intending to raise like 3.5 or 4 billion. Anyway, it came out about one and a half billion above that. It's insane.
That is crazy, man. Yeah, good. You know, congrats to everyone who has equity in that, the team and all the venture capitalists. So, what do you say? Should we remind listeners at just like the highest level like what Cerebras builds?
Yeah, there's some technology and history behind this whole thing, which is an interesting discussion that we can have, at least for me because I'm such a technology-minded person. So let's go get into it. Right. So this wafer company that we were talking about, Cerebras, right? The reason we call it the wafer company, the guys who make the wafers, is because typically how wafers work is just like a giant dinner plate, and then you got like many, many little GPUs on it. And at least this is what like Nvidia does. So they take out all these GPUs and package them separately. They just cut it out of this dinner plate and they package it and they ship it. You know, that's how typically GPUs work. But Cerebras was like, why do we have to cut up this stuff? We want to keep the whole wafer as a single chip. So all those chips that you would have otherwise cut out, they just hooked it up together with metal lines on the wafer. And that's how it came about. And they were like, okay, now this whole wafer is a chip, and that's about it.
Mhm. Which I will say it's pretty intuitive in that if you look at like Nvidia's roadmap, it was one die per chip, and then it's like, oh wait, we want to scale bigger, so let's have two dies. So you're taking this wafer, you're dicing it up into individual dies. It's like a checkerboard, and you're cutting out all the little checkerboard squares, but then all of a sudden they're putting them back together, and then they want to go to four dies, you know, and you can imagine a world where people want to go to eight dies. And so like in the long run, and by the way, when you cut these up now, they have to communicate with each other. If you're lucky, you can stitch them together and it feels as if they're one piece of silicon, but otherwise now you have to go networking and even switches and whatever. So like conceptually, I think everyone can understand like, oh yeah, don't cut it up, leave it all on silicon, let it communicate all with each other, have, you know, better network bandwidth, etc.
Yeah. So that's the whole idea of this, and the beauty of this is that you can fill all of these chips with like SRAM in it. And since SRAM-based accelerators are so much faster because the memory bandwidth is like so amazing, you can get the entire wafer's worth of SRAM for inferencing, which is amazing in theory, right? For certain use cases, which we'll talk about, it's amazing. This whole wafer, if you look at the wafer-scale GPU that we're talking about, it's about the size of, I would say, it's like 60 Nvidia H100s, right? And it consists of about 84 reticles, which is like basically that one shot that people were cutting out, right? It contains about 84 of them stitched together in a grid format. And so this is a piece of engineering, and we'll get into why that is so amazing. Now, for anybody who's dealt with these wafers and dealt with silicon technology before, one thing is very clear: you can never make a perfect wafer. A wafer is always going to have defects, and that's why you always hear people talking about yield, or what is the yield of 18A, and how that means how many defects does this wafer have, lower the better. And every wafer has them. There is simply no way to avoid this whole defect thing. And so now, what does that mean for Cerebras? Like, how come the Cerebras wafer is defect-free? No, it has defects. And the way it works is now instead of these giant GPUs that are there and stitched up, which is the analogy we use to just explain the idea, in reality each of them are actually very tiny. They're not as big as a GPU. Each of them is like much, much smaller. It's like 1/100th or 1/120th of a GPU in size. And these are their like processing cores, right? They basically have a little bit of processing and a little bit of memory each one, little bit, little bit. These little thingies are the ones that are like ultimately all connected together in the whole wafer. There are about a million of these things, okay? And out of the million, the exact number is like 970K or something. Anyway, simply just think about it as a million. Out of these million, about 900,000 of them are the ones that are actually working at any time. 900,000. And the reason it's 900,000 is because you have to overcome these defects. So whenever they figure out that this particular core has a defect in it, what they do is they have a fabric, networking fabric that is on the wafer, and all they do is they just route around it. Like, oh, this chip is bad. So let's just go to the spare one right above it, and we'll route around it. Like, we'll hook up the wires, just avoids the defect. So they look at a wafer. I don't know how they do this, by the way. Do they inspect every wafer? Because every wafer has different defects. Anyway, they look at a wafer and be like, okay, like this, this, this, this, this, you know, these processing cores are all terrible. Let's route around them. And you reconfigure it. You get a defect-free wafer.
Which has like 44 GB of on-wafer SRAM operating at 21 petabytes per second memory bandwidth. That's amazing.
Amazing. Yeah. Two things I want to add in here. So for listeners, you know, zooming you back out. So when Vik is talking about yield, as you know, because there's just like this statistical noise, the stochastic things happening, you get a dopant in the wrong place and something might short circuit or it might be like just an open circuit and it doesn't work. If the size of the chip gets bigger, then you have, you know, it's its area. So you have like a much more significant surface area for a defect to happen. And so the bigger and bigger your chip gets, the more likely that that chip is going to have some bad defects. And so you might think, oh well, if the chip is the whole wafer, surely every wafer is going to fail, right? And what Vik is saying is, no, no, no. They're making a wafer, yes, but it's full of all these teeny tiny little cores. So each tiny little core has a tiny surface area. So it's going to have fewer defects. So each core will have good yield, but yet at the size of the dinner plate, you're still going to have some that don't work. And so literally, I think when they power on the wafer, they just test every core, and maybe it only happens once. I don't know how often this happens, but they test every core, and then yes, whichever ones don't give a signal back, they say, okay, row 10, column 13, that core is dead. And so you just map around it. And then they just know that once they run their software, that probably some sort of orchestration system knows to not orchestrate to that little area. So pretty cool. It's a really interesting, innovative way to tackle like yield and sort of like harvesting good cores and routing around other ones dynamically. Then I will say also Vik's SRAM point. So there's different ways to store memory. We've talked a lot about memory. SRAM is with transistors, six transistors. And so that's the beauty of having a big silicon wafer just full of transistors is you can allocate transistors to memory, fast memory, or you can allocate it to logic. And in this case, they're allocating about half of all the transistors on the chip to 44 gigs worth of SRAM, which ends up being quite a lot.
Were you saying that I think each compute core is roughly 50/50 compute and SRAM? Is that how it is?
I think so. Yes. Because I think they want the SRAM very close to the compute core. So it's...
Yeah. Yeah. Yeah, it's almost like processing in memory, if you will.
Yeah. So, yeah, that's basically the idea. You give it these little cores have 50% silicon dedicated to SRAM, 50% to compute. Have a lot of them, stitch them together, route around the broken ones, and you have a working wafer that works at enormous SRAM speeds. And it has 44GB of capacity. Now, we have to talk about a few things. 44GB is not nearly enough to hold any kind of thing. Okay. So that brings up some concerns. The second thing is, how are you going to deliver power to this thing? This is a big, big question because you essentially have a rack's worth of chips on a single wafer. Seriously, that's what it is. When you put like, you know, let's just think about it as 84 reticles. 84 reticles could be 84 GPUs, roughly speaking. That's a lot of chips. Like the NVL72 only has 72 chips. So you're talking about 84 chips that is a rack-scale GPU count essentially in a single wafer. Now you can imagine that this requires some serious power delivery techniques and some serious thermal issues that have to be dealt with. Right? So these are the basic things that we have to address in some detail. I recommend that we get to the SRAM question and the limited bandwidth slightly later. The reason is that there's a lot of stuff to talk about, but I'll just mention briefly what the power delivery looks like. Each wafer consumes about 23 kW of power. It's like enormous. Like if you think about a 1V supply that is feeding these GPUs, you're talking about something in the tens of thousands of amps of current that have to flow into a single wafer. So there is no way that you can supply the wafer with a power connector on one side and expect all this current to flow to the other side of the wafer. I point that way, but it's really not that big. A wafer is a 12-inch wafer. So it's a foot, like, you know, that's about that big. So, but it's still a lot. You'll drop a lot of power from one end of the wafer to the other end. So the way they do this is that they have these specialized vertical power delivery connectors that come in across hundreds of points on the wafer and deliver power directly in a vertical way. That's the only way you can deliver power to all these chips. So that whole thing was like completely unique to how Cerebras built its architecture. The second thing is cooling. They have this entire like crazy cooling system that has, you know, vertical flow of like microfluidic channels in the thing, and you have to cool the whole wafer at once because remember a wafer can have hot spots. You know, you've got hot spots there, you have to cool all this. So the cooling is done with what they call the engine block, and it's just this honking piece of metal that has this complex construction. If you go into the Cerebras website, you'll see it. You know, it's amazing. The whole cooling problem is also amazing. Like you have to cool a rack's worth of wafers in a small space, right? That's insane. The one other thermal aspect of this is that the wafer actually expands too. So not only do you have all these problems, the wafer expands. So their connectors are also custom-designed and patented by Cerebras themselves. They have this unique material that goes there and controls the coefficients of thermal expansion in a way that things match with the board and the connectors and the PCBs and the power delivery. All of this has to match. So they have like a whole patent that Cerebras actually owns on this to just deliver power and cool it. So that's my spiel on like how complicated it is to develop a wafer-scale engine.
Okay. Wow. Yes. Okay. So fascinating. So, first of all, you said it's not like you can just put some power pins on one side and route the current all the way through, but they actually have to have a grid come in from the top or maybe it's from the bottom and deliver the power to all the little places individually. And then on top of that, getting the... so that's, you know, getting power in is complicated, but also getting heat out is complicated. So, it sounds like they have to engineer some big crazy engine block, which yes, we should all go look at a picture to figure out per dinner plate how to get all the power in, how to get all the heat out. Now, tell us very quickly when you say that the wafer expands and you're talking about thermal coefficient, like tell people what that means.
Whenever stuff heats up, it expands, right? Essentially, the wafer also expands. I was looking at some numbers. It expands by about a tenth of a millimeter. And that's a problem. You know, alignment goes out of whack. And not only that, when you have attached it to a printed circuit board on the other side, and you're delivering power through this printed circuit board, there are different coefficients of expansion on the printed circuit board versus the silicon wafer. They don't have to expand at the same rate. So what is a tenth of a millimeter here and the silicon might be a hundredth of a millimeter on the printed circuit board? Like now you've seriously got connectors and stuff that are not going to stay connected anymore. They're going to rip off, you know. So that is a problem to solve when you're putting that much power and that much current through a single wafer. So they have some unique solutions towards this. I'll just leave it at that. Yeah, they had to solve this all by themselves. This is not a unique industry problem. This is a unique problem to them.
Amazing. So, quick question, just riffing here. The idea of having a wafer instead of dicing it all up and then packaging and connecting everything. At first, it sounds like, oh, it's actually going to be a lot cheaper and then you can yield harvest correctly and everything because you're not processing one, you know, $20,000, $30,000 wafer, chopping it all up, packaging it, interconnecting it. But on the other hand, you're talking about these, there are different trade-offs like, okay, great, you've got the big wafer and it's full of all these 900,000 harvested processing elements. But it's really complicated from a mechanical and thermal and power delivery perspective. Do you know like how does that impact the cost? Like this big engine block thing must not be cheap. So is it kind of like you're not... it's not like a cost advantage one way or the other.
I don't think this whole thing is about cost at all because all those processes exist. Yes. But ultimately the engineering, and if you think about the non-recurring engineering, NRE costs to develop something like this, including by the way, we didn't mention just stitching different reticles together is not easy because it requires patterning slightly differently because the mask when you pattern a reticle only has a shot that is a certain size. That's why chips are a certain size because the shadow it kind of casts on the wafer is a certain size. So if you have to connect chips up together, that itself is a manufacturing complexity that they have worked out with TSMC over the last decade. Right? So this is right from the get-go a non-standard wafer fabrication procedure all the way through the power delivery, cooling, expansion, mechanical problems, everything, and is challenging. This is a very hard problem Cerebras has solved. It is, you know, kudos to them, and we'll get into, you know, how people have failed at doing this too, and it's amazing that we now have a wafer-scale engine from a company that's working and has gone public. This is history in itself, really.
Sure. Totally. Okay. So it's definitely not about cost. So, when they started down this path at the get-go, they said, hey, we're going to have to make a ton of technical innovation to make a wafer-scale engine work from manufacturing, power, cooling, all the things. But it's going to be worth it because it gives us a ton of compute and it gives us a ton of on-wafer memory. And probably at the time the company started, which was pre-ChatGPT, 44 gigs of SRAM probably seemed like plenty. But let's dive into this. We're in the LLM era where even a small to medium-sized model is more than 44 gigs, right? Like Llama 70B already is going to be bigger than that depending on how you quantize it, but you need enough storage for activations and KV cache as well. So let's talk about like what happens even with inference when you can't fit all the weights and all the KV cache on one wafer.
Yeah, that's the whole problem, right? If you can put a model that's within 44GB and it runs SRAM-based inference off of a single wafer, you get extraordinary speeds. I mean, you get tokens per second that you can't dream of with GPU-based systems. And you can even compare it to LPU. LPUs don't have as much as you're not talking about a wafer-level system, and a single LPU has a few hundred megabytes of SRAM, not 44 GB, and you have to hook up a lot of them together to fit a model, and then you have networking overhead, all that concern exists. But now, as long as you can fit a model in 44GB, it's great. But you know, modern frontier models are actually much, much bigger than that. So then the question becomes, how do you do it? Now you have to put multiple wafers in a rack and you have to split it up. You have to split up the model between these wafers. And remember that as much as power was delivered on the whole face of the wafer, the networking doesn't work that way. Networking still leaves through one end of the chip and is like significantly slower compared to the on-wafer bandwidth of data movement, which is a big bottleneck. Right? At the moment you have to go off-wafer, you're in a bottleneck. That's the problem.
Gotcha. Yes. So, as long as... So, you're saying if we have a small enough model that can all fit on the wafer, then you can unlock crazy tokens per second that just aren't even reachable with GPUs and maybe not even with Groq's LPU because they only have, like you said, 170 megabytes or something very small of SRAM. And so that makes me think, okay, there's got to be use cases where it's a small model and you want it to run crazy fast, crazy high throughput. You know, so I think of like Google rewriting ads on the fly where they probably only need a small model and they just need to know like, hey, this is Austin and here's a tiny bit of context about him. So when you advertise, rewrite the ad in almost real time so it still comes back really quickly. But you're saying when you actually want like useful enough language models that don't fit on one wafer, getting information off the chip, which by the way, the chip is a big square essentially of a ton of tiny little chips. Getting information, you still have to get it off the edge of the WSE somehow and over some network communication and into another wafer. Have they said much? Do you know much about how this sort of like scale-up interconnect works at all?
I don't know much about the interconnect, but one of the ways you can deal with it is that you do parallelism, which means that you can do various kinds of parallelism. So what that means is basically you break up the whole problem into what is called pipeline parallelism, one option, which means that wafer one handles, you know, a few attention layers, wafer two handles a few attention whatever, then wafer three handles a feed-forward network, and then, you know, you pipe it through this. But then it's not that straightforward because data has to flow through the pipe between various parts of the inference process. Then you've got tensor parallelism where you say, okay, look, we'll break up the matrix into five parts and we'll run it across five different wafers. Tensor parallelism. And finally you've got expert parallelism, which is, oh, we've got like, we'll run this expert on this wafer and this expert on this wafer. The ultimate benefit of running a wafer-level system is diminished by the fact that you have to somehow break it up into wafers and that you don't have communication bandwidth between them.
That's like the fundamental downside as to why it goes against the very basic concept of wafer-scale engines.
Okay. Got it. So ultimately the wafer-scale engine is best when everything fits on the wafer. And you're pointing out that well if we break up the problem, there's ways to break up the problem into sort of sub-problems that can fit within the wafer, but at the end of the day they still have to communicate with each other. But maybe you can overlap some of the communication and computation by using parallelism to keep things tightly, you know, into the wafer. But at the end of the day that is going to be a bottleneck is this off-chip, off-wafer I/O essentially, which by the way I think I saw SemiAnalysis had this great article that came out yesterday. I tried to skim as much as I could. You know, these are like PhD theses that take you like a month to read because they're so long and have a nice big team. But I believe I saw somewhere in there that Cerebras was experimenting with like wafer-scale photonic interconnects to go like, oh, it's hard, like there's not a ton of bandwidth between every core and off-chip. But what if we put another wafer on top and that allowed you to route information essentially in the Z direction or almost like 2.5D? Like you could go up to this photonic wafer, connect wherever you need, and then come back down. I didn't read too much about it, but I thought interesting. It also sounds complicated and challenging to take two wafers and connect them like that.
It's already hard enough what they did. Now you want to put like, there was also talk of like stacking SRAM wafers on top of this or DRAM wafers and bonding it.
Do you not want... do you want... do you not... have you not solved a hard enough problem already? You want to make it harder.
Totally. Well, okay. So, and that's an interesting point which is like, okay, if you're looking at all these other AI accelerator startups, they're talking about memory hierarchies to say like, like MatX, we, you know, we talked with Rainer Pope and he was talking about like, oh, can you put weights in SRAM and KV cache in HBM, and other people, Qualcomm, Intel, maybe D-Matrix, others have talked about using DDR as like another tier of memory storage. And so the question is could Cerebras's SRAM-only wafer also use DRAM in a way that's not just like pipe it off this slow pipe that they have over to some rack full of DRAM or something. And so then, so you know, people postulate, oh well what if you took a wafer of DRAM and just attached it to the compute SRAM wafer. But also my head is like, well Vik just told me that's how power is getting in and that's how cooling is getting in. So if you're going to start slapping other wafers and you got a wafer sandwich, like how do you cool it all, how do you power it all? Sounds complicated.
Yeah. Yeah. I'm thinking of a multi-stacked wafer, you know, Cerebras wafer-scale engine photonic...
Layer memory wafer, and then next, Cerebras wafer stack, and then like, wait, how do you power all this stuff? I don't know, man.
Throw some high bandwidth flash in there. You know, why not?
Why not? You know, got to make the problem harder. This is not hard enough.
Yeah, right. I mean, we want jobs for our kids, right? Like, they need to work on challenging things.
Yeah, yeah, yeah, yeah, yeah. We should talk about basically what this means for their business and who's going to use this stuff, really, and why it's useful or whether it's useful at all. I think we should talk about the use cases, not the deep silicon tech. Yeah.
Yeah, all of that. But, but one maybe as a transition, this is very technically hard, and you mentioned earlier someone has tried this before this way for scale. Tell, tell us quick.
Yes, yes. I mean, this is a good story. I think this is a good transition to going to talking about business. Okay. Because just when we said that, oh, this is like super hard, and you know, Cerebras has done great to solve all these technical challenges. Oh, by the way, here's some more you can solve or whatever we're doing over here. This has been tried before. And this is the story of Trilogy Systems. In the 1980s, Gene Amdahl raised about $230 million, which is about close to a billion dollars today, and to build basically the same idea that Cerebras has achieved today, the wafer scale engine, except that at the time it was a 2.5-inch wafer. Okay, it's a small wafer, not like this giant dinner plate. It was a small, you know, saucer wafer. But then the ambition was amazing. It was like way ahead of its time because that was a time in the 1980s that Gene Amdahl said, okay, look, I'm going to make like a wafer scale chip. Why should I cut it up? I want to do it. And their idea was the same. They're a bunch of smart guys, so they said, you know, we'll just route around defects like Cerebras does today, and we'll make it work. But anyway, the story goes really like crazy, but they really didn't succeed, okay, because the yields in the 1980s for even a 2.5-inch wafer were just too low. Manufacturing processes were not at the sophistication that they are today. We have figured out a lot of things in wafer manufacturing today that actually makes wafer scale engines possible. But then it got crazier, right? The story of Trilogy Systems got crazier. So what happens in like early 1980s, I think 1982, is that there are these storms that flood the factory. So it's a $33 million factory, and then like water starts seeping into the air conditioning and all of this stuff. And what happens is the pipes start to rust, and it starts to blow microscopic dust into the clean room. And nobody knew what was happening to the yields. Like, they didn't know that this is due to water seepage and rust that now the clean room is getting sprayed with dust, and all the wafers are dying because of this storm that came through. So they took months to figure this out, and they blew through capital because it was blowing dust into the clean room. And at the end of the day, they're running out of money. So they wanted to complete this wafer scale engine. So they like, okay, let's make a Hail Mary IPO. In 1983, they went and said, we are going to go IPO, and they raised like $60 million without a product in hand. They didn't have a product, okay. And they used all that money to still try to make a working wafer, and just nothing happened. And then ultimately the public market lost it. I'm like, okay, that's it. You guys are not going to do any of this stuff. And so the stock at the time, which was like at $12 a share, like eventually plummeted to near zero in the next couple of years that followed. It was like totally terrible.
