Andrew Feldman, Cerebras Co-Founder and CEO: The AI Chip Wars & The Plan to Break Nvidia's Dominance

Our AI algorithms today are not particularly efficient. In a GPU, most of the time it's doing inference, it's 5 or 7% utilized. That means it's 95 or 93% wasted. We won't be as dependent on Transformers in 3 years or 5 years as we are now, 100%. The fundamental architecture of the GPU with off-chip memory is not great for inference. Now, they will continue to do well in inference, but it can be beaten, and I think they know it.

Well, Harry, thank you for having me. I appreciate it.

We saw the rise of a new workload, and this is every computer architect's dream. We saw a new problem to solve, and what that means is maybe you can build a new machine better suited to that problem. And so in 2015, and the credit goes to Gary and Shan and JP and Michael, my co-founders, they saw on the horizon the rise of AI and that what that meant was there'd be a new problem for computers, that what the AI software would ask from the underlying chip, the processor, would be different. And we came to believe that we could build a better machine for that problem. That's what we saw. You know, obviously we didn't see it exactly right. I underestimated it. You know, this is my fifth startup and the first time I underestimated the size of the market by a lot. But what we did get right was that this was going to be big and it would put a different type of pressure on a processor and that it would put pressure on the memory bandwidth, that it would put pressure on the communication structure. So that's what we saw. We dove in. It's been an extraordinary 9 years.

The way to think about a chip is it does two things: it does calculations and it moves data. Right, this is what a chip does, and sometimes along the way it stores data. And so what AI presented was a very unusual combination of challenges. First, the underlying calculation is trivial. It's a matrix multiplication, and an FMA can be developed by any second-year electrical engineering student. So you say to yourself, holy cow, this has a huge number of very, very simple calculations. The hard part with AI work is results and intermediate results have to be moved a lot, and therein is the most complicated part. They have to be moved to memory and from memory, and they have to be broken up and moved among GPUs. And what we saw was that this was going to be the hard problem, and that if we could solve for that problem, we would build an AI computer that was faster and used less power.

The first step in computer architecture is deciding what you're not going to do, right? That's what are we not going to be good at is really the first important question. I think to answer your question, you say is the computational work for training from scratch different from fine-tuning? And the answer is it's not different. It's approximately the same. Now, inference and training have some different requirements, and generative inference in particular has some very challenging requirements on exactly the communication dimension that I mentioned. In generative inference, you have to move all the weights from memory to compute to generate a single word, and you have to move them again to generate the next word, and again and again. So if you have a 70 billion parameter model, not a giant model, and each weight is 16 bits, you're moving what, 140 gigabytes of data to generate one word. This is an enormous amount of data movement across memory, and that's called what, what consumes that needs is memory bandwidth. And if you have an architecture like we saw in the GPU, that is your fundamental limitation. It's a fundamental architectural limitation, and that was what we went to wafer scale to solve.

They use a memory called HBM, a type of DRAM, and it is phenomenal memory, but it's slow. It is slow and high capacity. And when they set the architecture for graphics, that's what you wanted. You didn't have to go back and forth to memory very often. SRAM, on the other hand, is unbelievably fast but has low capacity. And so we wanted to use SRAM, but if you build a normal-sized chip, you can't hold a model. And so by going to wafer scale, we were able to put down a huge amount of SRAM and get the benefits of speed and enough capacity. If you build a normal-sized chip with SRAM and you want to do a 400 billion parameter model in inference, you might need 4,000 chips, or if you want to do a DeepSeek 671, you might need 6,000 or 8,000 chips. What an administrative nightmare. I'm sorry, you can keep it on as much as you can on one wafer or two wafers or four or 10. You get all the benefit of the SRAM, and because you've been able to use the wafer, you get this tremendous capacity as well.

Look, there wasn't really, unless you went to wafer scale, there wasn't really a credible other choice. This is the way GPUs had always been made. It's called a graphics processing unit, right? That's the way they were built, and it was part of their advantage against a CPU was they were built this way. But now they're dedicated chips like ours, and what used to be their advantage is now weakness. And that's a fun market to be in when over a very short period of time what you're good at becomes your weakness.

They do know this. There's not a lot of choice. Well, I mean, they don't make memory, so they're a consumer of other people's memory, right? And that's SK, right, the Hynix guys, or Samsung, I mean, Micron. There are only three or four or five companies that make huge amounts of memory. Not many choices. But it's part of a complex architectural tradeoff. You know, the flip side, you could say it's worked really well for them, right? Look at where it's taking them. But in comparison to those of us who wafer scale, it's a small set. It's a set of one, us, right? We have real advantage against them on inference.

In our business, there are a lot of ways to skin a cat. And our way is different than NVIDIA's way, it's different than the TPU, it's different from Trainium, they're different. Right now, and every day since August 26th when we launched inference, our way has been the fastest way across a whole set of models tested by Artificial Analysis and others.

This isn't our first dance. We've been building computers for a long time. And when you make a choice like wafer scale, you have to weigh the tradeoffs. We use less power. We use less power because one of the most power-hungry things on a chip are the I/Os, right? Are moving data off-chip. And so if you are moving data off-chip frequently, you're using more power than if you can keep it in the silicon domain on-chip. So we knew we would use less power. We knew if you went to wafer scale that you had to solve some problems that people said were impossible to solve, like yield. So we had to invent techniques that allowed us to yield wafers. In fact, we invented techniques that allow us to yield as well or better than others who are building much smaller chips.

Okay, a wafer begins, it's a 12-inch diameter circle, a slice of silicon, and your chip is punched out of this, the way your mother might take a cookie cutter and cut out cookie dough. And during the process, at some point, just like your mom might have done, she lifts up the edges and all the little bits are removed, and what's left are just the cookies. Those are your chips. Now, what happens is there are a set of naturally occurring flaws, and that's like your mother closing her eyes and throwing up a handful of M&Ms. Now, the bigger the cookie, right, the higher probability you hit an M&M. The bigger the chip, the higher the possibility that you have a flaw. And traditionally, what you did when you had a flaw was you threw away the chip, or you sold it as a less valuable part. You shut down part of the chip and sold it as a less valuable part, something called binning. So every wafer is going to have flaws. The bigger your chip, the higher the probability you hit a flaw, and the more part of silicon is wasted when you throw it away. This is what everybody thought was known truth. And one of the things our team realized was that there are other ways to handle flaws. Like what if instead you built your computer, you built your processor out of hundreds of thousands of identical tiles, and say there was a flaw, say you just shut down that tile and worked around it. Say you had a row or a column of redundant tiles that when you needed them you could just pull in. Now, that had been traditionally the technique used in memory making, and the memory yields are extraordinary. And so it occurred to us that if we could build a computer, build a processor built of hundreds of thousands of identical tiles, we could use redundancy such that when there was a flaw we could just leave it there, shut it down, work around it, and pull in one of the redundant tiles. And that had never been done in a computer before, and that's at the heart of our architecture that allowed us to yield and deliver whole wafers. Nobody ever been able to do that in the 70-year history of our industry. Really smart people struggled. I mean, Gene Amdahl, one of the fathers of our industry, had a company called Trilogy that crashed and burned trying to do this. And we figured it out.

I think it varies. I think, look, if you go to get a cancer diagnosis on, God forbid, your mother or your wife, I think 93% accuracy is just plain not as good as 94% accuracy, and you pay a lot and wait another week to understand what the accuracy is, right? You pay a lot. Right now, on the other hand, if you want Llama 405B to generate data to help you tune Llama 70B, maybe you can wait a few days, three days, a week more. You don't, there's no urgency there. On the other hand, if you want an answer from Perplexity, right, you don't want to wait 45 seconds for a search answer, right? You don't want to wait in a chat, you don't want to wait 3 minutes for R1 on GPUs to give you an answer. What we know is that in interactive mode, milliseconds matter. In interactive mode, what Jeff Hinton over Google years ago showed was that you can destroy your user's attention with milliseconds of delay. So being the fastest matters, everything in that domain. So I think what you have to do is sort of be thoughtful and say in some cases being the fastest doesn't matter. We'll call those batch lots of maybe cheapest matters there. In other domains, there is no search if you got to wait eight minutes to get an answer, right? That's not a product. When you go fast, a whole set of new opportunities open up. I mean, right, Netflix used to mail DVDs, right? That's what happened when the internet was slow, they'd mail DVDs.

If you remember Blockbuster, right? First, I mean, let's look at the history of that. You're exactly right. First we used to drive to Blockbuster to get a DVD, right, or a video. Then Netflix was mailing them to us, right? And then we got broadband, and suddenly Amazon's a studio, right? It changed everything. And speed in inference does the same thing.

It begins with the following: training makes AI. That's how we make AI, and inference is how we use or consume AI. And so understanding how big the inference market is is understanding the number of people who are going to use it, how often they're going to use it, times how much compute each use takes. And right now we are in this rare time where the number of people using AI is growing, the frequency with which they use it is growing, and the amount of compute used in each instance of use is growing. That's why you're getting this extraordinary growth, and that's why it's off the charts right now.

What we made until the middle of 2024, right, what we made in AI was a novelty. It wasn't very useful. Late in 2024, what we made began to be useful.