Cerebras CEO: Why GPUs Can't Do Fast Inference

This is the largest chip built in the history of the computer industry. It's 58 times larger than a GPU. And for AI, bigger chips process information more quickly and therefore you get answers in less time. For AI world, big chips are undoubtedly the best way to go. There's no mode in inference. It takes you eight keystrokes to move from a GPU to us in the cloud. We solved a problem that nobody in the history of computed solved. And we delivered it in 2020 and nobody cared. Nobody cared. Nobody bought it and nobody cared. Everybody said we were crazy. It would never work. So then we built the next one.

Yeah. What happened I think was for a long time AI was sort of a novelty, right? It was like a parlor trick. It was cool but not useful. And what happened somewhere around the middle of 2025 was the AI got smart enough such that people began to use it and we remember we make AI with training but we use it with inference and suddenly people wanted to use it and the minute you want to use it the minute it's productive right speed matters right fast tokens are more productive and so the conversation moved from everything else to How do we make our inference faster? How do we deliver tokens more quickly because those are more productive tokens? We get more done in less time and therefore it's more valuable.

So the right metric is tokens per second per user. That's how fast you get the first token all the way through the last token in your response. And it's true for queries from the chat, but it's also true for agentic loads, right? If they're sort of multicycle turns, waiting is amplified. And so what you want is blisteringly fast responses so that the AI feels like it's in real time. You can engage with this.

I think that's right. And I think that's a very good analogy. I think if you think of something like Netflix, right, when the internet was slow, right, Netflix delivered DVDs and envelopes. You would get a DVD in an envelope and when the internet became fast, they didn't get more efficient at delivering DVDs and envelopes. They became a movie studio, right? The speed enabled them to become something completely different. And that's what speed does in general and in particular for AI. It opens up a whole new domain. It allows you to use the AI differently. You will stay longer. You will come more often and you work on harder problems.

That's great. I mean, how big is the market for slow search? How big is the market for dialup is zero. How big is how long will you wait for a website to resolve? Will you wait 8 seconds? Nobody waits. See? And so it's the exact same with AI.

That's right. Well, it's running and running and running. I think that is not what people want.

Well, I think there used to be one chip to do it all was called the CPU, right? And there emerged a co-processor to do discrete graphics and as the AI workload became interesting. We focus chips more and more on that particular workload and today several companies make traditional GPUs. So Nvidia and AMD make very standard GPUs. There are a group of companies who the hyperscalers make some of their own parts. So the TPU...

TPU is Google and Tranium is AWS. Yep. And then there were group like Cerebras and we were among the pioneers to build a part from scratch optimized for AI and nothing else and that we weren't inside of a hyperscaler. We weren't optimized for a hyperscaler's problem or for one lab's problem. We were building a chip for a collection of AI problems and all our thinking was around how to accelerate AI.

Yeah.

And to the general chip versus ASIC chip it was actually interesting because you guys and your CTO celebrated when Grock was acquired. Was that what was that? Was it a recognition by Nvidia that your vision was right all along? Yeah, I think one of the things Nvidia's sort of most durable moat was the perception that the GPU could do everything and it was all you needed for AI. And the acquisition of Grock for $20 billion and the structure and the speed with which they chose to do it made clear to everyone that that wasn't true. that the GPU architecture couldn't do fast inference and that this market was large and growing quickly and we were the fastest at it and the largest and you know our sales were more than 10 times the Grock and they paid 20 billion for the number two player.

That was a good day. Right. And the recent announcement of Halino between OpenAI which is a very large customer of yours and Broadcom. Where does it fit in that picture? Is that one of those highly specialized ASICs for inference?

That's right. Halino is a part that has been a long time coming. It was announced with Broadcom before OpenAI did the deal with us. Remember we did a huge deal. This is probably the largest deals in Silicon Valley history north of 20 billion dollars. I think they have a yawning need for silicon and one of the things that OpenAI has been I think the best in the industry at is looking at an exponential curve adoption rate of growth and not being afraid of what it says. Right? Others have had to go out and strike really bad deals to get capacity where OpenAI saw this coming. They struck big deals for memory, for compute, with us, with others. They've really been sort of visionary in understanding what it means to extrapolate from an exponential curve and that curve is AI usage, right? It is growing so unbelievably fast. The whole industry is chasing demand. Faith, right? We usually well often it's the other way. Often people are building it hoping it will come and in our case all of us are chasing what people already want to do, let alone what they might do in the future.

Yeah, multi silicon environment is a healthy ecosystem, right? I don't think anybody would say that the x86 environment was a healthy place. There were for 20 years there were two players and if you notice when a new workload came around cell phone workload it was very similar but required low power and battery and both lost right Intel got zero share AMD got zero share and a company nobody previously heard of became the largest seller of compute in the world and that's ARM and that healthy ecosystems have lots of different ways to solve problems.

No, no, that divide is real. I think they are an industrial adversary that I think they have made really interesting investments that made investments in power in their grid which is a real weakness in the US. If you're in France you have nuclear which sort of turns out to be pretty cheap power and pretty clean. But in China they made huge investments in their grid. they are behind in chips but their approach was at the next level as open-source models where they producing some extraordinary models not as good as GPT or Anthropic or Google's Gemini but very good and I think they don't have the same chip strikes that the US does but they've got leadership in other debates potentially in power which is what we need for data centers.

We don't sell to China for geopolitical reasons, for business reasons, for regulatory reasons and for some geopolitical reasons.

Yeah, I think if you look at the way the ecosystem for apps and cloud emerged everything you can do you should do on your phone or your laptop. But the ability to get real processing power to a phone or to a laptop is constrained because they're generally working off a battery. And so you want to do as much as you can as close to the data as you can. But the truth is in most situations for real compute. And that's exactly the way it's going to be with AI. We're going to do a lot of work on the cell phones on the laptop but for the big work you're going to go to the data center and that's where our focus is. Our focus is data center compute.

I think that if you spend a lot of time staring at the public markets and watching their ups and downs every day you're watching the wrong thing. I think that right the wisdom received from the great thinkers of public market investing is that in the short term the public markets are voting mechanisms who's most popular but in the long term they're weighing mechanisms who's created the most value the most weight and so what we can do as a newly public company is we can not look at the day-to-day fluctuations and we can focus on just building extraordinary technology, winning new customers, making our existing customers happy, ensuring they buy more, being better every week, and so I think the day-to-day fluctuations are out of our hands. And that's about the voting.

That's not exactly our experience. I mean obviously we have enormous demand from OpenAI but we have huge you know dozens of other customers who are trying to place in very big orders and historically bubbles were when supply got out ahead of demand right when in the 90s we built out telco infrastructure right we built out fiber years before it was going to be used and it took six or eight years and it all got used but it was a sort of if you build it they will come mentality whereas what's different about AI right now is we're all trying to catch up we're trying to build data centers faster we're trying to increase our demand our supply chains for demand that's already here today and only a very small portion of the world are using AI anywhere close to its potential and we're already sort of overwhelmed with compute. We're overwhelmed with the demand for memory which is a real weakness and the GPUs is not a problem we face and the ecosystem they're constraints left and right and that doesn't feel like a bubble.

So there are three major bottlenecks right now. The first is all GPUs and all ASICs except us use a type of memory called DRAM and a particular flavor of that memory called HBM and that memory is made by three companies in the world, Samsung and SK Hynix. Three companies and they're sold out. That's the number one problem. We don't use it.

Build more factories and so you have this huge problem for GPUs and even now for CPUs and you can't get this memory because of our architectural choices. We don't use it. The second constraint was a process at TSMC process was called CoWoS and this used silicon as a motherboard and this is what Nvidia and AMD and others use as a motherboard. So they put their chips and the memory chips on it and it's more efficient than a traditional motherboard. That process is sold out is highly constrained. We don't use it. Third limitation in the space is 3 nanometer factory space at TSMC. So this is the most aggressive factory. So with the most advanced technology, our chips are at 5 nanometer. So we don't use the 3 nanometer technology. And so...

That's right. They build factories and the factory etches transistors that are a certain distance apart. And the smaller the distance apart, the more transistors per square millimeter. And the more you can do per square millimeter. And so the history of the computer industry making chips has been we've gotten better and better at putting transistors closer and closer.

And so we used to be at 16 nanometers apart and then we went to 10 at to seven and to five at three and they're working on two 1.8. And so right now the bulk of the GPU world is all three and there's tremendous congestion at the factory.

Tooling at the factory and we use the 5 nanometer factory and this is our chip here.

This is the largest chip built in the history of the computer industry. It is 58 times larger than a GPU. It has, you know, two and a half or three thousand times more memory bandwidth. And for AI, bigger chips process information more quickly and therefore you get answers in less time. And obviously you don't want a bigger chip for a laptop or a cell phone or for lots of other things, but for AI world, big chips are undoubtedly the best way to go.

Including in a system like ours.

It means that AI when we do AI the agentic they the AI processor is like the brains and the CPUs are like the body. They're taking action. They're doing things in the digital world under the direction of the AI which is running on the accelerator on the server system. And so as we do more and more AI work and more and more agentic work, we're making more and more calls to CPUs and therefore the demand for CPU is just through the roof.

That's right. Okay. Yeah. Let's go back to the product in a second, but let's talk about you and your journey a little bit for people to have context. So, you know, obviously you pulled and hold of an incredible IPO with perfect timing. And I know it was a little bit...

That's the way you have perfect and very much to this point. So like you guys started building this you know bigger chip focused on inference in you know over a decade ago.

2016 which you know makes perfect sense in retrospect given how long it takes to build a technology like this but like what was the vision right 2016 I guess it was like four years after ImageNet deep learning was a thing.

It was all revolutions and that's why we don't believe the right thing to do is to embed the latest and coolest model into your circuitry. That's a mistake. You know, we're the fastest in the world of transformers and our architecture was set before transformers existed, right? We're the fastest in the world of diffusion before and our architecture was set before diffusion. What you want to do is get the underpinnings of those so that when the market moves you can be good at that as well otherwise...

To the ASIC question earlier so that's what others do they build the architecture of the model into the...

It's been a structural mistake so you're doing the brand new horizontal platform on any kind of model.

Yeah. And if you can accelerate that whatever the model makers invent you can make faster. Okay. And that was our approach.

I think the lessons are large and many. I think what one of the things you know I guess experience is another name for having made mistakes and learning from them right I think you know we have as a team built dozens of chips over the past 25 years and the returns to experience in chipmaking is enormous and we built a different type of computer at SeaMicro a type of computer optimized for low power and optimized for a workload that was very different than AI for something like web browsing and but the fundamental underpinnings the questions you ask as a computer architect are always the same what can I do to make this work faster right and is there enough of it to make it worthwhile these are the two questions we ask.

And you know the flip side of that was I mean wouldn't it be a surprise if the GPU which had been optimized for graphics for 20 years and been pushing pixels to a monitor was suddenly good at a new world. it'll that be serendipitous and we came to believe that that it wasn't the right architecture for it. It was just better than the CPU and that we could build an architecture that would be vastly faster. It would use less power and could drive down the cost of it.

Yeah.

So for us we at the beginning we were honest with our VCs and told them we're going to attack a really hard problem. We weren't going to build something that was a little bit better than a GPU. And that our idea, our strategy was that you will never beat a great company like Nvidia by doing something a little bit better than they're going to buy everything for less. They're going to have pricing pressure. They're going to be able to bundle. And the right strategy would be to do something incredibly hard in engineering that was way better. 10, 15, 20, 30, 50 times better. But to do that, the ordinary and obvious paths are all closed. Everybody else has taken the bar. And so what we observed was that speed in inference was going to be a function of memory and that there were two types of memory. There's this DRAM of HBM and they can store a lot but there's four. There's another type of memory called SRAM that is unbelievably fast but per unit area can't store very much and that for graphics everybody had always used DRAM they used HBM and that the AI workload was fundamentally different in graphics you move data to the GPU and then you work on it for a long time and then you send the result. So the time spent in total of movement plus work is dominated by work. In inference and AI, it's the exact opposite. You move a huge amount of data, all the weights from memory to compute and you do one calculation to generate the next word and then you have to do it again. So all the time is dominated by the movement of data. So that's why GPUs have so much trouble being fast. Okay. So we observed that if we chose a strategy using SRAM, we could be faster. But then we have to overcome the tradeoff with SRAM. It can't store very much. That led us to the solution. If we could build a part vastly larger than any part in history, right? at the size of a dinner plate. We could stuff it to the gills with SRAM and it and thereby get the benefit of SRAM that it was fast and overcome the weakness that it can't store very much. And that led us to a strategy called wafer scale. This chip is made from a single wafer that comes out of TSMC.

A wafer, all chips are cut from a wafer. A wafer is a circular piece of silicon that's 300 mm across and the process of chip making stamps out like your mother does with a cookie cutter. Stamps out chips. The biggest chip that had ever been built before us was 800 square millimeters. 840 to be exact. And this is 46,000. So we had to invent all sorts of new technology to build a chip this big. And once you build it, you have to invent ways to power it, to cool it. There are no vendors waiting for you. Yeah. Right. Because you look like nothing else ever made. And that took years. And it was a deep tech problem. It had never been solved before. And we had an 18-month period where we were spending 8 million a month and we couldn't build them. So for your deep tech founders, you have...

Yeah. And why so much? What was the cost just to understand how these businesses? Because what everybody thought was hard, we solved quickly. And what nobody else knew about because they'd never actually done it turned out to be really hard. You know, imagine I tell people, imagine the first group that was going to climb Everest and they're at base camp and they're having tea with a group that just failed. And the group that just failed says, 'Halfway up, there's this part. It's unbelievably hard. We couldn't do it.' Okay. Your team climbs up, makes it all the way to the top, comes back, they're having tea again, and the team that made it leans over the team that hadn't made it. Said that part in the middle, that wasn't the hard part because nobody had gotten past it. Nobody had gotten past certain things. So, they didn't even know what to be afraid of. We now know. And it was something a step called packaging. And that's how you fix a wafer to a motherboard, how you deliver power to it, and how you cool it. And nobody had done it before. And over that 18-month period, we became the best in the world at it from approximately zero, right? And we did that by failing again and again and using good engineering methodology and doing a failure analysis every single failure. So we failed differently again and again and again. And we told our board, you know, we met with our board every six weeks and yeah, this is the strategy. Nobody's ever done this before. This is what we're going to do. And we had to invent new materials. We had to invent new...

We ended up building things that everybody else had partners who could do. But in August of 2019, we announced we had solved it. And my co-founders and I, the first time it worked, it was sitting in a tiny little lab and we couldn't believe it. We were the first few things that treated me to make one work. And we just stared at, you know, watching a server run is about as exciting as watching paint dry. And there we were, the five of us just staring at this machine, not believing it might have worked. We made it work.