Welcome back everyone to the Cube's live coverage here in Atlanta for Supercomputing '24, SC24, as it's called. I'm John Furrier, host of theCUBE, with my co-host, Dave Vellante, also the co-host of theCUBE Pod every Friday. Check it out. We're recording at this Wednesday because we're traveling. Our next guest, Andrew Feldman, entrepreneur, co-founder and CEO of Cerebras Systems. Andrew, welcome to theCUBE because we've been talking about for now three years about this whole systems culture, systems revolution. So you are not only living it, you're building it, you're rolling it out. You have the big chip.
Thanks for coming on theCUBE.
Thank you for having me. I really appreciate it.
So first of all, before we get started, show the device because I think this is super important. It's a chip. Explain what this is.
This is the largest chip in the history of the computer industry. So this is the chip that it is 46,000 square millimeters. As you guys can see, it's the size of a dinner plate. Traditionally, chips are the size of a postage stamp. A chip this big processes more information in less time using less power, and that's the goal. What we do is we don't sell the chip-
Yeah, sure. We don't sell the chip. We use it in a system we build to deliver extraordinary compute performance, both for training and for inference.
Awesome. And I bring this up because I want to set the table with the size of the chip first, because traditionally, smaller, faster, cheaper was the old way. Okay, now we've got bigger and better with these clustered systems. We've been writing about it on SiliconANGLE. Dave's heading up our research team, and I've wrote in research notes that it's the systems of servers together, Ethernet, networking components, interconnect. You're seeing bigger chips because they're optimizing...
Core performance that's needed. And GenAI is requiring massive resource. This is a fact, people now, it's no debate. That's just a fact. Okay, now the next level question is that what software runs on this? How does it change the existing configurations of the classic, I call, the Holy Trinity of the compute industry, storage, networking can compute? And then now you've got connected cloud, on-prem edge, full distributed computing environment at scale. This is the internet.
This is where we are. What's the significance of this? How does the chip size matter? What kind of software is coming? I know it's a big question, but inaudible.
Well, it is a big question. It's an everything question. I think obviously big chips aren't right for everything. They're wrong for your cell phone and they're probably wrong for your car. But in AI, we're not interested in the behavior of one chip. We're interested in the behavior of tens of thousands of chips and tens of thousands of little tiny chips creates a tremendous complexity of trying to tie them together again, all right. Remember when chips start, they start as a wafer. They're cut up into little pieces, they're put in different machines, and then we tie them back together again to get them to behave. We put Humpty Dumpty back together again. Our view is why are we cutting them up? What if we could yield the largest chip in history? What if we could keep more information on the chip? We would use less power, we would produce results in less time and we would make it vastly easier to program. And that was our vision eight and a half years ago at Cerebras and it's been a phenomenal run since. We have deployments around the world. It's been an extraordinary run.
I remember, I first heard of Cerebras in the news, but then Anastasi in Tech did a deep dive. She's amazing.
And she talked about the power efficiency, the performance, the number of transistors, the more efficient software capabilities that you have. And she also talked about the yields and you just mentioned yields. And I thought, oh, maybe the yields are going to be worse, but you had-
Better yields. And explain how you achieve that with such a large phone factor.
Sure. So many people told us this couldn't be done. And the first reason, and the world is filled with people who say, 'It'll never work, it can't be done.' We have no interest in doing business with them. Those aren't our people. Our people are the engineers who say, 'If he says it can't be done, I want to do it,' all right. Those are our people. The first thing, those with a mouth full of, it'll never work. What they said is you'll never yield. And that's because every wafer has some flaws that are inherent to it. And traditionally, as we put little chips on them and then we cut up the wafer into little die, we tested each die. And if they had a flaw, we throw it away. That's right. And everybody said, 'Well, you'll never have a full wafer this big that doesn't have a flaw. How are you going to yield a part?' And what we knew was that there were other ways to manage flaws. And in fact, in memory, they manage it very differently. In DRAM, they have almost perfect yields and they have almost perfect yield because they have a repeated tile design. They have hundreds of thousands of bit cells, each identical, and they have redundant rows and columns. And when they have a flaw, they map it out, use one of the redundant cells and keep going. And so we came up with the idea, mostly my co-founders, JP and Michael and Sean and Gary, they came up with the idea that if we built a repeated tile design with hundreds of thousands of identical tiles, if one was a flaw, we could have some redundant ones layered in and we could shut down the flaw and use the redundant tile. So the whole idea was built to withstand flaws, not need to eliminate them. We yielded it right away. It took us about 18 months, about $10 million to solve a problem everybody in the industry said could never be done.
I mean, I think about log-structured file back in the storage days. This is obviously in real-time.
That's exactly right. I think that was some of the magic of doing work that other people say can't be done.
Talk about some of the examples you guys have in production right now, because the research that we're seeing in the marketplace relative to your value proposition is that technical people love this. If you go to the MLOps folks, people who are grinding right now, then you got an onboarding wave of developers coming in.
You guys had great benchmarks we just covered it on SiliconANGLE this week news. Talk about the news or the performance gains, record numbers and talk about the efficacy of the benchmark. And if you can, comment on the, I won't say whitewashing the benchmarks, but I mean you could fudge the benchmarks if you throw more power at it. I mean anyone can get more anything, so talk about how should we evaluate benchmarks, talk about the news and the benchmarks.
So at the show we launched inference support for Llama 405B. This is the largest open source model. And we launched performance numbers of 969 tokens per second. To give you an idea, Azure, Nvidia running under Azure is at 13 inaudible tokens per second. All right? So we're more than 75 times faster than hyperscalers. We are offering Nvidia product. This is the fastest in the industry, bar none. And so we're really proud of that. And today, since we launched our inference services in August, we were the fastest and have been the fastest every single day at Llama 8B, Llama 70B, And as of yesterday, Llama 405B. We're so fast that it changes the way you can use Llama 405B, and you can use it to compete effectively with the largest closed source models. So that's the first answer to your question. We do both inference and training. We have partners around the world. Our largest strategic partner is a group called G42. And with them, we have deployments that have built super computers measured in exaflops. Built them in Santa Clara, California, built them in Dallas, Texas, and now we're building them in Minneapolis, Minnesota. We've trained leading models. Right now, the premier Arabic-English model LLM is a model that we worked with G42 together to train. And it's now being used by hundreds of millions of native Arabic speakers. We've trained models in Catalan, in Kazakh, in Hindi, all of which is in our training business. In our inference business, business is exploding. Remember, training makes AI and inference uses AI. And right now, people want to use AI like crazy. And what we're seeing is overwhelming demand for inference.
And the technical people like it. Talk about the inference as the killer rap because training is like going to school. I don't go back to fourth grade. I train and then maybe I reinforce my learning, but I ensure-
You're trying to solve for X.
I go to school, I get trained, and then I graduate and I infer in the real world. That's like AI brain and then I reinforce it. This is AI. AI is very brain-like. How do you guys do on inference and how do you sell to your customers? Because is it, I mean, I can see Amazon and Azure, I want to build my own system. So where are you guys targeting? I know you're going right after Nvidia. You guys are pretty clear on that. The big green machine, I call it. What is the killer app for inference and how does that compare?
Look, Nvidia is a great company and nobody's done better over the last 10 years. I think in 2014, they were worth 10 billion. And today they were, what? 3 trillion. I mean they've had an extraordinary run. But this is a very big market and there's room for a lot of winners. And we're going to do our best to put ourselves in front there, so we're one of those winners. I think high performance inference is a game changer. What OpenAI recently showed is that you can use performance inaudible to get better accuracy. What we all want is more accurate models. And what they've shown is through techniques like Agentic models and through techniques like chain of thought, you can use speed to ask the model to improve itself in a train of thought flow and the accuracy of the model improves. And when you're 10, 20, 50, 70 times faster than the competitors, you can use some of that time to improve the accuracy of the model. You can give the user a better answer and they won't even notice. And so what we've pioneered is the fastest inference, bar none.
So I want to come back to a couple of things. So it seems to me that, well, of course the economics of training LLMs are just horrendous, right?
It's an expensive process.
If you have to adhere to the scaling laws and you look at the price per token, it's just painful.
So you help with that problem?
We do help with that problem.
Okay. It's still a huge problem even after-
It's a huge problem. There are two parts of the problem. There's the capital that goes into buying the equipment, and then there's the operating expense, which is almost all power, right?
The power used at the data center. And we cost less to buy and we use less power to generate FLOPS. So we help on both of those tokens, but even then training giant models is expensive.
Okay. And then the other thing is I've talked to financial institutions that say they don't want to use a closed LLM, proprietary LLM. They don't want to use Llama because they're afraid of the fine print. And so they said, 'We're going to build our own.' Now we'll see. They used to say that about the cloud...
But they have money and they're talented. And a couple of them have said, 'Well, we're bringing this Cerebras and we're going to build our own model.' And so that's another opportunity. I know you can't name names, but is that a trend?
I'm happy to name names. I mean, customers like GlaxoSmithKline have trained their own models on our machines. Customers like Mayo Clinic, we've announced, trained their own models on our machines. We've announced customers like TotalEnergies. So we've signed an MOU with Aramco. And as I said, our work across the G42 companies, including Core42 or MBZUAI and a collection of others in the United Arab Emirates, and they're leaders in their field, all of whom are using both open source and training their own, using their own data to create advantage through AI.
And I know of at least two others that you didn't mention, so I won't. And then the third piece is on the inference side. Our vision of Agentic is that these agents, people think these agents are like God agents. No, they're worker bees, but they will learn from human reasoning traces.
