Agentic is a little bit like the Nvidia of AI. It's about being able to break things up into parallel tasks. Why that's leading to an absolute explosion in the usage is AI using AI using AI. One of the interesting things about the Grok architecture, the LPUs, is that we actually have a kernel-free architecture. I think you're going to see many, many smaller players making the chips because the stakes are so high. And so if you don't have a chip to launch, that's a very expensive mistake. As long as there are unsolved problems in civilization, we will have a need for more compute. Right now, cancer isn't cured.
People still get old. With AI becoming as inexpensive as it is, it's going to increase the demand for AI to the point where people are going to spend more and more on it. They're going to need more compute. And so I just think we're going to have this endless demand for compute.
Jonathan, we were actually both Google alumni. When I was at Google, we had this running joke in my team that if we ran out of quota for the day to train our models on TPUs, we might as well just take the day off. I know you pioneered TPUs and then you left to build your own company that makes a chip. What did you see at Google that made you want to build something different?
We didn't have enough compute. What had happened was the speech recognition team had trained a model and that model was better than human beings at transcribing. This was the first time they had ever achieved that. The problem was they couldn't put it into production. They had actually limited the deployment to the Nexus phone, the old Android phone.
Oh yeah, okay. So they limited it to Nexus, not so much as a feature, but because they had so little compute, they could only support the Nexus user base. I happened to be having lunch with the speech recognition team in New York City and they mentioned this problem. I started as a 20% project porting their model to an FPGA, created a general architecture, and then it turns out inference was needed quite badly and it became a chip. Actually, Jeff Dean did an analysis and was like, given what we're going to spend on this in capacity, let's just do an ASIC instead. My response was, how hard could it be? Turns out it was very hard, but we didn't know, so we jumped in.
I heard you use the term 'success disaster' in the past. I think that captures it very well, and I've experienced that at Google multiple times.
Yeah. So, Nvidia GPUs are great for training, but at inference time they're memory bottlenecked. How does Grok change the memory architecture to address that?
Well, first, it's important to think about the trade-offs. There's no such thing as a free launch. What you're trying to do is get the lowest cost per token because the cost actually determines your capacity, right? Everyone's in a race. People would pay more for more capacity, but if I'm spending twice as much, I'm getting about half as much capacity per dollar. That's really what I care about: can I get this many tokens per dollar? You also need speed. The trade-off is if you want the absolute best cost per token, you're just going to use a GPU as it is, with a very large batch size, and it's not going to be as fast as it otherwise could be. What we did with the LPU was we were able to scale to multiple chips without using any external memory and spread the model out across those chips. So we could use much faster SRAM, which allowed us to generate tokens faster but at a lower cost. If you know what a Pareto curve is, if you look at the Pareto curve of GPUs versus LPUs, they're quite different. There are portions of the curve where the GPU is better economically, and there are portions where the LPU is better economically, usually at the faster end. When you put the two together, it fills out that middle zone. So between the GPU, the GPU plus LPU, and the LPU, you now have the best cost per token. You have the most capacity at any speed you want to run at.
Another differentiator for Grok is that it uses static scheduling. The order of operations is predefined at compile time. Why is that an advantage for LLM inference?
Let's use an example in calendar scheduling. If I want to have a bunch of short meetings, 15 minutes long, I have to schedule them because the person I'm meeting with has to show up at the right time. If I'm going to have a five-hour-long meeting, I don't really need to schedule it; you just show up. If you're 30 minutes late, it's 30 minutes out of five hours. With inference, you're doing super low latency, small batch size. You need to schedule all those operations so that each portion of the computation gets done quickly and frees up the hardware for the next part. You're not stalling all the work that has to come behind it. In training, this is less important. For inference, it's absolutely crucial.
A lot of state-of-the-art LLMs today use a mixture of experts architecture. At inference time, for every query, a different set of experts potentially gets activated. How does that work on a chip that does static scheduling?
The question is what's being statically scheduled. I have this 15-minute slot on my calendar, but who I meet with can change. In the LPU, we have the ability to do scatters and gathers. That means depending on which expert is needed, we will fetch a different expert. It still runs for the same amount of time; it's just a different expert. If we have different-sized experts, we could even route to another chip, but then there's a little bit of a bubble in the pipeline. The determinism gives you a lot more ability to predict the timing; it doesn't restrict you to what you can run. The LPU architecture is particularly good at experts because the smaller the batch size, the better. Experts are particularly bad on batch size because if you're using external memory, you're pulling in one expert from outside and then you have to amortize that across a bunch of computation. If I'm reading from DRAM and then running that expert, I might need a batch size of hundreds to make economic sense. If I'm using an LPU, I might need a batch size of 10 for it to make sense, which means I don't have to wait for as many queries to batch up in order to run it. That brings the time down and the efficiency up. LPUs are almost perfect for expert models.
Speaking of architectures, when the transformer is replaced by the next shiny architecture, do the LPUs have to be completely redesigned, or are they orthogonal to the current shape of LLMs?
The age-old question. When we designed the LPU, the 'Attention Is All You Need' paper had not yet been published. There are a lot of things that rhyme between attention and some of the other things that existed at the time, like convolutions. They are quite different, but it's all linear algebra. If you've built an optimal chip for linear algebra, then you've built an optimal chip for most of these architectures. You may decide to optimize for the size of the matrix multiplication, which could be different from one architecture to another. I've seen some people try to go very specialized, but what ends up winning the most is flexibility almost every time. If I was to tell you I was going to limit your ability to change the model, you could never change it again, but I could run it 10 times as fast, would that be interesting? The answer is probably not, because there's probably going to be a 10x improvement in algorithms. There's just been a recent change in the way attention works to shrink things 10x. Algorithmic improvements happen so quickly that flexibility is often what matters more. The LPU architecture is specifically designed to be super easy to program so that when new architectures come out, they can be adopted and the latest algorithm can be running very quickly.
The 'L' in LPU stands for Language. Does that mean that currently vision and audio models don't benefit from the same speedups? Or, as you mentioned, it's a general enough architecture to support other modalities as well?
One of the biggest users of the Grok cloud that exists is speech-to-text users, and we also had some text-to-speech for a while. The reason is it's super sensitive to being real-time. Many of those models have convolutional layers embedded. This is where having a general architecture matters, because otherwise you wouldn't be able to run all this voice stuff on it. Where it really comes into play is it actually improves the quality by being faster. It's a little counterintuitive. You can split up audio segments into very small chunks and just run that chunk. But if you only hear a small clip, you don't get the full context, so it's much harder to predict what those words are. When people are using slower chips to do audio processing, they chunk it up more to make it real-time, which increases the error rate. It's like having two different people transcribe a speech at the same time, but each only heard five seconds of the speech at a time. They would have a lot more errors because they wouldn't have the full context. Because the LPUs were able to do speech transcription at hundreds of times faster than real-time, they could work on much larger chunks and actually improve the error rates on these models.
The use cases we talked about so far—language inference, audio—are mostly autoregressive, but vision models today are diffusion-based, and some LLMs are diffusion-based as well. Diffusion LLMs are a lot faster on a GPU than autoregressive LLMs. Does that ranking still hold on a Grok chip?
Diffusion models benefit from the total amount of compute you have. Let's get down to what autoregressive means for the audience. Autoregressive basically means I'm going to figure out what this word is, then figure out what the next word is. Sort of like playing chess: I figure out one move before making the next, as opposed to figuring out all the moves in parallel. In language, it's hard to figure out what the 100th word is until you've figured out the 99th word. However, you can start breaking that down, saying some words matter more than others. I'll predict the important word and fill in other words around it. I'm seeing a lot of people trying to use diffusion models to generate language, and they're not getting great results. The reason is it's really hard until you've made a decision on what you're going to say here to figure out what you're going to say here. It's like a hundred people writing a speech, each not being able to see what the others do. In diffusion, information diffuses in time and space; the further away things are, the less impact they have. From a quality point of view, if you want to generate music using autoregressive versus diffusion, the autoregressive version will be soulful, deeper; you'll like it, but it might have a crackle or odd noise. Purely diffusion will be crystal-clear elevator music with no soul. Put the two together, and the important moments can be done autoregressively with context, and you fill in the rest with diffusion. Just like we paired the LPU and GPU together for decode in LLMs, I think the successful versions of diffusion in LLMs will likely combine autoregressive and diffusion together.
You've already alluded to putting together GPUs with LPUs. Nvidia announced earlier in March at GTC the Vera Rubin supercomputer, which is dedicated for inference, especially for agents. Can you tell us in what ways GPUs and Grok work together at inference time?
