Advancing to AI's Next Frontier: Insights From Jeff Dean and Bill Dally

I mean, speaking of this, it seems like we're going to have a lot more of these agents operating in the background and that one of the things that's really going to be important is how do we get ultra-low-latency inference so that these systems can do their work autonomously faster, because I think that's a really rapid driver of how effectively they can solve problems. How do you see the next architectures you at NVIDIA are working on delivering major latency reductions? How do we think about getting to, instead of hundreds of tokens per second, thousands or tens of thousands?

Yeah, I actually talked about this in my talk at 10 o'clock this morning, which I think was recorded if you missed it. And there are a bunch of answers to that question. When you look at the performance curve for inference, it's a trade-off between, at one extreme, if you're willing to give up latency you can get very high throughput in tokens per second per dollar or tokens per second per watt. Then you can move down that curve and basically, running smaller batch sizes, you can optimize more for interactivity, more single-user tokens per second, at the expense of that throughput. As you get down to the right side of that curve where you're really optimizing for latency, it turns out that the bulk of the delay is communication latency. And so we're working on developing architectures where we drive that latency down to what, at NVIDIA, we always refer to as 'speed of light.' If you look at a typical LLM where you're doing a bunch of feed-forward stages, then a bunch of attention stages, and that's one layer, and you do that for anywhere between 50 and a couple hundred layers, every time you finish one of those stages, there's usually an on-chip communication to get to the next stage. And then after you finish a layer, it's often an off-chip communication to get to the next layer, and sometimes even between stages within a layer, depending on how you've partitioned the problem. On-chip, we've got trial designs whereby statically scheduling things so that there's no routing overhead, there's no queueing, there's no arbitration. We basically can achieve two millimeters per nanosecond, basically the time of flight over wires on the chip, and get from one corner of the chip to another corner of the chip in 30 nanoseconds. That really drives the latency down from what is several hundred nanoseconds today. And then off-chip, a lot of the latency is actually in our PHY, because for years we have optimized these PHYs for the maximum bandwidth without too much consideration for latency. And so to push the extremes, we wind up with heroic digital signal processing, trying to pull this super noisy signal off the wire and run all sorts of fancy math to figure out what the bit is. And even then, the bit's kind of noisy. So then we do heroic forward error correction to pull the right bit out. It turns out if you back off in the bandwidth just a little bit, go from, say, 400 gigabits per second per pair to 200, all of a sudden all that goes away. You can just detect the voltage on the wire, you get the bit, and it's just serialization latency at that point. It's a couple of clock cycles going on and off chip. So we see our way to build whole routers, similar to the routers I built at Cray 20 years ago in 2006 when we did the Black Widow. The router was less than 50 nanoseconds pin to pin. I think we can do that again. And by doing that, I can see us running relatively big models at 10,000 to 20,000 tokens per second per user.

Mhm. So let's see. One of the challenges that always exists when you're building hardware, especially in a fast-moving field like machine learning, is if you start a new hardware project today, it might pop out the other end and actually be in data centers in two years if you're lucky. We'd like that to be shorter, but then it has to live for a certain number of years. And so you're really trying to predict where the machine learning field and AI fields are headed in the two- to five-year time window. That's always been a super hard thing to predict well. I'm wondering if you've come up with good tools or mechanisms to help do this kind of crystal ball reading effectively.

Yeah. So we try our best, and part of the way we do that is by trying to develop our own models, both LLMs like NeMo, and world models like Cosmos, or robotic foundation models like GR00T. But as hard as we try to do that, we're trying to look out ahead, developing a model that's not yet the one that people have. We get surprised there. There's lots of people out there working on this. They're super clever. They're coming up with great ideas every day. And so we kind of have to future-proof our hardware. We do that in a number of ways. First, there are things you do that are just good for all models. If we can come up with a way of doing math more efficiently, perhaps due to better number representations, that makes everything better. Similarly, if we can organize our on-chip communication to reduce data movement, that makes everything better. Where we tend to run into places that we really wish we knew what was coming down the pike is when people change the model in a way that changes the provisioning, sort of how much math you have versus how much memory bandwidth versus how much memory capacity versus how much communication, because even if you make each of those four ingredients really efficient, you've got to decide how much of each to have. Then when somebody comes up with a different model, like going from grouped-query attention to multi-latent attention, it changes some of those ratios pretty significantly, and all of a sudden you're leaving some pieces of hardware idle and others busy. There's really no way around that. Except perhaps in the future, if we see a bunch of different models needing different sorts of provisioning and the volumes are large enough, we may just do different SKUs provisioned in different ways and kind of hedge our bets against that kind of future.

I guess it's my turn. Not yet.

Yeah, I mean I think first I'll clarify, Chinchilla scaling laws are optimal for training. If you start to consider inference performance of the trained model, then it may be that you don't necessarily want to follow exactly that. I also kind of disagree with the assertion that we're running out of data. I feel like there's an awful lot of data in the world that is not yet being used to train these models.

Well, we train on some video data, but I think there's a lot more video data, and associated audio data, that we aren't necessarily training on yet. I think real-world robotics or autonomous vehicle data is going to be fairly plentiful. Synthetic data is another recourse for how do you generate really interesting high-quality data.

Yeah, but it still seems to help the model sometimes, because if your model is very powerful, the one that's generating the synthetic data, then it actually does seem to help quite a bit.

So I think that's where, and then there's all kinds of techniques we're not yet using that were in vogue for, say, image models, convolutional image models, years ago. So I think things like data augmentation, of which you can think synthetic data is a little bit of that, techniques to prevent overfitting are interesting. You can use dropout or distillation as ways of regularizing the model. So I think there's a lot of opportunity there to really make models better with more compute and more passes over the data, where you won't overfit necessarily.

Yeah, I mean I do think the current pre-training regime is sort of like you take your model, you throw your random number generator at it to initialize it, you strap it to a board, and then you stream all the internet data past it and it learns what it can from that observation, but it's not actually taking actions in the world. It seems like we want to interleave some of that with something where the model gets to take actions in some environment, be it simulated robotics environments or trying to predict answers to questions or things like that, and then going back to learning. Maybe it's also a bit more directed in how it chooses what data to see next, as opposed to a predetermined ordering of the data. I think that would actually be quite interesting as a way to dramatically improve essentially the learning efficiency. For a given number of tokens, how much can the model get out of it? I feel like taking actions in the world is going to be super useful for that. We do that in post-training, but that's a very limited form of that. I think interleaving that kind of thing much more at the pre-training stage makes sense.

I guess it has to get smart enough where that makes sense. When you start out with a random number generator, it probably can't do that, but after you've thrown some number of tokens at it, show it 10 billion tokens and it can probably do something.

Yeah, so that's a really good question. I'll go further. I think it's not just that inference is starting to become important. Inference is the job now. It's easily 90% of the power in data centers that is going into inference. The good news is there's a lot of similarities between both. It's interesting that, for example, number formats like NVFP4 that we developed originally for inference have proven to be remarkably effective at doing training as well. But it's still a very different task in terms of what the demands are, particularly in terms of the memory system, because with training you've got to keep all the activations around for the backprop step, and in inference you just throw them away as soon as you're done with them. So there's a larger need for memory capacity, and it drives, like what I said before, a different amount of provisioning between memory capacity, memory bandwidth, math, and communication. We see that same difference in need for provisioning even in different stages of inference. For example, the prefill stage is a lot like training in that you get all these tokens you can operate on in parallel. So it tends to be very, very math heavy and dominated by communication energy if you're not careful about localizing your data. Then you get to the decode stage of inference and you now are doing, especially if you're latency optimized, matrix–vector or matrix very skinny matrix products, and you're extremely memory-bandwidth-limited and latency-limited. Even that is broken into the feed-forward and the attention stages that do different things. I think that it's a natural division over time to build different components provisioned for these different stages. I suspect you'll see at least three flavors: one that's good at training and prefill, one that's good at attention, and one that's good at decode. That one that's good at decode may even bifurcate into the attention decode and the feed-forward decode.

Yeah. It's so sequential. You've got one token, you've got to run it through everything.

Yeah. Right. And diffusion LM can do even better, possibly doing blocks up to a few hundred.

Oh, I've got to somehow queue up the next one in my head while I'm answering yours. Where are my darn questions here? Okay. Okay. Yeah, so I'm going to skip that one.

Yes. Yeah. Yes. So new models are appearing on a regular basis with new forms of attention, diffusion transformers, hybrid state-space transformer models. What do you see on the horizon for models? What excites you most in the model zoo at present, particularly in attention, across the board?

Right? That's not just constant factor. That's changing it from n squared to some n log n or something.

Okay. But what you do for each chunk can be much smaller. So that helps. Then there's some interesting things where you can, for example, cluster the attention state and then only look at the clusters that are near the thing you're trying to do, and those actually can be sort of n log n or perhaps even better. So that seems quite promising as well. I think the trade-off there is generally you lose a little bit of quality relative to the full quadratic attention. So it's a question of whether that's useful. Then I think there's also a notion of the models being very good when you put the right information in the context window because they can pay attention to it, but the attention window is not big enough to be what you would really want, which is: I would like to attend to all information, all the documents on the internet, or for me personally, all my email and photos and so on. That's much, much bigger than a million tokens. So I think the right way to give you the illusion of attending to, say, a trillion tokens instead of a million is to have staged forms of much lighter-weight retrieval mechanisms that then identify what are the documents you really need, pull them in, and then attend to that. You don't need all trillion at once for a particular task. You'd like to somehow identify which million of those trillion you really need, pull them in, and then attend to that. Maybe at that stage you don't go from a trillion to a million. You go from a trillion to 10,000 documents or 10 million tokens. That's maybe 10 or 20 million tokens. Then you look at them with a much lighter-weight mechanism to identify what are the million-ish tokens you're going to put in. I'm pretty excited about that, because I think that kind of hierarchical attention mechanism will enable you to do things that just the attention window size won't permit.

Very great Nature paper, by the way.

So we're trying to use AI wherever we can in our design process, and there are a number of examples. For example, every time we have a new semiconductor process, we have to port our standard cell library to it. It's about 2,500–3,000 cells. That used to take a team of eight people about 10 months, so 80 person-months. Then we developed a program based on reinforcement learning called NVCell. I think we're up to NVCell 2 or 3 now. It's overnight on one GPU. The results are actually better than the human designs in measures of goodness of size of the cells, power dissipation, delay. It matches or exceeds the human design. So it's a huge productivity gain and eliminates an obstacle to moving to new processes because now we can move cell libraries over very quickly. Then we have this program called PrefixRL, which applies reinforcement learning to a really age-old problem in computer design, which is where to put the look-ahead stages in a carry look-ahead chain. This is a problem that's been studied since the 1950s. This RL program goes at it like it was an Atari video game. It puts something in here and it grades itself. It's not trying to make the fastest adder. It's trying to make the adder that barely meets timing and is as small and low power as possible. It comes up with totally bizarre designs that no human would ever come up with, but they're actually 20 or 30% better than the human designs by those metrics. At a more holistic stage, for a number of years we've had a series of LLMs that we called Chip NeMo and Bug NeMo. What we did is we took a generic LLM and then we fine-tuned it by feeding it all of the design documents proprietary to NVIDIA. So this is stuff that you can't get outside the company. It's all the RTL hardware design docs, all of the RTL for every GPU ever designed at NVIDIA, all of the architecture specs for those, all sorts of stuff. Now you have this LLM that's actually very smart about GPU design. One of the biggest gains of this is when you have a junior designer. It turns out the senior designers spend an enormous amount of their time explaining to the junior designers simple things like, how does a texture unit work? And in particular, this texture unit. Now they don't have to ask the senior designers that. They can ask Chip NeMo that. Chip NeMo will explain to them in great detail how the texture unit works. They have a follow-up question, it will answer the follow-up question. So it improves productivity that way.

Yeah. And the same thing when we have bugs, we can have it summarize bug reports and try to do attribution on bugs, say which module is this bug in, who do we ARB? We always ARB somebody for a bug, 'action required by,' and so we'll decide what designer gets ARBed for a particular bug. Then, at the more exploration level, and particularly in NVIDIA Research, we're now using agentic systems to try to do a bunch of thought experiments where we say, 'Okay, here's a bunch of different ways we could go. Figure out what this parameter space looks like, suggest new ways of doing things, run simple architecture experiments, and whittle the design space down to a smaller number of things that we can look at.' We're trying to collapse the time from when, at NVIDIA, there's a stage when we're kind of done with the exploration effort that we call we have the F-model. It's basically an executable model of the GPU. To where we tape out the design, we have all the geometry done, it goes out to TSMC to make chips. We'd like to collapse that space. It turns out what the really long pole in that space is design verification, and so we're particularly looking at how we can use AI to prove the designs work more quickly. There are also a couple stages in there where we have to refactor the design because we have the RTL design done, but then we've repartitioned things for floorplanning and now we have to move parts of the logic from one module to another and make sure everything still works. Those are the kind of things that I think we ought to be able to automate, and we should be able to collapse that time period down. But all that said, I'd love to have the end-to-end stage where I could simply say, 'Design me the new GPU,' and I'll go out skiing for a couple of days and I'll come back and it's done. But I think we're a long way from that, although it is making us much more productive.

Yeah, and potentially even split up in the same way we split up human chip design teams, where you have this agent focusing on this part of the chip and then this one over here with some well-known interfaces between them, and then they have to negotiate changes to those interfaces. They'll have the same kind of meetings we have, but between agents.

Okay, I have to ask a question now. I should have organized this in two columns or something. Okay, let's see what we've got here. Yeah. So on that topic of agents, if you have teams of virtual engineers and swarms of agents collaborating on complex projects, what are the biggest orchestration and continual learning challenges that hardware innovations could dramatically accelerate? Things like hierarchical rewards, sparse activations, dynamic model growth. What do you see happening there?

Do you see this happening right now for coding tools?

Let me see. So energy is something I've seen you talk about in a bunch of your talks and slides. The thing that really sticks with me is what we ultimately are trying to do is a multiply-add operation or something, which is a fraction or a few picojoules depending on the precision. Then in order to actually make that happen, we have to bring that data from some part of the memory system, and that's a factor of a thousand more in many cases, more energy. Given the fact that we're really pushing and trying to make these systems as energy efficient as possible, I'm curious what you think are going to be the big improvements. I mean, you spoke to some of these, like shrinking the distance that things have to go across the chip; that can help. But maybe you have a more broadly characterized status: put the numbers straight.

To put the numbers straight, typically doing a multiply-add, say for NVFP4, it's like 10 femtojoules. To pull those four and a half bits in from HBM4 at on the order of maybe three or four picojoules per bit, you're talking about 15 picojoules. It's a thousand times more energy to read one NVFP4 number from external memory than it is to do a multiply-add. But reading a simple SRAM bank is also on the order of about 10 femtojoules. So the key thing you can do to reduce energy is to just don't move data. And people are laughing, but I'm serious. No, that's how you do it. That's absolutely what you have to do. For example, we're looking at designs where, for where we can put things in SRAM, we have an SRAM. We can read a row of a matrix out of that SRAM, have a row of an activation vector, and do a dot product right there. We've done two things: we've done a thousand multiply-adds with no data movement, we just read the SRAM, everything is there in place, we do a thousand reads from your left hand and a thousand from your right, and then we've reduced the data from being a thousand long to one by doing the dot product. We're trying to look at a lot of techniques like that. The problem then is that SRAM is expensive in terms of area per bit. It's an order of magnitude more expensive than DRAM. The technology we're really excited about but it's not quite ready is where we can stack the DRAM on top of the chip that's doing the computation. Then we can basically cycle the DRAM mat, as they call it, sense the bit, and bring it straight down. It turns out most of the read energy of reading the DRAM isn't actually reading the DRAM; it's moving the bit from where you read the DRAM over to where the pins to the GPU are. By putting this DRAM directly on top of the GPU, we can get an order of magnitude more bandwidth with an order of magnitude less energy per bit, which winds up being the same power but with way, way more performance. Those are really the things we're looking at doing. Of course, more efficient models will help also. If you can get the same accuracy with less work, that reduces energy also.

Yeah, definitely. I mean I think models that kind of reuse their parameters many times are kind of nice in some ways.

But reuse is, again, part of that curve trading off optimum throughput, optimum tokens per watt against optimum interactivity. When you're really worried about interactivity or when you're only running a single model and you've got one token, it's got to see every weight, the reuse is one.