Bill Dally - Trends in Deep Learning Hardware

Okay, well, thank you. Thanks for inviting me to Berkeley. I always have fun when I come visit. Today everybody's lives are being revolutionized by deep learning. I think it's going to change just about every aspect of our life, hopefully for the better. I think it's kind of what we make of it. I decided to get an unbiased view and ask ChatGPT what ChatGPT was good for. This was the reply: how can deep learning enhance the human experience? Healthcare, smart assistance, education, transportation, entertainment, environment, and so on. As a concrete example, this is Alagent, an agentic system built in our Toronto AI lab at NVIDIA, which basically acts like a graduate student in quantum chemistry. If you ask it a question like, 'What is the activation energy of caffeine?' I find that when I drink caffeine, it activates me. It basically answers that question by using a graph on the right, a bunch of specialized agents that each carry out a certain task with tools. The tools may involve preparing an input file with the molecule, running a density functional theory simulation on a supercomputer, interpreting the results, running some additional codes, and eventually coming back with the actual answer. This was relatively quickly put together with the agents fine-tuned for their different tasks, and it sort of acts as maybe a second-year graduate student in its ability to carry out tasks that the research supervisor gives it. I could go on with lots of examples, but this is not a talk on applications; it's a talk on hardware. So how did we get to this point where we have these great tools that act like power tools for our brain and make us many times more productive? When I need to write a piece of code now, I think about what I want the code to do and I have Cursor or Claude write the code for me. I don't actually write it myself. It's kind of moved me up a level of the management chain. Deep learning really has three main ingredients. The first ingredient is algorithms, which I've illustrated here by the figure from the AlexNet paper on the bottom. Most of the algorithms used for deep learning, like deep neural networks, convolutional neural networks, training them with backpropagation and stochastic gradient descent, have been around since the 1980s. When I was a graduate student at Caltech, I took a course on neural networks that John Hopfield gave a number of guest lectures in. I thought these were really neat, but we didn't have enough computing power to do this, and I sort of classified them away as an interesting curiosity. The next ingredient is data, and large labeled datasets like COCO, Pascal, and ImageNet came around in the mid-2000s. ImageNet was 2009, with enough images labeled to train these networks. But still, there was a missing ingredient. You can think of this as the fuel in the era of deep learning. The missing ingredient was hardware that was fast enough to train a reasonable size model like AlexNet on a reasonable size dataset like ImageNet, which at the time was a little over 1 million images, in a reasonable amount of time, which was two weeks. That was the spark that ignited that air-fuel mixture and really set off the deep learning revolution. Since then, it's been a wild ride. As the primary developers of deep learning hardware, we feel a great responsibility to continue making the hardware faster every year because that's gating the performance of deep learning. Starting with AlexNet, the number of operations it takes to train a state-of-the-art model was a few hundred petaflops. Now we're into many tens of billions of petaflops to train a mixture of experts transformer model. During this period, the blue line is when we were building image networks running them on the ImageNet benchmark. Progress was going up at about three times per year. But then in 2017, some people at Google wrote this paper called 'Attention Is All You Need,' and people started building transformer networks, which put us on this 16x per year trajectory. The increase in demand has been 10 million over the last 10 years in the number of operations it takes to train a state-of-the-art model. So how do we meet this demand? I'm going to give some history. I'll start with ancient history. This is a slide from a talk I gave at the Computing Consortium talk in Washington earlier this year about how government-funded research has been essential to building many American industries. It really lays the groundwork. A lot of people think GPUs appeared and everything was fine, but GPUs didn't just appear. There was fundamental work on parallel computing and then on stream processing, which laid out most of the fundamentals of GPU computing with kernels and producer-consumer locality. These were all DARPA-funded research programs. The thing that most directly led to GPU computing was a program I ran at Stanford to build a streaming supercomputer. Out of this came a language called Brook, which evolved into CUDA. We ported that to the GPUs of the time, which had programmable shaders but did not have compute shaders yet. Then I went and consulted at NVIDIA on the NV50 to put the compute shaders into them. One of the graduate students on the project, Ian Buck, who by the way is now the General Manager of Data Center Computing at NVIDIA, went to NVIDIA and took Brook and turned it into CUDA in collaboration with a guy named John Nichols. A lot of this stuff would never have happened without government-funded academic research. It served both to develop the fundamental ideas and, probably more importantly, to train a bunch of students who then went and took those ideas to industry and made them into real products. I'm going to zoom in on this part of it. I won't go into that in detail. This shows the increase in GPU performance on AI inference over the last 12 years. It's 5,000x. I drew it as a linear plot because when I was on the faculty at MIT, I consulted for DEC and they had this rule that if you were to present to an audience that had anybody above the rank of vice president, you weren't allowed to use a log plot because they wouldn't understand it. Looking at this audience, I think I'm safe to go to this one. What you see is that we've had a steady increase, basically doubling performance every year over the last decade. This is pretty remarkable because everybody thinks, 'Oh yeah, it's Moore's law. It's just technology.' It's not. We've gotten very little of this from technology. The technology nodes are labeled here: 28 nanometers is in black, 16 nanometers is green, 7 nanometers is blue, 5 nanometers is purple, and Blackwell, which is red, is in 4 nanometers. I've gone and looked at each of these GPUs on this chart and asked where the performance came from. Through Hopper, 2.5x, through Blackwell, 3x came from process technology. Out of the 5,000x, 3x is from process. Everything else is from better architecture and clever design. The single biggest contributor, 32x, is from number representation. When we started doing deep learning on GPUs in 2012, we were just starting to ship Kepler GPUs. The first Kepler GPUs went into the Titan machine at Oak Ridge National Laboratories, which at the time was the fastest supercomputer in the world. That chip was designed not for AI because we had just started playing with AI two years earlier in 2010, which was too late to really affect the design of Kepler. It was designed to do graphics and supercomputing, high-performance scientific computing. For graphics, we had FP32; for high-performance computing, we had FP64. So we were doing AI with FP32, which is gross overkill. The energy required to do an arithmetic operation is dominated by the multiplies, and multiply energy goes as the square of the number of bits. If you think about doing a multiply in long form, you generate a number of partial products equal to the length of one operand multiplied by the length of the other operand. If you double the length of the operands, you quadruple the energy. Commensurately, if you go from 32 bits down to 8 bits, that's not a reduction of 4 in energy; it's a reduction of 16. A lot of our journey along here, 32x of that 5,000x, is from going from FP32 down to FP4, which is what we're shipping in Blackwell. I'll go into a little bit more of that later and how we're able to make the low precision work. The next thing is complex instructions. I know this comes a little bit as heresy to people at Berkeley where RISC was invented, but it turns out that carrying out an instruction takes a lot of energy. I'm going to flip to the next slide and then come back to this one. Even GPUs, which are very efficient in how they do instructions because they don't do any speculation or branch prediction, all the things that make an instruction in a CPU very costly but also make CPUs very good at single-threaded code, the cost of carrying out a single instruction without doing any work, just fetching the instruction, decoding it, fetching the operands, normalized to 45 nanometers, is 30 picojoules. In comparison, the energy of doing a floating-point multiply-add, two arithmetic operations, is 1.5 picojoules. So you're basically spending 20 times as much energy on overhead as you are doing real work. It's kind of like a corporation. Over various generations of NVIDIA GPUs, starting with the Pascal generation, we introduced a dot product instruction. So now instead of doing two arithmetic operations, you do eight. You basically take four numbers, multiply them pairwise together, and sum that all up. Now the overhead is only 5x, but still you're doing more energy in overhead than in operations. It wasn't until the Volta generation that we introduced matrix multiply instructions. A single HMMA instruction, half-precision matrix multiply-accumulate, takes two 4x4 FP16 matrices, multiplies them together, that's 64 multiply operations, and then sums that into an FP32 matrix. If you add all the adds up in there, there are also 64 add operations, so it's 128 operations done with a single instruction. Now the overhead is down to 22%. In the Turing generation, we introduced IMMA, integer matrix multiply-accumulate, which does the same thing but with two 8x8 INT8 matrices multiplying together and then summing into an FP32 matrix. Now our overhead was 16%. It is now, without going into a lot of detail, 11%. This means that our overhead is so small that our efficiency is essentially identical to that of a dedicated accelerator aimed just at deep learning. If you had hardwired it for the algorithm, you could at most do 11% better. By the way, you can't because hardwired accelerators have overhead too. It's not all arithmetic energy. But we get the advantage of having a completely programmable engine, so we can adapt very quickly as algorithms change. Complex instructions are the next big gain, about a 12.5x gain over this period of time. Process has not been completely a bust, but compared to the heydays of the 80s and 90s when CPUs doubled in performance every couple of years almost entirely due to process, and you would get sort of 8x per generation in performance per watt for doing the same function unit, it's kind of disappointing that over 28 nanometers to 4 nanometers, essentially an entire order of magnitude of scaling, we get 3x. Sparsity, exploiting the fact that most of these matrices are not dense but have a lot of zeros in them, gives us 2x. I'll explain that a little bit more later. On Blackwell, we kind of cheated. Blackwell is actually two die, two reticle-limited GPU die on one substrate that we call one GPU. That last jump is doubling the die area as well. At the same time we were doing all of this, an equal gain, at least equal gain, came from people coming up with better models. For example, looking back at those vision models, going from AlexNet and VGGNet, which basically had a fully connected layer, a bunch of convolutional layers followed by a number of fully connected layers, and their arithmetic performance was dominated by those fully connected layers. The next year, ImageNet was won by a network called GoogLeNet, which eliminated the fully connected layers and increased the efficiency by more than an order of magnitude in terms of the performance you got per arithmetic operation. Let me jump ahead here. This brings us to our current generation, Blackwell, which is an amazing piece of engineering. It's two reticle-limited die. The connection between them is something we call NVHBI, 10 terabytes per second of bandwidth, connecting these two die together at a few femtojoules per bit to transfer across. It's really a nice piece of circuit design that was done within NVIDIA Research, so I'm kind of proud of it. It has 8 terabytes per second of HBM bandwidth from these eight stacks of HBM3 memories. It's an enormous amount of computing power in one thing. But as I showed earlier, we needed 10 to the 7th performance increase to train state-of-the-art models over that time, and we got 5,000. So where does that other several thousand come from? The answer is parallelism. For both training and inference these days, it is typically not done on a single GPU. We chop it up into pieces and exploit parallelism along at least three dimensions. The first, and the easy dimension for training, is data parallel. If I have a 100 million image dataset I'm going to train on, I can break that up into what are called batches of images. I can send one batch of images to this plane of GPUs and the next batch to this plane of GPUs, and so on. They all train on those images separately, and then at the end of the batch, they exchange their weight updates so everybody learns from every image even if they didn't see those images. Then you go do the next set of batches. Data parallelism is easy because the communication is relatively small. The training is done essentially separately on each batch, and there's this one little exchange of weight updates at the end. But that's not enough performance to get us to the 10 to the 7th. The next easiest thing to do is the vertical dimension here, what's called pipeline parallelism. If you look at any neural network, it's a bunch of layers. If you look at transformers, there are typically a couple of matrix-vector multiply layers followed by the attention layers, and that's one layer of the network. Llama 70B is 80 copies of that. You can take each of the 80 copies and run them on a separate GPU. You can even take each of the matrix-vector multiplies that are being done in those layers and run them on a separate GPU. Then all you have to pass from one GPU to the next is that set of activations that comes out of that layer's computation. So again, it's a relatively low communication overhead decomposition. The harder thing to do is going in this direction, tensor parallelism. It's where I take a single matrix multiply or matrix-vector multiplier and I break it across GPUs. I do that by taking the matrix and slicing it. Think of the simple case where I have a matrix. I slice it in half. I multiply both halves by a vector. Now I just have to take those two parts and add them back together. So there's a reduction required to do the tensor parallel. We typically take all three of these dimensions and exploit them to get that last factor of a few thousand to get us up to 70,000 in that period. To make that feasible, we've worked at NVIDIA not just designing GPUs with ever-increasing arithmetic performance due to complex instructions and new precision data types. We've also worked very hard on the communications, starting in the Volta generation with our first generation NVLink and NVSwitch. The latest instantiation of this in Blackwell is a single cabinet we ship with 72 Blackwell GPUs and 36 of our Grace CPUs, connected entirely electrically. The reason we do it electrically, and we think this is something that hopefully in a few years we'll overcome, is that today optical signaling is about an order of magnitude more expensive per gigabit per second, about a factor of three more energetic, and also significantly less reliable. It's one of these things that tends to fail and require components in the data center to be replaced. By packing a lot of GPUs together, 72 of them, this winds up being a over 100 kilowatt rack. We can keep the connections between them short enough that they can be done entirely electrically, which reduces the cost and energy.

That's a standard sort of a standard 19-inch rack. It's about six or seven feet tall. The switches are in the middle. The longest cable, from a GPU board to a switchboard, is under a meter. From there, we scale up. This is actually an older version of that rack, not quite as densely packed. We take these racks that are connected with NVLink, which is our scale-up network. We use it to scale a single solution running tensor or pipeline parallelism. When we connect it with, in this case, an InfiniBand network, a lot of people today will do either InfiniBand or Ethernet network to connect these racks together to do the scale-out. That's typically where the data parallelism happens, or different clients running different workloads. As an example, within NVIDIA, we run our own data centers both for internal use and for product use. Over roughly the same period, our largest system, which was actually pretty small back in the Kepler generation, has grown by 70,000x in terms of its computing performance. So not quite the growth that's been needed to train models, but it just shows that everybody is building larger data centers these days. That's the hardware side of things. If it was just hardware, I think the space would be a very different one today. The software is probably the harder part of this. One way to think about this is that the computations you do in deep learning are essentially matrix multiplies. Back when I used to teach the introductory logic design class at Stanford, I think if I assigned toward the end of the quarter a design of a matrix multiply unit to the class, most of the students would do a pretty good job. They'd come up with a pretty efficient matrix multiply unit if you added the IO to connect it to DRAM and to some networking devices. So turning out the hardware is actually the easy part. At NVIDIA over the years, we started playing with deep learning in 2010. It came out of a breakfast I had with my Stanford colleague Andrew Ng, where he was working at Google at the time finding cats on the internet. Why people would want to do that is another question, but I simply observed he was using 16,000 CPUs for this computation, and we could probably do it on a few GPUs. So I assigned a researcher at NVIDIA, a guy named Bryan Catanzaro who now runs our Applied Deep Learning Research group, to work with Andrew and port this software to GPUs. The software that came out of that effort became cuDNN, which is one of these blocks here. We basically start with our hardware here and produce layers of software on top of that that build on the CUDA ecosystem to provide turnkey solutions for total end applications running from Modulus, which I think we have since renamed to PhysicsNeMo, which does physics simulations like computational fluid dynamics, hydrodynamics, and DFT calculations. Clara is our healthcare software which uses generative AI to generate proteins and molecules of various types. Drive is our AV stack. Isaac is our robotics. Metropolis is our smart city. And so on. For whatever application a customer has, they can typically at least get started with software that works fairly efficiently right out of the gate. I think this is actually probably one of the bigger barriers to entry in this field. Anybody can knock out a matrix multiplier. Putting the software stack behind it to make it useful and efficient is much harder. One measure of this is the independent benchmarking done through an organization called MLCommons that every six months publishes MLPerf. They alternate between MLPerf Training and MLPerf Inference. The MLPerf Training results just came out in the middle of November, and NVIDIA won every MLPerf Training benchmark. It shows the improvement generation to generation, going from Hopper to the first generation Blackwell to the current, what some people call Blackwell Ultra, the GB300 generation. Over that period of time, it's been a 5x improvement in performance. What doesn't really show in this graph, I thought I would show this one which is from an older version of MLPerf, going from 3.1 to 4.0. We just did 5.1, so this is about a year and a half old. On the same hardware, it gets faster over the period. This was a six-month period from 3.1 to 4.0. One benchmark got two and a half times faster, another one 2.9 times faster. That's because getting efficiency is non-trivial on these applications. When 3.1 came out, Hopper was hot off the press. We didn't have time to tune any of the code. We just kind of ran the obvious out-of-the-box code and got pretty good performance, the gray lines. But after tuning the code, things get a lot better. So that's where we are today. Let me talk about what keeps me up at night and what I think our challenges are going forward, since this is a trends talk. One of the things is, for a while, deep learning was running a model. Whether it was an image model, you took an input image, fed it through a bunch of layers, got an answer out: that's a cat. Or if it was an LLM, you took an input text string, fed it through an LLM, got an answer out, another text string. Today, most applications are not that simple. They're agentic. You are running the LLM, but you're running it many times by an agent that actually has a lot of retained state. It has a memory and can use tools. A great example of a tool is a calculator. If you want to build an AI to do calculations, you'll find that if you just take ChatGPT all by itself, it is remarkably bad at doing calculations, although it's probably better than I am just doing calculations in my head. But if you take a simple LLM, one that may even be only 7 billion parameters, and you give it a calculator, it winds up being really good at doing calculations. Just like I'm much better at doing calculations if I have a calculator. I don't have to do them out by hand on paper. Tools could be that. It could be like in the quantum chemistry agent, running density functional theory computations to calculate the energies of things. So agents with tools are far more effective than agents just using the LLM to do things. But it means the amount of work you're doing per unit of answer to the user is going up. The other thing that drives unique demands for hardware is if you look at running an LLM, there are two unique phases. During what we call the prefill phase, you feed in the prompt. In this case, 'Is apple a fruit?' Most prompts are longer than that. In fact, prompts can be up to millions of characters long because very often for the prompt, it's not just the question you ask. By the way, if you type a prompt into ChatGPT, it gets fed your prompt. It also gets fed their prompt, something called the system prompt, which gives it a lot of context about don't do this and don't do that, and when you answer this, make sure not to offend the user and the like. Very often you'll also say, 'I want you to answer this question, and here are all the financial reports for this company from the last five years. Use this to answer that question.' That will go into the prompt. So prompts could easily be millions of characters, millions of tokens. Tokens are roughly a word. What's great about the prefill phase is it's a very parallel computation. You get all these tokens at once, and you have to multiply them by these linear layers and then by the attention layers. Because you have all the tokens at once, there's lots of arithmetic to do for each time you read those weight matrices out. Then you get to the hard part, the decode phase. In the decode phase, out of all this prompt and prefill phase, you get one token. Then you have to feed that token through the whole model. Say it's Llama 70B, all 80 layers. You have to read that entire model, all 70 billion parameters of it. You have to do all those multiplies. You get one more token out. So you have both a huge amount of memory bandwidth you need. You have to read all the weights of the model for every token, as well as the attention matrices. And you have a latency requirement because typically there's some user-level service objective, something like 100 milliseconds per token you'd like to get. Actually, for some things I'm about to talk about, that number gets harder. So you've got to solve this in a very small amount of time going through all these layers to get one token out. That drives unique demands for memory bandwidth and latency. Another trend that's happened starting with DeepSeek V3 and OpenAI o1 is what people call reasoning with chain of thought, where you don't run the LLM once. Some of the online models will actually let you see, you can ask them to show you their work. You'll ask, 'Is apple a fruit?' and it'll basically produce a bunch of internal thoughts: apples are round, they're red, they grow on trees, fruits are this and that. Then it'll feed that back into the LLM again. The first output will be a string where it's kind of thinking about apples, and then it will produce another output. It will think some more about apples, and eventually finally it'll tell you yes, an apple is a fruit. For 'apple is a fruit,' that's kind of a trivial thing, but for some more difficult questions, like solving some of the Math Olympiad questions, working through those intermediate steps is really important to get to the right answer. Another thing that people will do that takes even more tokens is they won't just produce one stream of thought. They'll say, 'Okay, here are three possible output strings.' Then from each of those, I'm going to produce more possible output strings. I'll have some way of judging these by quality. If the quality falls below some level, I won't pursue it further. But I'm still evaluating a lot of nodes before I finally get to the output string I produce to the end user. So what does this mean for a hardware designer like me? It means really high token rates. If I want to deliver 100 tokens per second to a user, which is what many people would consider interactive, I may need to have 1,000 or 2,000 or even 10,000 tokens per second coming out of my hardware to get there because I'm not just producing a string. I'm producing a string that gets consumed and produces another string and so on before I finally produce something for the user. Agents require even higher rates because they're using their tools and typically you'll have a whole network of agents. If every token I have to read the entire model, let's say it's Llama 405B, and the whole KV cache, which might be on the order of 20 gigabytes per user, that number varies depending on the length of that input prompt because the KV cache is your cached attention computation from the input string. At 1,000 tokens per second, that says I need a memory bandwidth of 405 terabytes per second. Blackwell is great at 8 terabytes per second, but you need more than 50 Blackwells to basically provide that amount of memory bandwidth. So memory winds up being very critical. You can view the demand here as a memory demand and arithmetic demand. A lot of the arithmetic demand is driven by the prefill phase, which is arithmetic intensive, but the memory demand, particularly for memory bandwidth, is driven by the decode phase. The other thing this drives is the latency requirement. If I want to hit that 1,000 tokens per second, let's say I want to do all 480 stages here, 80 layers of 6 stages per layer, if a stage would be one matrix multiplier equivalent, I've got to do each of these stages in 2 microseconds. That includes getting the input data in there, doing the arithmetic computation, getting the output data to the next stage. Typically that involves sending messages between different compute units. It takes some of that 2 microseconds away to actually do the math and read the memory. I've got to have communication delays that are in the hundreds of nanoseconds. The communication delays become very critical here. This is driving some of our evolution of architecture. The final thing that keeps me up at night is the fact that every week somebody sends me a couple of papers to read where somebody has come out with a new model. One that was particularly interesting about a year ago is this DeepSeek model, DeepSeek V3. It had a bunch of things that weren't really all that earth-shattering. In pieces, it's all incremental, but put together, it was pretty impressive. It had something called mixture of experts. This is actually something which makes a memory problem easier. Rather than reading every parameter for every token, you would look at your token and say, 'Oh, this is a token about this category.' Therefore, I'm not going to send it to every parameter. I'm going to send it to a subset of my parameters. I would read some number of, say, 6 experts out of 100 experts that would get that token. This basically greatly reduces the amount of parameters I have to read for each token. It's a huge efficiency gain. Most high-end models today are mixture of experts. The other thing it did is it collapsed the attention calculation. I won't go into a lot of details here, from group query attention, which is kind of the state-of-the-art at the time, to multi-head latent attention, which greatly reduced the amount of math you had to do for the attention calculation by projecting it down to a lower dimensional space before doing the computation and then projecting it back up. Basically, there are all sorts of these things that we haven't seen yet that haven't been developed yet. We have to make sure our hardware is flexible enough to handle all of these. So how are we going to do that? What are some future directions? I'm going to talk about some stuff. I naturally can't talk about some of the things that are a little closer to reality. But one thing that drives where I look for improvements is I ask where do we spend energy today? This pie chart is a budget of where the energy goes in not a GPU, but in one of the AI accelerators we built in NVIDIA Research. What you can see is it's math limited. 47% of the energy is going to math. Basically, if you're doing a good job, that should be the case. Most of your energy should be going to that math because that's the payload. What does the rest of the energy go to? It goes mostly to memory bandwidth. Reading and writing the accumulation collector, that's where you're putting the results, 15% in the accumulation buffer. There are input buffers and weight buffers here. 6% is data movement, the communication. So highest priority, let's make the math better. Second highest priority, let's make the memories better. Third highest priority is let's make the communication better. I'm going to talk about some of the things on the left-hand side in future slides. One of the ones I'm not going to talk about, but it's probably worth mentioning here, is for making the memory better. One thing we've been working on very intensely is 3D memory. Memory exists at a bunch of different levels. The memory you read most often is sitting right next to the arithmetic units, and reading it is very low energy, on the order of a few femtojoules per bit. But if you have to read HBM DRAM, it's about 5 picojoules per bit. So it's several orders of magnitude more expensive. One way to reduce that cost, and let's just say one thing driving that cost is I have these reticle-limited chips, 30 millimeters on a side, with the HBM DRAM there. On average, I'm pulling one of those bits from the HBM DRAM and dragging it across 20 or 30 millimeters of linear communication to get it to where it's going. That's a huge amount of energy. One thing we see as a long-term direction is to put the DRAM directly on top of the GPU and localize the computations. Now we can read the DRAM directly vertically down through a bunch of things that we think can go along with that, reduce that HBM memory read from maybe 5 picojoules per bit to maybe half a picojoule per bit, and at the same time drive the memory bandwidth way up. But let me talk about some of these other things in a little bit more detail.

Yes, it would look very different.

It does, but I didn't have that pie chart. That's a good idea. I'll put that in my next talk. See where all the latency goes. That's a really good idea. So let's talk about number representation. A lot of us don't really think that hard about numbers. We let our computer worry about how we map bits onto the real number line. But it turns out it's a really essential task if you want to do efficient deep learning. There are a bunch of ways of representing numbers. Integer, we have mantissa bits, we just raise two to each bit position and add it up. For floating point, we do the same thing for the mantissa but then multiply it all by two to the exponent, so we get the scaling factor that can move us up and down the number line. One of the most efficient natural representations is the log representation. I'll go into that more in a while. If you're especially at low bit precisions, if you want to make every symbol count, what you want to do is use a codebook. I want to take, say, 8 bits of symbols, look up in a table of 256 numbers. Now I can basically take each of those 256 numbers and put them on the number line where they will do the most good. Some people like to try to mimic how biological systems work and they use a spiking representation. It turns out that in CMOS, this is almost the worst thing you can do because CMOS dissipates power when you toggle a signal. Say I want to represent a number between 0 and 256. The average number would be 128, so it would take 128 toggles to represent it.

With an eight-bit integer, on average four of the bits will change state, which is equivalent to two toggles. So it's 64 times as much energy to represent that in a spiking representation as an integer. We've played along with analog. I get very excited about analog because the individual operations in analog are essentially free, right? If you have a voltage and a conductance, multiply just happens and it turns into a current. The problem with analog is that while the operations are really cheap, there's no really good analog storage and it's actually very difficult to move with any precision an analog value from one place to another. So you wind up having to convert it to digital, and as soon as you do that, it turns out you lose. It actually is far less efficient than doing things digitally. So, how are we going to compare these number representations? Really two questions. What does it cost? Here there's operation energy and movement energy. What does it take to do a multiply accumulate? That's the operation energy. And what does it take to read it from the memory and get it to the multiply accumulate? Well, movement energy is just the number of bits, right? You're moving it across a wire. The memory and the wire don't care what those bits mean. They're just moving bits. And the operation energy really matters a lot what the bits are. So let's compare some of these things. Here's a comparison of spike, which I already kind of explained why it has really bad energy, but this is actually showing equivalent to integer representation power. The log representation, the floating-point representation, and the integer representation. What you see is both log and floating point have really good dynamic range. Integer not so much because it's spreading things uniformly across the number line. What really matters for accuracy here is described by error. Further to the right is better. For a given number of bits, log has the best accuracy and floating point, and integer is actually pretty bad. Now the best you can do is symbols. This is a figure from a paper that Song Han and I wrote at ICLR — actually was a best paper at ICLR in 2015 — where we talked about compressing the weights into a symbol table. What was great about training neural networks, you train them with backpropagation. Once you've computed partial derivatives, you can train anything, including your symbol table. So we actually just initialize the symbol tables randomly and then backpropagated into them and fine-tune the weights. It turns out that you had to play some games to avoid having everything clustered together. But as long as you did an appropriate clustering algorithm, we wound up getting near optimal symbol representations. This was a network where we also were doing pruning. So we were eliminating the weights that were near zero. Anything near zero we clipped to zero and took away. The distribution of weights wound up looking kind of bimodal, as opposed to just being a Gaussian. If you're trying to come up with a representation with 16 symbols to represent this, if you used integer, which is shown here by the X's, you'd be wasting a lot of your symbols because it's evenly spaced on the number line. You only have seven of your symbols where there's any density in the distribution you're trying to measure. Training this representation with the same number of symbols, which is the red dots, you're basically sampling the important part of that curve much better. This is the network that's not pruned. I forget which layer of which network this is. It's kind of a Gaussian curve centered on zero. If you want to represent the whole thing, you're going to be wasting a lot of your representation out here in the tails. I'll talk a little bit more about that later. But let's talk about the log representation since it's one of my favorites. Those of us who were old enough started doing computations used things that looked like this. It's a slide rule. What it's really doing is multiplication by converting to logarithms and then doing an addition. That's the real power of the log representation: it turns multiplies into adds. We're not going to talk yet about what it turns adds into. It's great. If you compare the dynamic range and worst-case accuracy to INT8 and FP8, it has much better dynamic range and accuracy than INT8. It's got comparable dynamic range to FP8 and about 30% better accuracy. I always find it good to view this graphically. The reason the log number system is a great representation is that the error is proportional to the value. On the left is log. This is basically a four-bit log representation. When the numbers are small, the quantization error — the size of these steps — is small. It's 9% across this entire range. Whereas with integers, the size of the error is the same everywhere, so it's enormous. When you're trying to represent one and a half, your error is a half. It's 33%. And it gets smaller proportionally because the absolute value of the error is the same whereas the number you're comparing it to is getting bigger. Now log is a great representation. One of the reasons it hasn't caught on that well is because floating point is almost as good. Log is on the left here. The quantization error, the size of these steps, is exactly proportional to the number. With floating point, it's sort of like halfway between the two. You start out with a small error and you keep that for a couple jiggles and then you make the error twice as big and you keep that for a couple jiggles, and so on each time you step the exponent one. So you wind up with the same dynamic range but with a little bit bigger error, but not nearly as bad as integer. Now what has kept people from really embracing logs is the adds. To multiply two log numbers together, it's easy; you just add them. That's a multiply. You're adding the exponents. If you want to add, you have to convert it to integer, which requires a table lookup, do the add, and then convert back. Think of our log representation as having an integer part and a fractional part. These four bits is just two to that value, so you're just doing a shift. But these three bits, you're doing a lookup because you have to look up what is two to the 1/8, two to the 1/4, two to the 3/8, and so on for each value of those three bits. That's expensive. It would be very expensive if you had to do it for every add. But think about what you're doing in a typical deep learning accelerator. You're doing a matrix multiplier or a matrix-vector multiply. You're doing a bunch of long dot products — a thousand or 10,000 elements. So what you can do is, say, I'm going to add 10,000 things. I'm going to sort them so that those that have a fractional part of zero, I'm going to sum them all here. I'm basically binning it into eight things. If I had a thousand things, I'd have 125 of each of these on average. They don't have to be equal. I basically partial sum these, then I do a single lookup of two to the 0, two to the 1/8, so on up to two to the 7/8, do that multiply, and then sum the eight groups together at the end. This is from a US patent on how to do efficient deep learning with log numbers. You can basically make the adds palatable by postponing the bad part of the add and doing it once for 100 or a thousand numbers instead of for every add. What happened there? That's not good. Hello. Okay, there we go. I must have touched something. Okay. The next thing that's really important in our journey to better number representations is to understand what numbers to represent and what numbers to throw away. Whatever number representation you pick — integer, log, floating point — pick the range optimally. What do I mean by that? When we started out building these machines for deep learning, we decided we have to represent every weight. If I have some weights that are way out here at minus 0.8 and 0.8, I would take my 16 numbers that I can represent and spread them on the number line like this, which gives me a really large amount of quantization noise and absolutely no clipping noise — the noise introduced by throwing numbers away. It turns out that if what you want to do is minimize your error, you're much better off representing like this, where my quantization noise gets really small. I'm clipping some numbers down to the highest representable number, but there aren't very many of those, so the amount of noise I'm introducing is small. In fact, you can pose this as an optimization problem where what you want to do is minimize the mean squared error of the representation for a given number of bits. You wind up with this integral which is hard to evaluate, but also you wind up with this iterative representation which is easy to evaluate and can be done in hardware very quickly. That's important because for the weights you can do this offline before you start the inference process. But for the activations, every time you produce these activations you have to decide what the scale factor is and how to clip them. The great thing about this is it winds up being worth almost two bits. Let's look at one of these lower ones because I can point at it. Those are too high. Let's say I start out with a four-bit representation and I'm not going to clip. The clipping scaler here is the largest number I can represent. So here I have a lot of quantization noise, no clipping noise. As I move to the left, I'm saying everything above three, I'm going to clip to three. Everything above two, I clip to two. I'm reducing the quantization noise by more than I'm introducing clipping noise until you get to the minimum. That's when the quantization noise and the clipping noise are the same. That's where I stop. That's this optimal point. If I go over here, I see that with four bits I'm getting almost the same mean squared error that I was getting with six if I didn't do this. This is one of the things that's allowed us to scale down our number of bits without really losing precision.

That is a really good question. My answer is I don't know. But it would be interesting to try to test that in some way by putting yourself at the minimum error point, seeing what accuracy you get on the actual benchmark, and then dithering one way or the other and seeing if the accuracy improves or not. It's much easier to minimize error than to actually do the computation required to compute the accuracy. But it'd be interesting to try that. It's a good idea. So the next thing we realized is that this whole issue of scaling and clipping can be done at different granularities. At the time we started this, we were doing it at the granularity of a layer. An entire layer of a neural network, we would pick one scale factor and clip everything optimally to minimize the mean square error for that scale factor. But then we asked a question — this is a paper we had in MLSys in 2021 — what if we did this at a smaller scale factor? For example, if every 64 elements of the matrix we took and scaled, now instead of having to take a huge distribution and scale it, we can take a smaller distribution and scale it. This wound up sort of buying us another bit. Actually, in the representation called NVFP4 in Hopper, it's FP4 grouped into vectors of 16 with an FP8 scale factor. Something I've been working on for a long time is sparsity. Song Han and I wrote this paper at NeurIPS in 2015. We showed that you could take a typical neural network of the day and for an MLP, lop 90% of the weights out of it and lose no accuracy. It was kind of an interesting result, and we had real trouble converting that to actionable advantage in hardware. The reason is when you're doing a dense matrix multiplier, it's a super nice regular operation. The numbers march by each other, you do the multiplies and adds, and answers fall out. It's all very regular. Once you start introducing sparsity, it becomes very irregular, and that irregularity introduces a lot of control. The thing that finally wound up working after many failed attempts was structured sparsity, which we introduced in our Ampere generation. The idea is we'll take a dense trained matrix, force the low values to zero with the constraint that out of every four elements at most two can be non-zero. We can squeeze it down so all the non-zero elements go away, add some metadata to say which ones they were. We use that metadata as a select input to the multiplexer feeding the activations in, so we feed the activations that correspond to the non-zero weights in, and it becomes a very regular computation. You're simply doing a select on a multiplexer based on that metadata. This works out very well, but I think there's still a lot of gain left on the table because we're doing only 2:1 and only for weights. I think one of the areas we have to mine a lot better performance in the future is doing sparsity on activations as well and doing more aggressive sparsity than just 2:1. Let me move on to the next step here because I see my timer counting down. Let me talk a little bit about accelerators. One way we prototype ideas for how to make deep learning better on GPUs is we take the ideas and put them in accelerators first. We've built a number of these over the years. These are some chips we've done in NVIDIA Research. Accelerators get their gains by a bunch of advantages. The single biggest advantage that accelerators get we've already taken care of with GPUs: the notion of instruction overhead. I said early on that we had this 20x overhead for GPUs doing a single floating multiply-add. This is a figure I pulled from a talk about CPUs, which are way worse than GPUs. If you're doing a 16-bit integer add, which is a 32 femtojoule operation in 28 nanometers, and you compare that to what it takes to do the add instruction on an ARM A15, this overhead would be even larger on a modern ARM Neoverse or on an AMD or Intel processor. It's four nines of overhead. So you're doing 10,000 times as much energy in the overhead as in the operation. It's only 20x on GPUs, and with the matrix multiply instructions, it's a few percent. Eliminating instruction overhead is probably one of the biggest things we get out of building accelerators. The other is we can pick operations tailored to minimize energy. This shows what I said earlier: the cost of doing arithmetic goes linear with bits for add but quadratic with bits for multiply. What I didn't say is that reading a memory costs more than any of that, and going across wires is even more expensive. One of my basic principles is every time you go out of a realm of locality, it's an order of magnitude more expensive. If I'm reading from a local SRAM, that's like five picojoules per word. If I read from an SRAM somewhere else on the chip, that's 50 picojoules per word. Don't think of that as memory energy; that's communication energy. I build my SRAMs with small macros that might be eight kilobytes a piece. If I'm building an on-chip SRAM that's megabytes, that's a whole bunch of these little 8KB SRAMs that cost five picojoules per word to read, and then it's 45 picojoules per word to get it to where I need it. If I have to go off-chip, it's another order of magnitude to read DRAM off-chip.

Yeah, the problem — and it's actually not cool — you want a big HBM machine. We actually want a big machine which is a bunch of SRAM with math units sitting right at the edge of the SRAM. It's a question of if it fits and how many chips it takes to fit. But the problem is if you provision for one phase, like the decode phase, you'll wind up being very RAM-heavy and math-light. Then during the prefill phase, you actually want a lot more math because you're doing many tokens at once with the same weights. You would provision way more math and less RAM. So you can't balance simultaneously for all parts of the workload. That's why we recently announced a CPX part, which is basically a prefill-optimized GPU. You can imagine that you can build decode-optimized GPUs. You may have to specialize the engine for different things to achieve that balance.

It's exactly the same as the GPU. I can balance the GPU and you can balance the accelerator. It's how much math and how much memory you provision, and how much bandwidth. Those are the three variables: how much math, how much memory, and how much communication bandwidth. I'm running out of time, so I'm going to skip over this slightly. One of our more successful accelerators is Magnet. This is the diagram of it. I'm not going to go into a lot of detail, but we basically use these to see the speed of light in terms of ops per watt. What you can see is doing some aggressive voltage scaling here, we're at about 100 teraops per watt doing deep learning inference on this particular accelerator. We use that as a bar to say can we get our GPUs to achieve that by taking the techniques we implemented in Magnet, many of which found their way into Blackwell. I should say that voltage scaling is a huge opportunity if your goal is to optimize performance per watt. There's this conflicting set of goals: you want to optimize performance per watt, and you also want to optimize performance per square millimeter. If I want to optimize performance per watt, I should not have stopped at 0.5 volts. I should have set really low threshold voltage and scaled down to like 0.25 volts, because I'm winning quadratically as I scale the voltage. But I'm also increasing the area I require to get a certain arithmetic performance linearly. So I'm getting more back on energy efficiency than I'm paying on square millimeters, but I am making my performance per square millimeter worse. There are some very clever techniques with this thing called voltage stacking. Because it's very hard to regulate down to 0.25 volts, what I would do is regulate to one volt and then take my matrix multipliers, divide them into four units, and stack them on top of each other so that the ground of one unit becomes the power supply of the next. Then I have to make sure they dissipate exactly the same amount of current to keep everything balanced. That can be done, and we've done some prototyping of that. Let me move on. One of my visions of accelerators in general — I build accelerators not just for deep learning but for bioinformatics and for SAT solving and some other things — is that it turns out 90% of the problem is the same every time. You have to build a memory system with both off-chip memory (HBM or GDDR, LPDDR) and on-chip memory. You've got to build communications off to the off-chip memory and on-chip network-on-chip communications off to the rest of the machine. Then you need some general-purpose computing. Then for whatever your application is, there's a tiny little piece you do. When one of my PhD students and I built a bioinformatics accelerator, the part that was the core of the dynamic programming for the Smith-Waterman algorithm was like one page of Verilog out of many hundreds of pages. So what you'd like to do is have a system — it's like a GPU — that has all these common elements: the general-purpose compute, the memory system, the interconnect. And then have a programming system where you say, 'Here's my application,' and the mapper decides how to take that application and map it in space and time, how much of that mapping goes to general-purpose compute blocks (the streaming multiprocessors of the GPU) and how much to custom compute blocks. With modern chiplet technology where we can stack those, you can imagine having one base layer GPU that can be specialized for many different applications by stacking those on top. That's just kind of a vision I have, not a vision the company has. Let me wrap up here, and maybe there's some time for questions. I think I'm over the hour already. I think deep learning really is improving the human experience, and we're only at the beginning of that. I think most of the real gain that's going to happen in the next few years is taking the models of the quality we already have and applying them more effectively to improving productivity in various human endeavors, just using them better. But at the same time, we are going to build better models. I think a lot of the current challenges we're facing — this disaggregated inference where the prefill phase is math-heavy and the decode phase is memory-heavy and completely latency-limited — is causing us to think hard about how to do our architecture. What's driving the latency problem even harder is this notion that we're going to do chain of thought and tree of thought. We're not going to run the LM once; we're going to run it tens of times for every response to the user. People are coming up with new models every day, particularly new attention mechanisms. I think that's where I see most of the creativity coming. Right now when you do attention, it's quadratic. I have to have every token I produce attend to every token in the whole input prompt and every one I've produced since I began responding. That doesn't make sense. Most words don't depend on every other word; they depend on certain keywords. There's a huge research area in pruning the attention, coming up with sparse attention, so you can get a scale factor for that attention that's less than quadratic. Deep learning was enabled by hardware. We had the algorithms, we had the data. It wasn't until GPUs came along that we allowed those two to be combined. Since then, we've been scaling at this breakneck rate, 16x per year in the transformer era, 10^7 in a 12-year period. To keep up with that demand, we have scaled the inference performance of our GPUs by 1000x in the last 10 years, 5000x in 12 years. 3x of that was from process technology; the rest was number representation, complex instructions, and sparsity. To get the rest of the 10^7, we have done scale-up and scale-out. You now need 10,000 to 100,000 GPUs to train a large model. It's not just hardware; matrix multipliers are easy to build. To get usability and performance requires very optimized software. I've given some details about some of the optimizations we've done in number representation by doing an optimal trade-off of clipping noise to quantization noise, doing that scaling on a relatively short vector — we wound up with 16 elements in Blackwell after doing a bunch of studies — and then exploiting sparsity in a way that can be regularized so you don't pay for the irregularity. With that, I'll show you a picture of where I live and I'd be happy to take questions.

Interconnect is critically important. There are three ingredients to building any engine for AI: math, memory, and interconnect. The interconnect is important both to provide enough bandwidth to take the problem and distribute it geographically, whether it's on chip or across multiple chips, but it's also a big driver of latency. One of the things we're trying to do is get closer to what's possible. If you look at our latencies going through NVSwitch and NVLink, it's on the order of many hundreds of nanoseconds per hop. What's possible is maybe 100. So there's big opportunity to go further. You mentioned NVQ link; that's a technology we're providing for people doing quantum computing. It's a step, but what's actually really needed for quantum computing to scale is a way of having qubits in two different boxes that wind up being entangled, so the fate of one is determined by the fate of another. That's something that we don't know how to do yet, nor has anybody else really tackled.

Actually, for the pie chart I gave you, that was from the Magnet chip. We were cutting the math operation energy at the input of the math unit, so we'd already accounted for the communication energy separately. That accelerator was very carefully optimized so that most of the numbers being presented to the math units were coming from very small memories sitting right at the inputs. But in the general case, you're right; a lot of the energy is moving the data.