Chief Software Architect, Nvidia (formerly Groq CEO)
CUDA: New Features and Beyond | NVIDIA GTC
The CUDA platform underpins GPU computing, providing the libraries, compilers, runtimes, and programming model used by ...
About Jonathan Ross
Jonathan Ross, founder of Groq and inventor of Google's TPU, has been discussing the company's near-collapse and subsequent partnership with Nvidia. He described a period when Groq was about three weeks from running out of money and considered layoffs, but instead implemented a salary-for-equity exchange called "Groq bonds," which he said 80% of employees participated in, with about half reducing their salaries to the statutory minimum. Ross stated that this move saved the company. He also discussed a $20 billion partnership with Nvidia, describing it as combining GPUs and LPUs for better performance, analogous to using both 18-wheelers and delivery vans in a logistics network. Ross has spoken about the future of AI compute, stating that "as long as there are unsolved problems in civilization, we will have a need for more compute" and that "there's no way to satiate the appetite for intelligence." He has advocated for revamping education to focus on asking questions rather than providing answers, saying "if you can come up with the right question, AI can go answer it for you." Ross also distinguished between intelligence and sentience, defining sentience as "your rate of improvement in your intelligence" and suggesting that AI is creating a feedback loop that accelerates societal intelligence.
Transcript (71 segments)
J
Jonathan Ross0:09
Hi everyone. Thank you for coming all the way over across the road to come and listen to me speak about CUDA. I talk about this every year, and there's always so much to say, and as always I'm standing here representing hundreds of people who've done an incredible amount of work and amazing things. So, you know, I'm the spokesperson rather than the person behind all of these things.
Typically with these talks, I start by presenting how I think about, how we think about, the landscape and the nature of GPU computing. That helps tie it into where we're going. I talk frequently about parallel programming and parallel algorithms. Obviously, it's what I do.
This is actually a slide I used at a talk a couple of years ago to talk about the way that different forms of parallelism get exploited in GPU algorithms in deep learning, in scientific computing, and so on. But I don't want to talk today about parallel algorithms. I'd actually rather talk about how they run. Those parallel algorithms have to run on systems that are huge, huge, like this. So it's not just a matter of how you break your program down so that you can divide your data and work on it together. It's also how you run this data, how you run these programs, where you run these programs, and so on.
A microcosm of that giant data center can be thought of as the GPU. This is the Blackwell GPU right here. It has 160 SMs. It has hundreds of thousands of threads. This alone can be thought of as a supercomputer. I looked it up; it's as big as the largest supercomputer in the world from 2004. So it's a really big powerful thing in one chip. And of course, these data centers are up to 100,000 or more of these things these days.
What I want to talk about in terms of parallel execution, parallel algorithms, is something called symmetric parallelism compared with asymmetric parallelism, because I see a sea change coming in the way that programs run on these systems. Symmetric parallelism, I'm defining, is when a parallel program fills an entire machine and every single core of that machine, every single node, is really running the same thing at the same time. Asymmetric is, of course, the converse where you're running multiple different things.
So in CUDA terms, CUDA has had symmetric execution for a long time. We have what's called a grid launch in CUDA where you take one program and you run it on all of these SMs, the 160 SMs in the case of Blackwell Ultra. You run the same thing everywhere. This is this very symmetric mechanism. If I'm running some sequence of work A followed by B followed by C, then first I run A and everything is running A, and then B runs, and then C runs. It's symmetric because everybody's running the same thing. It's also sequential because I'm doing all of my A processing before my B processing.
Right now, CUDA has offered almost since the beginning the ability to break this up and opportunistically get more done. Sometimes the GPU has elements of it which are idle. We have a thing called CUDA streams which allows you to say, here is lots of different work, you can pick and choose what you want to run, and you can create this sequence of operations. Several years ago, we took that one step further to define a task graph which encapsulated all of that work in a single workflow and says, here are all of the things that are going to run and how they're going to run. But all of these things do not necessarily fill the whole machine.
I can look at this stream and say, well, take a time slice through this, and it appears that at this particular time I might be having some A's and some B's and some C's running on my machine. But the nature of the way the GPU runs work is that the streams merely describe what can be done. If A3 jumped in first, it might have filled the whole system and you would not be having B2 and C1 running alongside it. So even though the parallelism is asymmetric, it's not really a guarantee, it's opportunistic.
One of the things I want to talk about is this move to requiring more of a guarantee. This move is happening because an enormous amount of performance and optimization opportunities are available if you can describe your work in this way. What I have here is a very figurative image of an inference workload. Typically you think of this as a production and consumption style. You've got a prefill where you take in the query. You take in the request from the user and you look at your giant amount of data and you process it to create the context, to create the thing that the decode is going to work with, and the decode loop iterates rapidly generating tokens.
Now, I stole this particular picture from our friends at Dynamo, but it's a beautiful description of where the difference is between symmetric and asymmetric parallelism and where things are moving these days. On the left-hand side, what you have is traditional serving. You have a prefill operation and then you have the decode operation generating the tokens after it. The challenge that you have here is if you look at that picture on the left-hand side, the prefill is very compute-bound. It takes an enormous amount of compute work. It's doing a lot of matrix operations. It's doing matrix-time-matrix-time-matrix, and so you need massive compute horsepower.
Whereas on the right-hand side, the token generation, the decode component, doesn't need lots of compute horsepower. What it needs is lots of ability to iterate fast through memory. It needs a lot of memory bandwidth. So the way that you would provision or configure these two systems is quite different. The idea on the right-hand side is to say, well, let's not try and put this all in one system sequentially. If you remember my ABC from before, let's intentionally run both of these at the same time configured somewhat differently. The benefit can be huge, can be a factor of 10 or even more in terms of speed-up to being able to do this, because now you've right-sized both of the pieces.
So in terms of how this would run, I've got my prefill worker running on one set of GPUs and I've got my decode worker running on a different set, and that allows me this configuration control. Now, all of this needs to be overseen by an enormous amount of complexity in the system just to manage what's running where, because these workloads are very unpredictable. They move around, they change, you don't know what the query is going to be, there are a lot of different things going on.
This orchestrator, I've filled in a Dynamo system there which is NVIDIA's orchestrator of this kind of disaggregated system, its job is central control. Its job is to manage what runs where and to do it dynamically. It's going to spin up these inference engines that are going to run prefill workers and decode workers and all of these things are going to be delicately choreographed together. What's interesting is it's going to depend on the workload exactly how this gets orchestrated.
If I've got a very context-heavy workload, if someone says, here is Wikipedia, summarize it for me, that's a massive amount of data. I'm going to be very heavy on that prefill section, on the blue section. But if instead I have something where I'm reasoning and I'm generating tokens and feeding it back into itself, I'm going to be wanting token generation prioritized. So what you end up with is this very careful dynamic control, very problem-specific, even dynamic within a problem, to control that balance between production and consumption. The goal is to not starve any component, that everything is moving at full tilt all of the time.
So let's bring this back to the single GPU, because as I said, the single GPU itself is a massive, massive system of 160 SMs, and I can think about this in the same kind of way. I can think about how can I take this GPU with teraflops of performance and hundreds of SMs and hundreds of thousands of threads. You can easily imagine that partitioning that up in a similar way, dedicating pieces to specific tasks, could have the same kind of benefit that the big system disaggregation would as well, except in a much smaller context. While you may not necessarily be able to run an entire AI model in the memory of one GPU, there are lots of smaller jobs where this kind of partitioning is really interesting.
Now, the challenge for the GPU has been that even though we have a lot of multitasking mechanisms and they've existed for a long time—streams on the left for CUDA which I mentioned earlier have been around almost from the very beginning. On the right-hand side I have what we call the Multi-Instance GPU that has hard partitioning but it does not have the dynamic-ness that is needed in these kind of workloads. In the middle we've got the Multi-Process Service which has done a lot of duty in this regard. It gives me the ability to partition but it doesn't quite give me the dynamic-ness. There have been certain restrictions there which make it difficult to get this guaranteed asymmetry that I'm looking for, this level of control coupled with this level of partitioning.
What we've done over the last couple of years is we've built this thing called green contexts. Green contexts is a mechanism which sits between streams, because it's within a single process, within a single application allowing dynamic action responding to what's going on, and on the other side the Multi-Process Service which gives me the partitioning which gives me the breakdown of resources that I need. It fits neatly in between with that ability to be a single application, a single controller like that Dynamo controller was, orchestrating my program and dynamically assigning resources to what I need as against having separate processes which is more about isolating different things and running them together on the GPU.
Here's a quick 'why, how would you pick between them,' but I really want to talk about the green context, because I think this turns on a new ability which really looks to this future of guaranteed asymmetry, of guaranteed parallelism that I was just talking about. To create a green context is pretty straightforward, even though I have a number of steps here. You find out how many SMs are on the GPU, 160 in this case. You decide how you're going to partition them and then you create a set of descriptors for these different partitions so that you can then use these descriptors to assign these green contexts to them. I've got these descriptors here so that you can do nesting of contexts and other such things like that.
Once you have these descriptors, once you've decided on your partitioning, you create the green context. A context is just a little bounding sandbox in which something is going to run. You create a CUDA stream from it. If you're a CUDA developer, you'll understand that the stream is the unit by which I submit and dispatch work. Then, very importantly, you just launch kernels into that stream normally. This is really the magic in the green context. I can manage the execution, in other words the layout, the mapping, the partitioning, separately from the individual work. It's the same as having a controller managing what's running and the things that are running which are just going ahead doing the thing.
I can create one of these streams. I can launch a BLAS operation or an FFT or some other program and it will fill half or two-thirds or whatever of the GPU, completely oblivious to the fact that I've given it a smaller amount of resources. This gives me the ability to do the multiplexing that I am looking for. All of this comes because it's a part of CUDA, with developer tools which give you deep insight into what's running where and how your program is operating, because honestly if you're doing this kind of parallel program control you've got to be able to inspect what it is that you're achieving so you can tune, you can optimize, you can see where problems arise.
This ability to dynamically control what runs where without having to impact the work that I'm actually running gives rise to a lot of different patterns. This is four patterns I draw on a slide. You could easily imagine another dozen patterns that you could come up with. But the key part here is there are all these different things you could do with it, because now that I can say what runs where and what overlaps with what, it opens the door to things. Low-latency reservation is a common thing, it's an improved form of priority where I've guaranteed that resources are available. Overlapped execution, nested contexts, these, but really the thing I want to talk about most is dynamic partitioning, because that's the thing that points at this disaggregated workload that I've been talking about: the ability to control what runs where.
For our asymmetric parallelism discussion, the ability to control it is key. It's the key thing that the green context is doing for me. The way that this works is that these different streams that I have, anything put in that stream runs only in the sandbox of the green context in which it's been targeted, from which the stream has been created. So my stream A is living in box A, my stream B is living in box B, and my stream C is living in box C. It seems very straightforward and obvious, and I hope it is, because the best engineering is invisible like that.
Another thing you can do with CUDA, I mentioned graphs before, you can capture the work of a stream into a single graph. In the same way that a single graph can span multiple streams, if those streams now target individual green contexts, I now have a graph that captures work that runs in different places. This is key. I can define a single workflow that spans all of my resources and that I can launch from one place. Now we're really thinking about this orchestration, about this control, the centralized control again.
This is actually not new for graphs. You've always been able to create a graph that spanned multiple GPUs in a system in one node, because those GPUs are connected by NVLink or PCIe. It's funny because we've had graphs for quite a long time and when I mention this, people say, 'Really? Graphs can run on multiple GPUs?' The answer is yes. It's always been able to do that. If I have a graph like this A, B, C, I can have A and B and C be different GPUs as well as different green contexts. The scale doesn't really matter. The graph just describes where things want to work.
Of course, if I look a little bit ahead here, that single node is almost always part of a cluster. It's part of a rack. It's part of a system. You can easily imagine that if I've got NVLink-connected GPUs that I can have a graph spanning, then maybe I want a graph to just span my NVLink 72 rack. Why not? It seems pretty obvious. It's not rocket science to imagine that this could happen. Now this is not something you can do today. I'm looking ahead to things that we would like to do, because I want to tell you about, well, if we would like to do this, what are the things that must exist?
If I have a graph that can span all of the 72 GPUs in one of my NVLink 72 racks, well then why wouldn't I want to span multiple racks? Or why wouldn't I want to span an entire data center? It seems pretty obvious to me that you'd want to determine what runs where on every GPU. I want to be able to define a workflow and assign different GPUs and resources. Think back to my picture earlier of what Dynamo was doing and say, 'Here's a bunch of these prefills. Here's a bunch of these decodes. Manage them in these different ways,' and having a vocabulary, a way to say this.
CUDA is not going to make the decisions that Dynamo makes because it doesn't know enough about the system. But giving it the tools so it doesn't have to do so much of the legwork itself, that's really interesting. To make this what seems a pretty obvious thing work requires actually some very sophisticated things under the covers. That's what we're thinking about at the system software level. There are going to have to be some core multi-node things that CUDA is able to do in order to enable this fairly simple and straightforward picture of a graph of work spanning all the GPUs that I want.
Excuse me. The first of these is naming. This is a quick aside. By the way, this is the worst picture I've ever drawn in PowerPoint. It's a 36-node radix-4 dragonfly network. I asked an AI drawing tool to draw it and it came out with something comedically ridiculous. So I was forced to draw this by hand. There's a lot of lines and clicking and things involved here. But it is important because when I've got a big system, everybody has to agree on what node B3 is and F2. If I say run this on D1, everybody's got to agree what D1 is. If I say move my memory from here to there, we've all got to agree on the naming. Topology and naming are fundamental.
It's also extremely hard to do in a consistent way unless you're the system software layer, because CUDA is running on every single one of these machines. CUDA can sit here and say, yes, we all know who F2 is. But that's really difficult to do if you're ten different applications all enumerating it differently. So these are the kinds of things that need to live at the system software layer.
Now, we already talked about multi-node execution. Obviously, I want a multi-node CUDA graph to be able to be launched from a single location but span GPUs anywhere in my system. But it needs a little bit more. A CUDA graph would need, in this scale of context, to be able to make a few additional guarantees, things that we really care about for being able to do our guaranteed asymmetric parallelism that I started talking about before. I want to know not just where things run, but when they run. I need to be able to say, 'These things are going to talk to each other. Make sure they're running at the same time.' Or, 'This one over here has to start before that one over there, because this is going to set something up that that thing's going to use.'
In a way, what I've got is a graph which is saying what we call a control dependency today is now acquiring a new set of dependencies, a new set of constraints which say, make sure that A3, B2, and C1 do run together. So I do get that picture that I was showing you earlier guaranteed instead of opportunistically, because that's extremely important at the kind of scale that we're talking about.
The final thing which is hard to do, and we're still very much thinking about these things—these are not things that we have in CUDA yet—but I wanted to share with you the kind of ways that we're thinking about this, is memory management. Memory management is obviously a system-level thing. Memory management and allocation live in your Linux kernel or your Windows kernel. It's not something the application takes care of because it is a global thing and it ties into naming and all these other things. The way we think about it is going to be a little bit more complex than just 'allocate me a pointer' because you need to think about more than that.
First, you want to say, well, here is a thing, my flower, let's call it a flower. Everybody agrees that this is the flower no matter what node you're on. The way that you break your work down is up to you. I don't know how to break it. Maybe you want squares. Maybe you want long things. I don't know. You're going to figure out based on your problem how you break it down. So you have to decide how the flower is spanned and allocated and arranged, and then you have to decide who gets to see what bit. This is a really not obvious thing.
But if I have a machine of 100,000 GPUs, which is a very plausible size these days, then if I make a change and I have to tell people about it, if I have to tell 100,000 people about it, that takes a long time. If I even make an allocation and I have to tell 100,000 things about it, that takes a long time. So instead, what you want to do is constrain and narrow the box of who's talking to whom, who can see what. That gives you a lot of opportunity to performance-tune your work. Then finally, the bottom line of all of this is once you've done all of these things, you get a pointer to your data and you just read it and write it.
In the same way that in the green context, a kernel running inside a green context doesn't know and doesn't care that it's in a green context, here we would make the decision that the granularity of multi-node CUDA is not one grid spanning many GPUs; it's different kernels across the system. The kernel itself is local, the grid is local, and the graph covers the entire system. That's a key decision point for us, because it means that if I look at the picture of what these stacks look like these days—a layered multi-node system software stack where everything, in this case vertical stacks on every single CPU where the orchestrator is managing everything—it looks all pretty here, but it's not really pretty. What really happens is that everybody bypasses everybody else and they're all talking sideways and up and down and it gets really crazy because everybody's just trying to hack things together right now.
The idea is that if I can insert a layer, a multi-node CUDA driver underneath, which gives you the naming and gives you the ability to manage your memory—we're not going to make the decisions for you, we're going to give you the tools to do what you need to do—then the idea is that you generate interoperability here. You make sure that everybody is now talking about node F2 being the same node. Ultimately this fuses down into just CUDA. CUDA becomes multi-node CUDA.
So this is where we're looking. This is CUDA and beyond, and this is the 'beyond' little part of the talk if you like. But I wanted to really let people know how we're thinking and have you think about it yourselves in terms of how this works with the kind of things that you're doing, because honestly nobody runs anything on a single-GPU machine anymore.
Now, it's not like we've done nothing in CUDA in general for multi-node and large systems. I talked about Dynamo already, which is a product that NVIDIA puts out that manages all of this kind of stuff. Underneath Dynamo are other things like GPU Direct Storage, something called cuFile as the API for that, and it allows Dynamo to be very specific and detailed in how it controls where the different KV-cache locations are stored. It also allows for low-latency transfers directly between the GPU and the storage.
We've also got a checkpointing mechanism. This is one of those really interesting things when you build something in engineering and people use it in ways you didn't expect. It's always fascinating. Here it turned out that checkpointing—there's this great talk by Modal which I've referenced at the bottom, I link talks at the bottom of these slides by the way relating to any of the things that I'm talking about—the idea of using it for elasticity instead of resiliency was a really interesting thing that turned up, and it turned out to be a very powerful thing to do.
Finally, of course, developer tools for this. We have to be able to investigate and inspect what's going on. Nsight has cloud development tools for managing this kind of thing as well. There's a whole website for this and talks coming up on that later this week as well.
So I'm going to switch gears a little bit, but it is related and you'll see how it ties in. Last year we announced CUDA Tile, and in December we launched CUDA Tile. CUDA Tile is an extension to the CUDA programming model to describe in more detail what you're working with in terms of arrays and tensors. You're telling the system more about your work and that turns out to be very powerful.
I'm going to look back at Dynamo and disaggregated inference for a moment, because the inference benchmark suite, something like InferenceX which sits on it, there's a whole layer of things all the way down through this. One of the core pieces of this is called the FlashInfer library. This is the library of highly tuned kernels which do the heavy lifting. These are the things inside that prefill worker and that decode worker box. One of the things we did with the Tile system is we took it and we started porting those kernels that are used in the benchmarks to Tile, because we wanted to move them to Tile, because they're more portable, it's easier to develop them, but also we wanted to make sure for ourselves that the performance capability was there.
What I've got here is a comparison of the pre-CUDA Tile and CUDA Tile implementations. Across the board you're seeing 90-percent-or-better performance. This is only about half of the kernels. We're still working on this. We haven't got it all the way there, but it's very encouraging to us. Here are some really interesting and great performance numbers to say that this tool could be beneficial to us because we can get high performance and all the other things, and let me tell you about all the other things.
What we did was we picked a couple of the kernels out of this and we said, let's take these kernels, because one of the things that is a key cornerstone of the Tile model is that the code that you write is portable across all these different GPU architectures. What we did was we took a couple of those kernels, a multi-attention kernel and a batch matrix-multiply kernel, and we ran them with no code change on Ampere, Hopper, Blackwell, RTX GeForce, and on the DGX B200, and what happened was there's an autotuner which picks a different tile size. Blackwell's a bigger machine than, say, B10, which is the L20. So the Blackwell tile is bigger than the B10 tile, but without change, the kernels just moved from GPU to GPU with pretty much an 80-, 85-percent-and-better performance curve.
On the right-hand side at the bottom there, we're still working on this. We only just released Ampere last week, so we're still tuning some of the Ampere compilation things. But the idea here is, if I can get kernels that are performant enough for that InferenceX benchmark, at the same time I write once and run everywhere. Now you take it from here and you start tuning. Then you tune for your local specific architecture. You start adding hints into your program and other things. You're still doing work to get to that peak 100 percent. But if you're starting at an 80-percent or a 90-percent starting point, that's a pretty good place to begin.
This encourages us enough that we are going to continue internally working with these tools and building lots of our libraries in Tile. I'll show you more about that in a moment as well. Part of the way this works is through this compilation stack where previously you had just purely a thread-level compiler. We now added a tile-level compiler which can do more, and CuTe which is the tensor-core compiler which lets you get full low-level control over the tensors. I'll show you a bit about how the compiler stack has evolved.
On the left-hand side is how the compiler stack traditionally was, which was kernels talking to thread level. We call it SIMT, which is a single-instruction, multiple-thread programming model. On the right-hand side is where we are today. It's a bundle of different things that you can target, because if you target the right thing, if you use the right tool for the job, you can get a lot of benefits, like some of these performance and portability things I was telling you about with Tile. You can pick whatever thing you want, but you should pick the best one because it makes your life easier or more performant or whatever you need to do.
Under the covers, CUDA has always managed portability between architectures through something called PTX. But as architecture-specific tensor-like things appeared, it became more tricky to keep things portable. The Tile layer reacquires that portability story even for the complicated tensor architectures of the GPU. What that means is that my library system looks slightly different on top of them. Where previously everything was bound down to a single thread-level representation of my program, now my library can pick and choose. CUTLASS has disappeared on the right here because it's turned into a compiler. But something like FlashInfer can pick and choose all of the things it talks to.
Now, this is actually really important, because if I can pick and choose what I talk to, we can take some of these FlashInfer benchmarks. In this case, this is the DeepSeek R1 benchmark for prefill on one of the InferenceX benchmarks. Instead of just porting 100 percent of the kernels to Tile—that's that diagonal line bar on the right-hand side, which shows you have the same performance that you had without, either in Tile mode or not in Tile mode—it turns out that if you can pick and choose, if you're sophisticated enough or you want to put the time in, some of the kernels are much more of a win than others. So you can mix and match these kernels by using these different choices of compiler to get yourself, in this particular case, a 17-percent improvement on a benchmark by swapping a particular kernel from one style to another.
That's a huge power to put in the hands of your tuning systems or your ninja developers. Or you can just let the system go and do what it will and still end up with, in this case, an exact break-even performance story except now you're going to be portable.
Internally we keep working on this and one of the things we're coming out with imminently in the next CUDA release is C++ CUDA Tile. We had CUDA Tile in Python first—CUDATilePy in the Python language—that came out in December. We're putting out C++ in the next CUDA release. The C++ story looks very similar intentionally. It comes through NVCC which is our C++ compiler, but we've made sure that the C++ idioms, while looking C++-ish, still have the same language and the same look and feel between Python and C++. I strongly believe that your intuition should just map across no matter what language you want.
A really interesting thing we did with this was we sat down and implemented an algorithm that does not look like the kind of thing you would put in Tile mode. This is a sparse matrix-vector multiplier. You've got to deal with sparsity. It's implementing something called the sliced ELLPACK algorithm. We wrote a kernel implementing this algorithm and we ran it on a test matrix suite which KSPARSE uses. Sparse matrices are all complicated and there are many different categories of them, but in a category of these matrices, our one kernel beat the state-of-the-art KSPARSE library because the compiler could bring a lot more to bear.
We actually looked at this number. We said, 'Wow, that's a lot more than we expected.' All of the things above that yellow line are where my one kernel is doing better than a library of dozens of kernels. We looked in more detail and we found, well, what is different? It turns out that if I take the Tile compiler and I express my problem in this higher-level array-based model, the compiler can do a lot more with it. So I can put one kernel where previously, in this specific case, writing it in thread level would require 12. Something which is 78 lines of code long can perform equally to 1,000 lines in 12 different kernels. That is the power of that compilation stack I was telling you about, the ability to target a compiler when your problem can be mapped onto it. The compiler can do so much more for you. Everyone would rather maintain 78 lines of code than 1,000 lines of code. You'd all like one algorithm that maps and, not only that, is portable between architectures. So this is actually a really powerful thing that we found.
Part of all of these things and figuring out what's going on is, of course, analysis. Tile is inside Nsight Compute, so you can analyze what's going on with your tiles. In terms of a timeline, from release date, from 13.0 which will run Tile code, to the release in 13.1, 13.2 came out last week with Ampere support, 13.3 brings in C++ and Hopper support. You can all flick through these slides. I'm not going to read them all to you. But there's also a 'Connect with the Experts' where you can come and discuss with us what's on the roadmap and what it means to you and how it applies to your applications.
One of the core pieces, though, whenever we build anything in CUDA is this. If you saw Jensen's keynote, he talked about the installed base, the existing pile of things that are already out there, the ability for something new to interoperate and work with something old. We built a quick test example where we're calling into the FFT library, we're calling into our parallel algorithms library CUB—or Cooperative Groups in this case because it's Python—to make sure that bouncing between these two things incurs no real overhead. In this FFT example, the worst case was just a few percent. So the ability to interface and interoperate with things is critical, otherwise your system becomes much more difficult to use.
We also recognized, back to my multi-node theme, that communication is critical in any application now. We took a multi-node FFT, a Fourier transform—don't worry about the code again, it's just broken into a few different sections here in case you enjoy reading code on slides later—but really we're just doing a Y-, then a Z-, then an X-Fourier transform with some communication in between. The communication is the key, because we were able to embed something called NVSHMEM, which is one of our communication libraries, into the Tile program to orchestrate the communication between the Fourier transforms that I'm doing.
The result was actually again a bit of a surprise to us. It beat the existing Multi-Process FFT by 10 or, in the case of 64-bit, more than 10 percent. A lot of that came from the ability to interleave the communications with the computation at the level that I was just showing you. This ability to make a small piece of code—that one piece of code I just showed you which is fit-able on a slide, although not very readable—produce results like this has really convinced us that this is something that we internally are going to be investing in. Libraries are going to be working on interfaces to Tile, programming some of their own stuff up in this Tile format because the flexibility and the performance are there.
Likewise our tensor-core-level programming model which is CUTLASS, which was the third box on my compiler picture for you, also is integrating these communications. The inline communications inside your kernel are critical to being able to manage low latencies and control the fine-grained asymmetric parallelism that you're using. In terms of our communication libraries as a whole, all of them are moving towards device APIs for communication libraries as well as the host APIs that they already have. This is a slide that I stole from my colleague Matt Nicely's talk. He's got a whole bunch of really interesting stuff about that. There's a link to his talk on the bottom there.
In terms of libraries themselves, as I said, we at NVIDIA are saying this is really interesting enough that we are going to start implementing our own things in Tile. We're going to start building Tile interfaces to things. The device extension libraries where you call linear algebra, Fourier transform, compression, all sorts of things inside your kernel, in the same way that communication in your kernel is powerful, so too is being able to call libraries in your kernel. Now the libraries are a complicated thing. We have CPU libraries, GPU libraries, device libraries. One of the things that's really interesting in the move towards Python is that the Python compiler lets us pull all these things together into a single unit which we call nvMath Python that does a lot of the dispatching and heavy lifting for you.
So you would talk to this if you're a Python application. You would talk to nvMath Python and it would pull in the correct libraries and it would do more than that. It would be able to JIT them and manipulate them in ways that I'll show you in a moment. nvMath Python itself is a piece of our overall Python stack, CUDA Python, which we are just about to release at 1.0. That means it's full release. We announced it last year and over the course of this year, we've been working on a lot of different pieces of it and it's now ready to really put that 1.0 moniker on it. There's a whole stack of all sorts of different things here and I definitely recommend my colleague Jonathan Decty's talk because he will go into much more detail about this kind of thing.
The key thing that we've been emphasizing in Python, just like in any kind of CUDA, is peak performance. Python is not normally the thing that appears in the same sentence as peak performance. But peak performance means low-overhead access to these C++-isms or these other constructs like CUDA graphs and fusion. One of the powers of Python is because it's runtime-interpreted, I get the ability to do fusion at the point where my code is called. So I can do fusions between different operators to make single kernels. In fact, one of the things that the nvMath library does is it looks at these calls that you're making. In that bottom left-hand corner, I've got a little number '1' right next to a single instruction which will look at all your types and it will assemble the right call at runtime and just-in-time compile it for you and give you a kernel that runs three times faster because it's able to bundle it all into one piece together.
Along with this, again, I keep coming back to developer tools because without them there's nothing you can do. Finally, we can debug CUDA kernels in Python. We can debug Numba-CUDA, which is the single-threaded compiler for CUDA in Python. You can debug it, which is absolutely fantastic news. You can also bring it into Nsight and so you can start profiling and examining and inspecting your code for Python. That's actually a much bigger deal. If you're a Python programmer, you know how much of a big deal and how difficult tools are. The ability to—you just can't write parallel code without the ability to inspect what's going on.
Now I'm going to take a quick look under the covers of the compiler stack because I want to tell you a little bit about how some of this magic I've just been telling you about goes on. Under that stack, if we expand it a little bit, there are lots and lots of layers. But the layers I particularly want to highlight to you are the dark-green boxes which are the things which I mentioned earlier are restoring portability for even very complex tensor codes. I get portability so I can run that one code on Ampere, Hopper, Blackwell, B10. You saw that graph. The ability to do that comes from these layers in the compiler stack.
But the stack itself is more complex. We've made the stack complex because we understand and we know that the things that we build inside CUDA are very often the targets for other compilers, not just for humans to write code. So we've got MLIR layers and we have, you know, some people target source code and they're welcome to as well. We even have a converter so that if you have Triton code, you do not have to port your code. You can keep your Triton code and plug it into this back end with all the composition and all the tools and all the other things that are available. You can just take your code and make it part of the rest of your stack.
The story of how these things are built is looking at all the different ways people are going to approach it. At the same time, we've been based on LLVM, which is a compiler layer for translating from one compiler format to another if you like. We've been working to take the LLVM compiler layer work, and if you're a compiler person you know what I'm talking about, to put that upstream so that we are always at the top of tree of LLVM. There's always been, up till now, a very big gap between the LLVM that CUDA supported and the LLVM at the top of the tree. Now we are taking our LLVM effort and what we call NVVM, which is the NVIDIA extensions to normal LLVM, and we are putting that upstream. So from CUDA 13.2, I believe it's 13.2, the Blackwell compiler is going upstream into LLVM 21.
On a final note on the compiler, there are other optimizations to be had as well. The compiler is a complicated beast. There are thousands of options and things you can tune and it's difficult for a human to tune it. In fact, it's so difficult, we built a machine-learning algorithm to do the tuning for you. We call it CompileIQ. The idea is that you can iterate on a particular kernel with a particular compiler version and you can let the system go and autotune your compiler settings. The result is that you end up with a file called the advanced control file which you can package along with your compilation so that you can control the NVCC compiler and the PTX compiler to really fine-tune what you're doing.
What I love about this is this is free performance. Meta has been working with us to try this out in their kernels and test it. These are numbers from Meta on their TritonBench benchmark on the left-hand side and some of the Helion work that they've been doing on the right-hand side. The first thing to notice is every single thing is a positive. If you spend the time to do this, you get a speed-up. A factor of five percent or even 10-percent speed-up isn't a massive thing to do just by turning a knob and waiting for a couple of hours for it to tune itself. This is huge. We're using it for our internal kernels. I also did want to show you some numbers from Meta and thank them very much for sharing their numbers with us, because these are real workloads that they're working on in the real world.
Now, a final thing for me to say here. I've talked about tools here all over the place. Every time I've talked about a feature, I've also talked about tools. But I think a tool that you cannot avoid in a toolbox today—if you do not have some kind of AI coding assistant or agent tool, then your toolbox is not complete. Every single one of us, certainly me and all the people I work with, I hope all of you, use these coding agents all the time in your development now. It is an incredible force multiplier. So we have invested in Nsight Copilot which can generate CUDA code. It's an assistant that plugs into Visual Studio as an extension. There's a whole talk on this, so please go and see the talk by some of my dev-tools colleagues.
We've also integrated Nsight Copilot into Nsight Compute analysis and your profiling and figuring out what's going on, because these agents are so powerful; why would you not want to have the ability to access them? Also on that note, some other colleagues of mine on the research side have been working on a way to benchmark and evaluate how good one of these coding agents is. How good is an agent? They've come up with this very interesting structure where they've produced 250 different benchmarks culled from an enormous range of different applications to test out these coding agents and say, what is the quality, how do you measure the quality, of an AI? It's very, very difficult. So this is their stake in the ground to say, here is a very well-thought-through, detailed, comprehensive suite of benchmarks for the AIs to be tested against to see, am I improving the quality of my agent or not? They have a kernel-generation competition. There's a QR code here. You can go to their website and explore the benchmark. Again, in the PDF, all of these links will be available online and you can just go and follow them.
Finally, I love that one of the most common things that is asked of me is, can I recommend a book for CUDA, which might be a bit redundant two years from now in the age of AI agents, but right now people are very interested in books for CUDA. My friend, my colleague, my very long-time and longtime lover of CUDA, Dr. Wen-mei Hwu, is coming out with a fifth edition—I think it's just coming out this week—of 'Programming Massively Parallel Processors.' Every book, of course, is obsolete the day after it's released. So today it's current. Get it today because it's the only day—no, that's not fair. But not only that, he's also having a Q&A. Even though I hate plugging my own talks, I'm actually moderating a panel of all the CUDA old-timers sitting on it talking about the old days. You should definitely come and check that out on Wednesday as well. That's the bottom link right here, because you'll have a chance to see Wen-mei and just ask us questions about what CUDA's been like over the last 20 years, because CUDA 20 years ago was really just finally coming together. I think the beta was launched in 2006.
All the way through this talk I've given you a list of links. There's always a table of this at the end. These are talks of my colleagues who provided all this information that I've been telling you. I give you two slides on everything. These guys are going to go and tell you about it for 45 minutes. These are the talks. If you are a CUDA developer, these are the ones you want to be either going to watch or streaming later. The 'Connect with the Experts' sessions let you come and talk to us all. I'll be at some of those as well. And that's the end. So thank you very much.