Ep 2: Domain-specific Accelerators with Dr. Bill Dally, Nvidia

Well, I'm really happy to be here. Thank you for inviting me.

I'm just really excited about a lot of things, and probably the one right now is probably building accelerators for certain demanding problems.

Well, I think the big one is domain-specific accelerators, and alongside that, parallel computing, and they're really very, very closely related. It boils down to the fact that if you know serial computers, single-thread processors aren't getting any faster, and if you want to add value to applications, you need more performance, then you need to have lots of threads in parallel. And to do that efficiently, you really need to specialize for a certain domain. And the gains to be had there are often into the thousands of fold compared to a conventional processor.

Well, in my mind, designing a domain-specific accelerator is really a programming exercise. It's understanding the application and reprogramming it in a way that is hardware-friendly so that you get rid of serial bottlenecks. You have to make the application very parallel. You get rid of memory bottlenecks, so you minimize accesses to large global memory structures, and you try to transform it so that storage can be local with a small memory footprint per element. And it's really that reprogramming that is the challenge of designing a domain-specific accelerator. Once you've reformulated the problem into a hardware-friendly form, crunching out the hardware is a largely mechanical activity. I think it could actually be automated in the future.

Right. It's absolutely redesigning the algorithm in a form that's hardware-friendly. That is the real challenge of accelerator design. It's very rarely the case that the existing algorithm without modification is very suitable for acceleration.

Well, I think you just have to let yourself do it. I mean, so when we started our Darwin project that I did with you, my former student, we actually went exactly down that loop. And what we found is if we took the existing best algorithms and threw everything we knew about hardware to accelerate them, we might get 4x. And yeah, I guess we could have stopped there, but we weren't very happy with 4x. So we asked ourselves, you know, why is it limited to that? And we found that it was because that particular algorithm had been very carefully tuned to get optimum performance on a CPU where memory accesses were relatively cheap and doing the dynamic programming, which is a core of the algorithm, was very expensive. So they had basically set the balance between what's called the filtering stage, where you take lots of candidates and filter them out, which takes lots of memory accesses, and the alignment stage, which is all dynamic programming, and they made it all filtering because that was cheaper on the CPU. And the first thing we did that kind of transformed the game was to say, okay, we don't need to stick to those thresholds of how much filtering to do. We could do less filtering because it's expensive. That means there's more work left to be done in the alignment stage, but that's really cheap. And doing that turned it from being 4x better to being 15,000 times better. And so I think architects just need to understand the application well enough that they feel comfortable changing it so they can change the details of how it's implemented, but understand that they're going to deliver the answers that whoever is using that application, in this case, biologists doing genetic research, want, and with equal or higher fidelity. And so in that case, it was a measure called sensitivity. We had to basically get equal or higher sensitivity than the existing algorithms, and then all the users were happy.

I think you could probably do it in either framework. The Darwin project we did at Stanford, and we collaborated with Gil Bejerano in the developmental biology department, who's a real expert on the algorithms and on the end uses of the programs. And so it was really valuable to have. I think it's actually for any architecture project, you need to have input from people who are ultimately going to be the end users, and that's relatively easy to do in an academic environment where you do have a lot of experts around. I think it's possible to do in industry, but it usually means forming collaborations with people outside the company.

A lot of different things happen. There's no one formula. So, you know, we're building a lot of deep learning accelerators, and there I think all the expertise we have in the company, we have some of the world's best deep learning researchers, and we understand the directions algorithms are going in and the models are going in. Because if you tune a deep learning accelerator for running ImageNet on ResNet, you're five years behind the times. I mean, almost nobody cares actually about the vision networks anymore. It's all the action's in the natural language processing and recommender networks, which are fundamentally a very different problem in a lot of very serious ways. And then we have other things where people are applying GPUs to running some problem, and they'll come to the company saying, gee, I'd really like to get more out of this. And we'll look at what they're doing, and sometimes we actually have an organization that actually helps customers tune their programs for GPUs. And sometimes in doing that, we'll see an opportunity where we could add acceleration. Another example was in graphics. We recently added ray tracing acceleration to our GPUs. It's a project we started in NVIDIA Research, and that was where, you know, here's an algorithm that people say could never run real-time, and through the combination of hardware that accelerates things, and here the gain of the hardware is probably like 10x, it's not thousands, because the software schemes were doing pretty well, between that hardware and applying deep learning so that we can actually generate relatively noisy images and then denoise them, we've been able to bring ray tracing to real-time. And you get really photorealistic graphics running on somebody's personal computer at 60 frames per second, which is pretty amazing. And there again, I think we had all the expertise. We had some of the world's leading researchers in ray tracing, which at that point in time was used for offline graphics, for rendering a movie at one frame per hour. But we could take their expertise and combine it with the hardware expertise and architecture expertise we had, and use that to craft a good accelerator.

Well, I'm not sure what you mean by legacy compute. I mean, all of computing are applications, and those applications solve some problem, whether it's simulating physics or whether it's analyzing data, or whether it's monitoring video feeds, which I guess is a form of analyzing data. What legacy means to me is that somebody has a system and it works for them, and there is inertia which keeps them from, you know, they don't want to make any changes, and progress always requires change. And so whatever they're doing on that legacy compute, whether it's data analytics, running over large data sets trying to glean insight about something, you could look at what algorithms they're using there, find where the bulk of the time is being spent. Do that same co-design where you say, okay, we've got to grapple over all this data, but is there a way of making intermediate references very local so we can avoid large global memory bandwidth problems and design an accelerator for it? And it's really a question of finding applications where the effort-reward ratio is right. When you look at things like deep learning and genomics, and to some extent even the ray tracing, what makes it really amenable to the accelerator is that there's sort of one core kernel. For deep learning, it's matrix multiply. For genomics, it's dynamic programming. For ray tracing, it's traversing a bounding volume hierarchy. And if you accelerate that one core kernel, the whole application speeds up enormously. And so those applications are the ones that are really nice, the easier, the low-hanging fruit for acceleration. There are other applications where you look at the code and it's several hundred thousand lines of code, and it's hard to tell any one place most of the time is being spent. That's a mess. That would be very hard to accelerate. And I think in many cases there, you have to go back and ask what is the fundamental problem they're solving, and do you really need 100,000 lines of code, or did that just aggregate over a long period of time of different programmers going in and adding a few lines here and a few lines there and not taking out anything that wasn't being used that often? And can you streamline it? Can you actually refocus and rebuild it from the ground up?

Yeah. So if you look at where the performance comes in accelerators, it largely comes from parallelism. I mean, specialization can buy you 10 to 50-fold depending on what the application is, but the cases where you get thousands is because you're operating in parallel. And once you're operating in parallel, you need to communicate data between the processing elements, between various storage and processing elements. And what you find very quickly is that communications becomes the dominant problem. It's where most of the energy goes. A lot of people say, no, the energy is in memory access, but we do get the memory access. The fundamental operation of reading or writing a bit cell takes almost no energy. All of the energy in memory access is moving the bit from that bit cell to the sense amps, and then from the sense amps, from a subarray to some global memory interface and ultimately to where it's being consumed. And so because it's so dominant, you need to do a very good job of that communication. And so it's an area where I've been working on it since the 1970s, actually. It really is nice because there's a good theoretical framework to understand that communication and then to optimize it in a way that you can make sure that every femtojoule of energy you spend moving bits is being spent in a very optimal way. And that can really make the difference between an accelerator that works well and one that is horribly inefficient.

That's a really good question. So there are things that are the same, things that are different. And what's the same is all the theory is the same. The problems are the same. When you're constructing a network, you have topology, you have routing algorithms, you have flow control. All of those things are the same on-chip and off-chip. Now the answers are often different. The reason why the answers are different is the cost models are very different. So on-chip, you're signaling with CMOS logic over on-chip wires, which are RC wires. They're lossy and they're resistive. You have to put repeaters in them. And that gives you a certain cost model and area and energy for moving bits on-chip. And it actually is a cost model where the energy is proportional to distance. So the longer a channel is, the more energy you consume on that channel. Once you go off-chip, you pay a big upfront cost for an I/O driver on the chip to drive a wire off-chip, and perhaps even for an optical engine to drive a fiber that's connected to that. But once you've paid that cost, in the electrical case you can go meters. In the optical case you can go hundreds of meters or even kilometers with no additional cost. And so because the cost model for the channels is different, the solutions change. And also because the channels are now very expensive, you have to pay this big upfront cost for the I/O, you can now amortize against that channel a lot of buffering, a lot of routing logic. So you often will go to more complex solutions. The fault model is different as well. On-chip, even though errors do occur and there's FIT rates for everything, particularly storage cells and SRAMs, a lot of on-chip interconnection network is done assuming a basically reliable network where then you'll put some error checking on it, CRC or parity, and have some fallback if a fault is detected, but it's a relatively rare occurrence. The channels off-chip often are as bad as 10 to the minus 8th or 10 to the minus 9th bit error rate. So faults happen on a really, like many times a second. And so you have to have some fault recovery mechanism that is transparent and recovers from that. But the theory is the same, and it's just that the optimal answers differ because you're applying different cost models to it.

There's two sides to latency. One is how much your application can tolerate, and the other is how much is essential and how much is gratuitous. And in on-chip networks, it turns out moving a bit on-chip is actually slower in terms of velocity because those RC lines don't operate anywhere near the speed of light. And once you go off-chip and you're on a good LC transmission line or on an optical fiber, you're typically running at about half the speed of light, depending on what your dielectric is in the transmission line. And so there's nothing fundamentally higher latency about going off-chip. You could have an on-chip network that, to cover, we build really big chips, they're 800 square millimeters, and so you figure over 20 millimeters on an edge, you could be going 40 millimeters on-chip, and that can wind up taking many hundreds of nanoseconds. And if you think about it, you can go on a single optical fiber, you can go about roughly a half a foot per nanosecond. So in hundreds of nanoseconds, you can be say 50 feet, 50 feet per 100 nanoseconds, several hundred nanoseconds, you can be almost a football field length away. Now, very often off-chip networks, because people can amortize more complicated routers, they do, and they build very complicated routers, and the routers can wind up having latencies of easily 50 clock cycles. I find that very painful sometimes. I built many routers back in the 1980s and early 1990s where we were 10 clock cycles from bits arriving on the input pin to bits going out on the output pin. And some of the latency is inherent because on these big router chips now, you have maybe 20 or 30 millimeters of traversal to happen to actually just get the bit from the input port through some on-chip network which forms the core of that router to the output port. And that's essential latency. But some of it is sort of designers being a little bit lazy with modularity. They say, okay, I'm going to design a block here and put some buffers before it and buffers after it, so I can think about that separately from other things. Whereas if they crafted it more carefully, I think much of that delay could go away. The other thing that eats up a lot of latency is sometimes expensive clock domain crossings. Synchronization failure is a real problem. And many people still use what I refer to as brute-force synchronizers to cross clock domains. And those in modern technologies can easily cost five or 10 clocks just to cross one clock domain. Whereas there are known techniques, we've published many papers about them, where you can have an average cost of a half a clock to cross a clock domain. And most people, because they don't feel enough pain about that latency, they don't feel motivated to go and apply the more sophisticated synchronizers.