Ep 11: Future of AI Computing and How to Build & Nurture Hardware Teams with Jim Keller, Tenstorrent

Thanks. Good to be here.

I was going to say, I thought you were going to say what's keeping you up at night. Well, I had a literal keep-me-up-at-night. I was just in India for a week. We opened up a design team there, and then I met the IT minister for compute. India has an initiative to promote RISC-V, high-performance servers, and India-based designs. So, been talking to those guys about it. Literally was up very early in the morning, so that's one thing. I think AI and modern tools and a few other things are causing change faster than anybody really thinks about. Like the tools are changing, the design points changing, the code is changing, and how do you build and design computers and software so you can go faster and use those tools, you know, top to bottom. Like we went from, you know, custom design to design using CAD tools to like SoC designs where you have multiple IP components and then you put those together. And now the design complexity keeps going up. Moore's law gives you more transistors, but you still want to make progress at a good rate. And then how do you do all that together? And so that's, and then that opens you up to applications, and then AI applications are really crazy, and I've been learning a lot about it, so. You think at some point like things would slow down, but the opposite's happening. Like things are actually happening faster. Although I tend to wake up more in the middle of the night thinking about things.

That's pretty funny. That's a pretty good metaphor, actually. I like that.

Yeah, so well, first one requirement, this is hard on big companies with lots of legacy, is you need new designs. Like this is really like every, I've said every 5 years you need to write stuff from scratch, period. And there's a big reason for that, because no matter how good your old thing is, as you add little improvements and patches, it slowly becomes tangled together. A friend of mine sent me this paper titled 'A Big Ball of Mud.' And it's a really old-school website with a picture of a big ball of mud on the top, and it talks about no matter how carefully you architect hardware or software, you have nice clean components, well-defined interfaces or APIs, over time, like this piece of software will learn about this piece, and this will do something because of that, and you know, somebody will communicate in a meeting, they'll figure out a clever way to make something faster, and pretty soon it's all tied together. So, you need to have something new. We're building a RISC-V processor, you know, a fairly high-end one. We spent a lot of time up front architecting it so that each subsection of the computer is really modular and has really clean interfaces, so that, you know, the mission, I told them it would be unbelievably great if we found 90, 95, maybe 99% of the bugs at component level instead of at the CPU integration level. And like SoCs went through that transition. Like if you buy high-quality IP from, you know, IP vendors and you make a chip, you don't really expect to find any bugs in the IP. So, and but if you're a company with lots of legacy IP and some of the IP was created by breaking up a more complicated thing and you never cleaned it up, you might find a large percentage of your bugs when you put pieces together. And you have to fix that. And so, new design gives you an opportunity to say, 'I'm going to go redesign this and make it clean at the module level.' And when I worked on Zen, some of the verification team came to me and basically told, 'Like Jim, we really want to test all the units with really extensive test benches.' And the old-school way of thinking was, 'Oh, yeah, sure. We create the whole design twice.' You know, you have a load-store unit and now you have to make a thing that tests the load-store unit, which looks a lot like the rest of the computer. Right? But they were right because actually the code that tests the load-store unit is actually simpler than the rest of the computer. And if you put 10 units in a row together, you know, fetch, instruction, I-cache, decode, rename, schedule, integer execute, load-store, your ability from a program to manipulate the load-store unit is tough because you're looking through five layers of really complex state machines. Right? Whereas if you want to make the load-store unit do all its things right from the load-store pins, you can do that. So, sort of like I was sort of getting there, but the verification engineers made the case it would be way easier for them to write more code at the test bench level and have modules that we test. And then as soon as you think that, then you say, 'Well, why does the interface between these two units have 100 randomly named signals?' Like, if you've done detailed computer architecture, you know there's a signal called stage-four fetch valid except when, you know, something happens, right? That's not a verifiable signal. Like, and computers are full of that stuff for timing reasons, for no good reason, for the way you had to fix a bug. 'Oh, now this unit needs to know that this is in a state,' right? Whereas, you know, computers have well-defined interfaces, you know, instruction, PC, you know, memory fetch, fill, exception, kill, stall, like... So, there's a really interesting challenge of like, how do you build a computer with well-defined interfaces? And so, your question, like whenever something's, I made this comment in the talk I did, whenever something's hard, it's because there's too much complicated, too much stuff in one place. Like, you have to break things down. And sometimes the problem isn't whether the tools are there or not, but that you've tried to solve something, you know, in too many ways. Like, you have an RTL problem, a timing problem, a physical design problem, and it gets to be too much, and you have to really figure out how to break that down and make it simple enough to do, and, you know, verifiable design, verifiable interfaces, you know, architecture thinking a little bit differently is important. But, yeah, I thought a lot about that. And then you can see it, you know, like some projects you have a lot of success in, and it's partly because you really took the time to architect the pieces up front and make them very independent and really clean. And you had discipline not to slowly turn it into a ball of mud, which is a natural human tendency, apparently.

Yeah, so there's a couple of things there. So, one is, you know, like 100% of the Fortune 100 companies from 100 years ago are gone. Like, GE is still around, but in a completely different form. So, companies do go through life cycles and almost all of them disappear over time. Like, and some get propelled for, you know, monopoly reasons or infrastructure reasons or something, but so success today does not guarantee success. Although, the time of that is longer than you think. Most companies don't fail in 5 years, they fail in 25 or 50. Right? So, that's a thing, right? And then Steve Jobs famously said, 'So, you have some new product, and then you make it a lot better, and then you refine it, right? But to get to the next level, you have to make another new product. And the problem is the new product isn't as good as the refined old product. But you can't make the old product any better. That's the best rotary phone that will ever be. The push-button phone is... it doesn't feel as good. It doesn't look as good. And you have to have the courage to jump off the refined high spot to a lower spot that has headroom.' Like this is a quote from one of his random talks. So I recommend people go, like some of Steve Jobs' keynotes were great. Like I watched a bunch of them. And he was very clear about what it means to design a product and then to believe in where you're going. So you have to, and it's really hard for marketing and sales people in a big company. So you go, 'Hey, we got this great new product. It's 10% worse than what we have today, but over the next 5 years it's going to be twice as good.' And they're all like, 'Well, we'll wait 5 years.' But then you're not working on the right thing. So you have to do that. So that's one thing. The other is I had this funny talk with a famous big company who's providing IP, and they had this interface which was a lot of wires and a lot of random stuff. And I told him like the interface is too complicated. And he goes, 'Yeah, but it's really, it's a small number of gates.' And I said, 'No, you don't get it. The wires are expensive. The gates are free.' Right? So one thing you do is instead of saying, 'I got this, you know, lean and mean state machine and I export all the wires,' is you take that and you put it into a box and you turn it into an interface. And you trade off, you know, at the time I thought we traded off gates for wires. Add a bunch of gates because they're cheap. Moore's law gives you lots of gates and have less wires. But a better way to say it is we trade off, you know, technology for complexity. Like if you go look at a DDR memory controller, for example, on an AXI bus. So, a typical IP you can buy from, you know, three or four or five people. AXI transactions are very simple. There's like 15 commands, and you mostly use read and write. So, at the controller, you say, 'I want to read 32 bytes of data at this address.' That's a really simple interface. Inside the controller, there's a memory queue. You might have 32 or 64 transactions. There's a pending write buffer. There's a little state machine that knows the current state of the DDR channel. Maybe there's two DDR channels. There's two DIMMs per channel. DDR DRAMs are really complicated widgets because they got a read cycle, a RAS cycle, a CAS cycle, a refresh time. They're in different states. At the read transaction, it's really easy. Read address. Write address data. You don't know anything about the state of the... Now, if you build a high-performance system, you say, 'The CPU is going to be optimized, and we're going to send read commands to the DRAMs, and we know that we're going to have to sequence the read command. So, we're going to hit it with this RAS early to get the DRAM.' So, you can export the complexity of that. And then the CPU knows exactly on cycle 167 after it's the first piece of data that's going to come out. We used to build CPUs that would wrap the return read transactions, so you got the requested word first. Like, we had all kinds of complexity. But nowadays, the transactions are really simple. Read or write at the memory controller, period. The data comes out at a random time. It always comes out in the same order. You don't export the complexity of the CPU to the memory controller. Now, partly it's for a good reason that the CPUs have really big caches and really good prefetchers. They're running at 3 to 5 GHz. And the memory controller latency is 150 ns. Like wrapping that transaction saves 0.2 ns out of 100. It's a dumb complexity, right? But you look at your design and you go, 'Well, how do I get the complexity of an interface to be so simple?' Right? And then there's a funny one, which is people always want to say, 'Well, why don't we have an industry standard for cache coherency?' Well, cache coherency is a distributed state machine. Right? So, now you're saying, 'Well, the people that, you know, Qualcomm are going to have the same spec as ARM as, you know, somebody else.' And that's a hard thing to do. Whereas specs like AXI, you know, there's a bunch of specs that are pretty commonly used that are really simple. And when you make them simple enough, then many people can use them. PCI Express is simple enough. Ethernet is mostly simple enough. So, you see these things that are common standards work on them. Like the first version of InfiniBand tried to optimize latency and they had a 1,300-page spec and nobody could build a device to the InfiniBand spec. So, they went through some soul searching and they radically simplified it and focused on being a generation ahead on PHYs and having a simple but good enough RDMA and some other things. And that became a product that people could use somewhat successfully. How do you avoid complexity? Well, at the top you have to decide it's a real goal and then you're going to spend something on it. I'm going to put an extra 100,000 gates in each interface so that the interface is simple. And in a computer with 200,000 gates, that's crazy. In the computer with 100 million gates, that's genius. Right? Like, there's a different calculation.

Yes. So, yeah, so this is a really good one. And I'd say in AI, we haven't figured out what the hardware-software contract is yet. And I'll give you an example. So, in the CPU world, and this is not quite true, but this is close to true, software does arbitrarily complicated things. Like if you go look at a virtual machine JavaScript engine, like it's amazing, right? There's really complicated things. And then I grew up, like when I learned to program, I programmed in assembly. Like I used to know all the opcodes for a 6502 and an 8086, and most of them for VAX. And then I learned C programming. C programming is great because it's an assembly language. You know, it's a high-level assembly language at some level. As an architect, you write C code and you can see what the instructions will be generated and mostly see how it's going to execute. It's pretty simple. But the actual contract for a modern computer is operations happen on registers, period. You put data in registers and then you do add, subtracts, and multiplies on them and you can branch on them. And then from a programmer's point of view, there's a memory model where you load things basically in order. Like if you load address A and then you load it again, you never get like an older value after you get a newer value. So it looks like you have ordered loads. And then you mostly have ordered stores. Like, and there's weak ordering models, but they mostly don't work because you have to put barriers in to fix it. So you basically, data lives in memory, you load it with relatively ordered loads, you do operations on registers, and then you store the data out with ordered stores. Right? And then there's a page model, paging model, a privilege model, a security model. And then, but those are orthogonal to the execution model. Right? And so underneath that simple model, you can build an out-of-order computer. And it took us 20 years to figure out how to do that. Rule number one is you don't violate the execution model that the software people see. So VLIW failed because they tried to violate the model. Weak ordering kind of fails because it violates the model. Like anybody, like people who did radical recompile to get performance, you know, with a simple engine failed. Like out-of-order execution is really wild because the instructions issue wildly out of order, but they don't violate that model. And to achieve that, we build register renaming or something called a kill. So, you execute a bunch of instructions out of order and some event happens and then you flush all the younger instructions from the kill point and you finish the older ones in order. Right? We have massive branch predictors, data prefetchers, but no matter what you do, you don't violate the contract on execution. And that means that the software programmers do not have to be microarchitects. Right? As soon as you ask them to be microarchitects, you failed. Itanium had like eight barriers. Nobody knew what they were for. Like I was at Digital when we built Alpha. We had a memory barrier and then we added a, because we had weak memory ordering, so we violated the execution contract and we broke all the software. So, we had a memory barrier, but they didn't know where to put it in. They even, the operating system had a double MB macro because they didn't know where to put it in, so two of them some places seemed to fix some random bugs. Like I'm not kidding. Now we added a write memory barrier, which we thought would make things better and it made it worse because they just put the write memory barrier in the memory barrier macro because they didn't know what to do with it. Right? So, it was like a worst-case scenario. So, now look at AI software. So, AI software has been developed mostly by programmers. And programmers understand the execution model pretty well. Data lives in memory. You declare variables, which gives you a piece of memory, or you do something like malloc and free, which is a, you know, some kind of memory allocator on top of a memory model. But generally speaking, when you're in a program, you don't talk about variables as addresses. They have names. And generally to do operate, you know, so you say A = B * C. Implicit in that is a load of A, B, and C. They go in the registers, you do operates on them, and send them back. And GPUs today are sort of executing that model. Like you have lots of very fast HBM DRAM, data all sits in memory. For every matrix multiply, the data's in memory, you load it in the registers, you do the operations, you write it back out again. So you're constantly writing in and out data. That makes sense. Now, at Tenstorrent, we believe that when you write that program, and you can see it very clearly in all the descriptions of AI, that program actually defines a data flow graph. So if you go Google Transformers or ResNet, you'll probably get a picture, right? And the picture will be a graph. And right, and the graph says there's an operation box where data goes into it, something happens, and then something flows out of it. And then they generally, they call the input activations and the local data for that operation a weight. And the number of operations they do is actually quite small. Matrix multiply, convolution, some version of, I don't know, ReLU, you know, softmax, and then a variety of what they call tensor modifications, you know, where you shrink or blow the matrix, you pivot it, you transpose it, you convert 2D to 3D. There's a bunch of tensor modifications. But the number of operators in that space is low. And then people are stylized on things like, you know, how do you exactly program ReLU? Like there's some implementation methods. So it's interesting that the programmers are writing code in PyTorch to a programming model that looks like standard programming. They're describing what they're doing in terms of graphs. Because that's a nice way to think about it. But the code itself we see is a mix. So the challenge is like how do we come up with a programming model that we all believe in and understand that can go fast and not have to do things like read and write all the data to memory all the time because some operator expects data to be in memory and the only good way to do it. And then I've talked to, you know, AI programmers are like, 'I'd happily recode that to make it twice as fast.' Like that's one view. And the other view is, 'I really don't care because all the upside in this is either size, bigger weights, bigger weights, more parameters. And the hardware is going to make it faster and then in the short run I'll just buy more processors.' So there's a really interesting dynamic about this. And it sort of feels like, you know, we first started building out-of-order processors back in the, you know, I guess I started working on it in '95. Like it had been around for a while. The IBM 360 did out-of-order execution. The 360-91, I think. It was amazing. But when I was at Digital, there was a debate about whether you could actually make an out-of-order computer work. Right? And there was a, you know, the competing ideas were, you know, there was super pipelined, superscalar, VLIW, out-of-order, and then little window, big window. There's a bunch of ideas about it. And what won, like clearly I think, but you know, there's still some people who are debating this, is, you know, out-of-order machines with big windows and really well-architected reorder buffers, renaming, and kill interfaces. Like that works. Right? And a really simple programmer model that works. So the interesting thing is, so that, and the GPUs, like some people tell me, well, GPUs just work, but Nvidia's thousands of people hand-coding low-level libraries. There was a really good academic paper that said, 'Hey, I decided to write matrix multiply and I wrote it in CUDA the obvious way and I got 5% of the performance of Nvidia's library.' And then they did the obvious HPC transformations of transpose one of the matrix, sub-block it for the known register file size, and they got 30%. And from 30 to 60% they started to hack. They know how big the register file is, how many execution units there are, how many threads, how many, you know. And the Nvidia library has, you know, they have what they call CUDA ninjas who are great programmers who know how to make that work. Now, the charm of CUDA is you write a CUDA program, it'll work. The downside is the performance may be randomly off by a factor of a lot. But when you're writing your code, if you say, 'Well, why would you write matrix multiply on a GPU? There's a big library for that.' Like so they're a library, so they have a program model that works and libraries that mostly solve your needs, and then now you're arbitraging the last 10, 20%. But that computer doesn't look anything like the way the actual AI programmers describe the program. Right? And so that's a really interesting thing. So we're building a graph compiler. So our processor is an array of processors which have some low-level hardware support to support data flow. And there's some interesting methods about how you take big operations and break them up to small operations and coordinate that. And the charm of it is it gives you much more efficient processing and less data movement, less reading and writing of memory. There's interesting things. Think of like, so you say, if I have the RAM big enough to hold all the data I ever need, it's a big RAM, there's lots of power. So if you break the RAM into a small spot, smaller thing, it's much more efficient for access, but then you might have to go to other RAM. So there's a trade-off between the RAM size, and that matrix multiply has this curious phenomenon of for an n by n matrix, it's n squared data movements for n cubed operations. Right? And which is sort of why AI works. Like AIs, you make the operations bigger, you get more computation than data movement. And then there's ways to optimize that further by breaking the big operations into the right size. So they're big enough to have a good ratio of compute to data movement, but small enough to be efficient on local memory access. But it's very much like you can see all the AI startups are taking different approaches at this. And it's not because people are, you know, trying to do something different, it's because there's a real problem there, which is, you know, how the programs are written, how they're described, what they look at are very different things. And like it's technically interesting. And I think the solution will be much better than, 'Oh, we'll just keep scaling faster memories forever.' Like that doesn't seem like the right approach.

Well, actually they're really small systems. So I think you have your metric wrong. So the human brain seems to be intelligent and people estimate it at 10 to the 18th to 10 to the 22nd operations depending on who you are. Right? So a GPU is currently, you know, 10 to the 15th operations a second. So it's off by something like six orders of magnitude. So we have a computer about this big, right? Which is an average, you know, intelligent operation computer. Right? And then to build that today with GPUs would take, you know, 100,000 GPUs or something, which is like the problem is in the GPU side. So we say that's big, but it's not that big. Well, big compared to what? Right? That's the funny part. Like it used to be it took a really big computer to run a simple Fortran program. And you could say that was a big computer, but now that computer fits on a half a millimeter of silicon. So the Fortran computer of the '70s is, you know, 0.1 square millimeters. So Moore's law fixed it. So size is a relative thing. So today, yes, to build a big training machine, you put together 1,000 GPUs and it feels really big and the parameter count is like, you know, 30 billion parameters and there's, you know, a petabyte of training data and we walk around going, 'Those numbers are so big.' Here's a funny number. So a transistor is 1,000 by 1,000 by 1,000 atoms. 100 nanometers. Just think about like a 7-nanometer transistor. They call it seven, but it's about 100 by 100 by 100 nanometers. And 100 nanometers, 1,000 transistors. So a billion atoms. And we use that to resolve a one and a zero. We resolve a one and a zero at about a gigahertz, which is pretty cool. So it's a billion ones and zeros per second out of a billion atoms. So is a billion a big number or small number? I don't know. Like it's a... machines look big. But like the thing, the computer in an iPhone would have been like, you know, 10 Cray-1 computers, which were big in their day. And now we think of them as a $20 part that, you know, fits in a 3-watt envelope. So it's a relative measure. And AI programs today are big compared to traditional computing, but they're small compared to actual, you know, most average intelligent people.

Yeah, well, the fascinating thing about like a current, like a thousand-chip, you know, ML GPU computer is first there's an accelerator model. There's a host CPU and an accelerator. Then in the GPU itself, there's a memory-to-memory operator model. Right? And then that node runs some kind of networking stack between multiple nodes, and then it's coordinated with something like MPI. So you have a memory-to-memory model, an accelerator model, a networking model, an MPI model. And so to make it all work, you know, this is before you even run a program. Like it's kind of amazing. Right? And you can look back when, you know, when processors had FPU accelerators, the FPU had a driver, right? So you had to send operations to the FPU and then poll it. But when the FPU got integrated together, the floating point just became a data type and an instruction in a standard programming model. So the accelerator model occasionally disappears. So there was a, you know, as floating point got integrated, there were still vector processors, which were accelerators for vector programs. And they died essentially because the floating point got fast enough that it was way easier to just have a couple more computers running floating point programs than it was to manage accelerator driver models. So the current software structure is, I would say, somewhat archaic and complicated. But it's built on well-founded things like GPUs accelerating graphics programs has been solid for years. There's lots, everybody looks at it and goes, 'Man, there's a lot of memory copies there.' And all the programming model the GPUs do is simple. But you know, that's a 20-year-old model. And networking works. And MPI's been used in HPC for a long time. And it's pretty well, you know, fleshed out. But the fact that, you know, to run an AI program, you need something like four programming models before you even write a PyTorch program, it's kind of amazing. And even the PyTorch doesn't really comprehend the higher-level thing. So they're running locally on nodes underneath some MPI coordinator thing. And yeah, it's fairly complicated. Now, if you had a really, really fast computer that ran AI, you know, those layers would go away. But we don't have that computer yet. And that's where that, you know, the excitement happens at the, you know, what's the right way to think about this stuff. And it feels very much like we're in a transitional place. And we've been through these before, like the change from in-order computers to superscalar, super vector, you know, out-of-order, VLIW, all that took like 15 years. And we're probably in year three of this on AI.