The Architecture of "Open" Intelligence

Hey, good to see you. Thanks for the intro. And by the way, it was A4, 5, 6, and 7, but for some reason they won't put that in the Wikipedia page, and I don't know why. That's brief.

Yeah, it's really interesting. There's a book called The Systems Bible, I think it's like, which basically says you can't design complicated things, and I really like that. And then the sub-rule is complicated systems, you know, evolve to be that way either through an addition of features or, you know, combination of simpler systems. And then there's a corollary to it, which is you can't fix broken complicated systems. And so in designing modern computers now, we used to do like a CPU as a chip, and then there was an integrated chip, and then there was a big pivot to go to IP-based chips. So you had a CPU as IP, fabric as IP, memory controller as IP. And every time you want to achieve more complexity, you have to sort of re-engineer for the component levels to be simpler and then build it up with combinations.

And in the last couple of years at Tenstorrent, we're building, you know, an AI supercomputer. We're going to basically have thousands of AI chips collaborate with millions of processors from a very simple software stack. And to achieve that, we really had to redesign the software stack and make it layers. And then recently, we've refactored our chip plans to be chiplet-based, and we just figured out how to make our systems also much more modular. And so by making everything have the right abstraction layers and modularity, we can actually build really complicated things out of simpler components. And yeah, I think that's a pretty solid foundation for doing new things.

Yeah, so there's a couple of things. Like GPUs with extensions of tensor processors have been quite effective on running AI, but the scale-up part has been built around multiple layers of software, right? So, you know, you have the program running on one GPU and another software layer, you know, collaborating GPUs and another software layer, and I think that software stack is quite complex and takes a lot of people to manage it. And the thing I'm really into is, you know, how do we build the layers of software in a way that, you know, the programmer doing an AI model can simply describe what he wants and then use the layers of software to do it. So that I think like we are now achieving that. It's pretty cool. And then the other piece is the current data center. There's like a host CPU that runs a driver to drive the GPUs, and then there's a separate set of computers to, quote, do general-purpose servers. And the thing I've been really interested in for years is kind of heterogeneous computing where the CPUs and AI processors collaborate together. And we're building an architecture. So we picked RISC-V so we can build our own CPU and we can connect it together with AI in a really flexible, novel way. I think there's a lot of innovation in this space, and like we recently announced Ascalon, our high-performance, wide, out-of-order CPU. We're going to build a server chip using the BoW fabric next year. That's really cool. But the way that processor integrates in with the AI is novel, and I think it's really going to unlock a lot of creativity on how to build future software stacks.

Yeah, it's really interesting. So fabrics used to hook a few things together, right? And then people built cache-coherent networks, which are famously hard, and, you know, and we scale them pretty far, but it's really complicated. Like, you know, 64-core, 128-core cache-coherent chips are remarkably hard, and that's partly because it's a lot of distributed state and all the components have to really know exactly what they're doing. Whereas network-attached compute scaled really fast, and that's partly because it's, from a state point of view, pretty lightweight. If I send you a packet, you keep track of packet counts. You know, they don't have to know what each other is doing. And then as we go into chiplet-based design, we want to put more silicon together, but it can't scale in a way that's complex, like, you know, let's say cache coherency and stuff. And then the AI mostly doesn't care about that. So the BoW fabric, we started tops-down to say, how do we build a fabric that naturally works with chiplets, does on-chip interconnect, does cache coherence locally where we need to? Like, I really like that design. Sometimes data is moved by the big processing agents themselves. You know, Tenstorrent architecture, we actually have multiple CPUs in every AI cluster. We're doing our future general-purpose CPUs the same way. So we'll have CPUs that run programs and stuff, but there's also additional CPUs. We have software-defined data movement at any layer.

The weird thing was when we say data sharing, sometimes we really mean data sharing, but sometimes what we mean is we have these very large data objects sharded across a very large number of processors, and then as the processors are computing on it, they want to have a very natural method to continue to exchange the updated partial products, to move the data around, to use the peer processors' memories as a way to keep all the data. And that's where you have the fabric layer, the data movement engines naturally talk in the software to coordinate this big stack. And it's sort of we're building it layer by layer. And again, you know, my short-term mission is to have a thousand chips, a million processors collaborating on very large programs in a heterogeneous way from a very simple software stack. And I think it's happening. And yeah, it's quite something to achieve.

Yeah. Well, so this is... AI is really interesting because there are parts of it that look very, very much like compile time. You figure it all out. You know just what to do. And people, you know, often say, well, if you just have this set of matrices and you know exactly when you're reading and writing all the data and, you know, it's going to be deterministic. I don't think that's true in general. You know, there's like local details, like you're doing computing and, you know, you're taking advantage of sparsity, the data is really uneven. You're constantly changing formats. You know, you may not even compute some kind of data, so it's not that, you know, stable. The mixture of experts world, as you're building these big collaborative AI models, but which experts you fire up, that's dynamic based on the computation in the matrix. So that's an interesting thing. You have all this variable-length inference chained together with batching, so that gives you another kind of dynamic stuff. But the thing that's starting to happen is as the AI is working for you, it's generating requests into databases, into the general-purpose computing. You know, you're going to have the AI generating programs which are then going to run on data to summarize it or bring it back. And long run, it's going to be a very dynamic computer with some things that are stable, like we assign these processors to do this kind of thing and we're laying out certain kinds of databases in certain ways, but boy, the access patterns and the compute patterns are going to be really dynamic, and your software stack has to deal with that. And it's way, way too big. Millions of processors is way too big to have cache coherency.

I think, remember, it's the layers that need to be simple. You know, I give this talk periodically. It's like humans aren't getting smarter, and like technically we're not that smart, and so we can't solve really big complicated problems, but we're pretty good at breaking problems down. And then sometimes you see sadder stacks that obviously have too many layers in them, and they spend all their time fighting about the abstractions. But more often, things slowly turn into a big ball of mud, which is a famous computer science paper. So if your listeners want to go look it up, just look up big ball of mud. It's great. And then knowing when to refactor and how to make the layers properly and how to keep things, you know, simple enough that we can make progress, that's really important. But the top-to-bottom stack is amazingly complicated, especially when you're going into bigger systems. You know, I think the challenge of doing small inference models on single chips is sort of getting solved, but doing very large models, inference, training, and, you know, the dynamic computational systems that we're building with pretty big scaled systems, that's a lot of work.

Yeah. So, you know, when we built Zen, we had to build our own chip-to-chip. And the first, you know, at the desktop and server level, we made a chip which actually had CPUs, PCI Express, and memory controller. But to accomplish the roadmap, we built it so you could hook two or four of them together. Right. The second generation of Zen, they built, you know, basically the IO memory die, and then which had all the interfaces for all the memory because people, you know, they wanted to have a scalable memory footprint, but we did that with numbers of DIMMs. They wanted always a fairly large number of PCI Express lanes, but a variable number of CPUs. And then they went to a two-chip architecture with a CPU die and a base die. But they're still building a custom I/O die. First with a bunch of wires, and UCIe, there's a standardized, you know, chiplet PHY, which is now being used. But the architecture around, like, how do you define a chiplet, how do you build it for test, power, power management, you know, negotiation, that's something that's actually fairly complicated, and building a framework for that. And we looked at groups, you know, working as committees to go do that, and I was not really satisfied with the speed of iteration and then the number of technical problems you have to solve. You have to solve a thousand problems. You can't meet once a quarter to talk about them. A thousand problems needs, you know, a team working and meeting every day to solve them. So we built our own chiplet interfaces with, we picked a bunch of wires definition because we really liked the team that was building it. They had a really good power and speed trade-off, but they're also really flexible and open to go work with us as we solve those technical problems. And then Tenstorrent acquired Blue Cheetah, and that's now part of our team. And I think that's going to work out really well for us because now I can guarantee multiple generations of chiplets to be compatible, both for me and for some of my, you know, partners. So I'm really into that. But building that chiplet frame and making it work, and then building the collateral for that, we named that Open Chiplet Atlas. Atlas is a thing that holds up the world. And there's way more to it than a spec. It's not a PowerPoint problem. It's, you know, there's all this IP to support all the different features, whether it be test or power management or negotiation or, you know, like configuration of the actual transport methods. So we built that and we published it. We published support IP for it. We're publishing verification and performance environment for that. And like, I really want that to be open. Now it's open in the way that we're building it for ourselves. We're guaranteeing it works. We're publishing the collateral that goes with it, and we're really open to input and collaboration on it. But it's not currently like an open committee where people go vote about it because I think really in the next couple of years there's a lot of innovation that has to be done, and it needs to move pretty fast.

Yeah, that's the traditional thing, and it's, and you can see like, like Ethernet and PCI Express have often worked, but sometimes the next generation was years late compared to the need, and, um, and sometimes they kind of accumulate features. Standards around like RoCE and Ultra Ethernet and stuff have not worked out very well. The committees kind of, you know, add lots of features and don't really, you know, deliver like the right minimal set. We saw this happen with CXL. Like a whole bunch of stuff in that spec is pretty good, but it became way too big as a spec, and people did too many different things for the compatibility. So, you know, I'm always interested how these things work. Like JEDEC has often worked really well for DRAM definitions, but not every time. And also some of the features that we needed in DRAMs never really happened. So.