I'm here with Clive and Riner. You two have both come over to my house often to talk about chips and hardware software codes, but for some reason James Hills decided to stick a camera in her face this time. Introduce yourselves.
Yeah, I'm Riner. I'm CEO at Maddox. We make chips for large language models. And then before that, I was doing inference stack optimization at Google for all.
Yeah, it's great to be here. I'm a guy on X that likes to talk about hardware and software code design.
Yeah. So I think one of the things you two have taught me a lot about is just this whole hardware software design code space, right? So Riner, you wrote the Palm paper and I thought that was the best paper of like 23, maybe 22, I don't know when it came out, but...
We had to stop publishing after that.
Yeah, that was the last paper that Google published that was like valuable in infra.
Yeah. And I thought that was super interesting. And then Clive, you worked on a number of really interesting things. And you're co-designing for a completely different space before.
So code design is when we can not only make the software better or the hardware better in isolation given just like a fixed workload or a fixed piece of hardware. We want to make both better and so we might want to change the hardware in a way that makes some sacrifices today that will reap rewards in the future once we can design for that future hardware.
The thing that always confuses me is like when you're doing code design, what exactly are you doing? Like, are you just changing the shapes of the map moles or like what kind of optimizations even exist in this space?
I mean, I think the first thing is the sacrifice is the key point, right? What is a typical ML research paper? It's I've made some change to my model architecture and therefore all the metrics look better as a result. That's the architecture thing. You can do that without code design. That is just ML research. It becomes code design when you say actually some of these metrics are getting worse. But that's because these are metrics run on the current generation Nvidia GPU. And so maybe I should look at a slightly more generalized metric like what is the gate count of this architecture or what is the energy cost of this architecture? What's the intelligence per picojoule of this thing? And then so if you look at that more generalized metric, I think that sort of starts to approach giving you the flexibility to make all of the changes across the stack rather than just localized to the current generation of GPUs. So I think very simple things are to take a case study like if you look at the difference between a swish activation function, evaluating swish requires evaluating some exponential functions, those require function lookup tables, polynomial approximations, there's a lot of multipliers in them, can be quite energy intensive. At the other extreme is to say ReLU is kind of the simplest activation function at all, compared to greater than zero is extremely simple to weight. And so maybe it's cheaper in hardware to use ReLU rather than swish, but maybe it's worse in model quality. And then now your question is which one should I use? I want to do that on the basis of energy cost rather than what one's faster on GPU.
The example I'd love to give is you can always make your systolic array or tensor core wider in order to get more arithmetic into one place. But a lot of models are just not that wide. So if you increase the size of the tensor core, you're not going to get any faster execution of that model. But if you tell the ML researchers, make the model wider. I know it's like not completely efficient for ML, but in net once you also account for the hardware efficiency gains, it will be better.
I mean, why not massively wide like one layer 10 million wide or something like that?
Well, if you can build a 10 million wide tensor core that can run at the same power as a 128 wide tensor core, like obviously that's the right choice to make, but obviously that's not possible.
So a bit of a philosophical question, right? You see papers like let's say DeepSeek V3, their attention mechanism was the first one at least of a public paper where the arithmetic intensity was equivalent to that of Hopper in terms of how much memory bandwidth versus compute for the attention mechanism. You look at certain papers from other Chinese labs where they're like, I think gen on gen, I don't remember which lab it was, it might have been Alibaba, it might have been someone else, but gen on gen the architecture was like very similar except they went from like 100 layers to like 70 layers and that just made inference like 20% faster, right? Are these co-designs because they're not really influencing the hardware. They're kind of looking at the hardware and the confines that it has and then making their model fit that hardware really well.
I think a lot of people call a lot of things co-design. But in no true Scotsman is this.
Yeah, I think in both of our narrow views, co-design means that instead of designing your model specifically for the hardware capabilities like DeepSeek has done very well, or designing your hardware for models, which Nvidia has done very well, you're looking at both at the same time and you're trading things off where you might actually make a sacrifice on model quality in order to get much more efficient hardware utilization, or the other way around such that end to end you're doing better.
It seems to be, and maybe just from the definition of it, you get hardware companies that do co-design, pure model companies maybe less so, or what's your read on that?
I mean, a public example is Trevor's eye, he posted a tweet screenshotting an image of an email and that email was like hey here's what we want for Hopper next to Nvidia. And Nvidia did implement many of these things. I don't know what amount but they did implement some of these things and then they released GPT 5.3 Codex. Is there more words after? Is it just any of the numbers?
Yeah. Anyways, they released this model and they say it's co-designed with Blackwell. Is that an example of a model lab with someone else's external hardware?
I think you do often kind of get these ping-pong relationships where you get feedback from the model designers and then the hardware designers implement it and then the hardware comes back and then the model designers get to take a look at it and then feedback kind of ping pongs back and forth. But maybe the larger ambition there is that we can look at both at the same time. How do you do that when the ML researchers don't even know what they're going to work on in like six to nine months and you're like hardware time cycles like the one I just mentioned is three years.
Yeah. I mean I would say hardware designers kind of have to predict where the model researchers are going and like maybe take a squint on what their model researchers are doing. I think it's possible to look at trends for sure. Recent trends, for example, seem to have been hey look, Nvidia has a whole bunch of control resources on all of their ends. Maybe we can figure out interesting ways to use them and do something more dynamic. So you can see people how they're responding to trends. I think long term you can predict the model is going to be exactly this shape. That's hard to do. But we're going to use more of the resources that are available and less of the ones that aren't is a pretty straightforward thing to predict, I would say.
One of the things I've heard from researchers is sort of some of these chips that seem quite well co-designed, they actually hate. So an example is like TPUv6e, Trillium I think is the public name, Ghost Light is the name that everyone actually knows. But anyways it's very common for people from DeepMind to be like oh this chip is really weird because it's got like a petaflop of compute and like 32 gigs of HBM. It's like quite a weird shape relative to the other chips at the time, right? It's got like 1/4 the HBM or something like that of a Hopper and only half the flops. And if you look at real utilizable flops, it's the same, right? It's quite a weird chip. It makes sense in many contexts, but I guess a lot of researchers hate it because it confines where they can do the research. Is there a way co-design can go wrong?
I mean, I kind of think you want to make a chip that people do hate to some extent. People don't hate a chip if it's got a very wide basin of operating points where it can work well. And I think that actually means you've left some resources on the table. So really you're looking for like you would like your model to use all of the resources at 100% efficiency or 100% utilization. But if you can do that and sort of leave some slack around then maybe you could have tightened something off and that is make a memory smaller because your batch size were too large or if you have multiple different ways to lay out a model on the chip then probably there would have been one that was most efficient and the fact you could do other ones is a missed opportunity. So, I mean, like I think it's a little glib to say that you want your users to hate you, but to some extent they should work pretty hard to actually get the most out of the hardware.
I mentioned one that's from perhaps your history. Maybe one that's also from perhaps your history is like, you know, the Dojo chips. RIP, right? Sad they're cancelled and restarted. Maybe I heard restarted. They are also like really weird, right? In terms of like, I guess it comes back to my question of like do people hate this, right? They're really really good at convolutional neural networks, there's like some really interesting data locality stuff but then like you have no memory bandwidth, right? It's pretty bad.
I mean one of the interesting constraints of chips is that they are two-dimensional and you do actually have to confront this fact pretty actively when you're designing an ML chip and that's why architectures like systolic arrays exist that are very naturally suited to a two-dimensional architecture whereas you also see architectures, many architectures that have like an L2 L1 cache that wants to be connected to all the different processors that are on the chip and that's maybe less suited to embedding into 2D.
I guess but in some way, Tesla's a huge buyer of GPUs, right? They continue every six months or so announce a big deal with Nvidia. Of course they have their Dojo chips. Sort of going back to the whole point of like you've constrained the user a lot, right? Maybe Dojo is not good at certain kinds of model architectures, right? Is there a point where you co-design too much and now all of a sudden you've left no flexibility on the table and you've burned the architecture into the silicon and you end up with something that no one can use for modern stuff because ML research is moving way faster in different directions.
Yeah, that's a challenge is that you have to as a hardware designer kind of predict the future and you don't know where the future is going to lead. I think the most reliable way to think about this is just like when you're doing ML research, you are thinking about scaling loss and you're thinking about what is the direction things are going so that I can launch a big run and then now we're thinking about it in terms of what is the direction things are going so that I can design a chip for where the models will be.
And then the other thing you said is 2D which makes sense on a chip level. Racks are three-dimensional though. So there's a point where that struck, right?
That's true. Yeah, you can definitely come with some very interesting network topologies in these racks. There's the Torus for TPU famously. Nvidia has a I guess you would call it a flattened butterfly.
Isn't it just switched all?
Yeah. All sorts of fancy names for the same thing. What else is next generation TPUs have dragonflies.
That's interesting. The Traniums, I don't know what the hell you'd call their architecture. It's like sort of a Torus. Now they've added switches, but there's like switches in different trays and like you can do all kind of, but like there's multiple. It's weird. It makes sense though once you train an Apple model.
Yeah. Yeah, I guess how does one like do you view it as just a spectrum of like I'm overoptimizing for a problem or not or how much of this is like the art of just like oh I think models will be there.
I mean I would say part of it is just you're providing the raw materials. You have to figure out like there's sort of a whole spectrum of where you can provide those raw materials. If you provide ingredients that are too small like individual gates that's what an FPGA is. It's 10x less efficient than an ASIC. So don't do that. Make the unit the grain size bigger. Like systolic array as the grain size. That's really pretty efficient at that point. And so pick the grain size appropriately. Think about which fusions should be in place and are naturally there. So maybe some operations always get fused together and so you can have them as a bigger grain size. And then just try and expose those things with as little overhead as possible. So I mean that's sort of just like a hardware-centric approach to it. It's not really saying necessarily what are the models wanting and predicting what the models are, but it's just like you can bound your downside by doing that. Like I'm giving you all the things. Maybe I don't give you as much of the things that I think you don't need, but mostly I'm giving you the raw materials to work with. So I think that's a pretty safe hedge. What you can do better than by taking it into account what you know about models is say well actually so sure we know that multiply and add should always be fused together into fused multiply add but maybe I also know something about attention computations and I can take advantage of the fact that there's always a softmax there I can do something special for that and so on. And so there's sort of one angle on this is just say bound your regret on the hardware design with as much information as you're willing to use about generally what I know about models.
