I think you'll like the first meme then, Chris, because Rohan asked me a few days ago to present and I threw something together like 20 minutes ago. But this meme I saw on Twitter today or yesterday and it was really good because it encapsulates how ridiculous programming GPUs is. Half of these I had to look up myself because I'm not actually that great at it, but the one thing I did know is it didn't call out Cutlass. But yeah, there's a ton of crazy complexity when you're trying to program a GPU really quickly. Check out Cutlass, check out the YouTube video that's linked there that's on CUDA Mode. Other thing I was just going to talk about was a bunch of GPU data centers because I thought that was cool. Let's talk about them. This is in Arizona near the airport. Each of these buildings is 48 megawatts. When you build a data center, you design something and then you stamp it everywhere. Microsoft has like over a hundred of these data centers across the US, each of these rectangular buildings. But when you think about, hey, what do you do if you're OpenAI and you've already trained GPT-4 on like 20,000 plus GPUs? The next step is 100,000 GPUs, obviously. So what exactly does that involve? There's these three buildings, each of these buildings has 30,000, 32,000 GPUs in it. It's a ton of money. There's all this stuff about networking that I won't get into. But the interesting thing is, if you're OpenAI and you think that raising $7 trillion is actually a reasonable thing to do, what is the next thing you do? You build this data center that is not 100,000 GPUs, it's 175,000 plus of the next generation GPUs. Again, this is hundreds and hundreds of megawatts, extremely large. And if you're OpenAI and you have problems with Microsoft being slow at building things, what do you do? You also go to Oracle and you tell them to build you a 100,000 cluster of GB200s as well. They have many more data centers all across the US and other places that they're trying to build. I just thought that was interesting. You got to think about the dollars and cents here. Anyone who works on CUDA, I said it was super complicated, I couldn't even explain that meme to you. I think there's maybe like five people in the room that can explain every single item in that meme from the beginning. I think PR is probably one, but I don't know, maybe Tree is probably another one, but I don't know who else could. But the interesting thing here is these data centers. That Oracle one that I showed you, it's like $12 billion that OpenAI is going to pay Oracle over four or five years. Think about what does 1% efficiency mean then. If you think about Nvidia shipping a million GPUs a year, this year, like data center GPUs, H100s and then early Blackwell, you have, hey, Tree just increased the efficiency by 10% of every GPU in the world. That's like $8 billion plus. Tree is a G and he gave it away for free. So the motivation here is really, be more efficient, because that's a lot of money and a lot of power that's just being burned uselessly if you're not efficient. Also just throwing in some other slides, Elon's crazy, he's building like 50,000 GPUs in a factory. He's also extremely crazy in terms of buying an old factory in Tennessee and converting it into a data center. None of these data centers are green, even though people tell you they're green, because wind power, renewable is like this, controller is like this, it's all gas powered. Don't let them lie to you. They'll buy these things called PPAs, which are basically you buy power from a renewable site but then the power is not necessarily what's feeding your data center and you can claim that you're green, but you're actually not. A lot of the power is wind or solar, but that does stop and so you have to have gas as a backup. So your efficiency is also about saving the world from not burning. You have people going crazy, building 100,000 clusters everywhere. These are just xAI and OpenAI. I won't talk too much about the networking stuff, I think that's sort of like in the topology and all that. Another thing is, if you can get data centers across the world to work together, at least data centers in the same state or region, Jeff Dean claims he's figured a lot of stuff out here. MLS is the conference, he claimed that they've figured out a lot of stuff. If you do, then that means that you don't need to build another 100,000 GPU cluster, you can just connect your old ones together, maybe. But the real thing that I've been working on recently, just for educational purposes, is all-reduce. Llama 70B is like 80 layers, and there's multiple all-reduces in each layer. When you think about it, if you want to do 30 tokens per second, there's like 5,000 communications you have to do between the GPUs every second. When you look to 405B, it's releasing on like July 20 something, so really soon. It's 400 billion parameters, 405, 410. It can't fit on a single GPU, you need the whole node, you need all eight GPUs. How do you have these communications within the layers without murdering your inference rate performance? If you want to achieve 30 tokens per second, it's over 100 layers, it's like almost 10,000 communications you have to do between the GPUs. So what I've been trying to learn or do, there's libraries already, but I've been trying to figure out how to do different techniques and profile them. I'm not going to use the gracious GPU that someone provided because I kind of need more than one today. But I also found a random place, not random, this is a Singaporean company where they stick the entire server in a vat of liquid that's made out of hydrocarbons and that's how they cool it. It was pretty cool. This is some early performance, trying different things out, NCCL, ring, tree, these are all different ways to do the all-reduce. Whether you go in a ring around all eight GPUs, so there's eight or seven communications, and then you broadcast it all, or if people aggregate up, or if you do all-to-all. I'm working on doing the all-to-all now. The reason this is really important is because the unit of compute is not an H100 node in the future. That's what it is going to be for Llama 405B, one server, you have to figure that out. But the next thing that's happening is the GB200 rack, and Nvidia is going to ship, majority of their shipments next year will be this rack format. That means you need to now figure out how to make 72 GPUs communicate over NVLink and schedule the communications and not just use Nvidia's crappy NCCL because it's really poor. That's really about it. I just didn't have a rhyme or reason, but that's what I'll be working on.
