Hello everyone. I'm Mark. This is a panel on AI hardware. I'm really thrilled to have some pretty incredible guests. So, here first off to my right there's Sharon Zhou who's a VP of AI at AMD. We have Peter who's a CTO of CoreWeave, one of the largest NeoClouds in the world. Over there on the right we have Nitan who's a CTO of Crusoe Energy, another one of the largest NeoClouds in the world. And then we have the chief poster at SemiAnalysis, Dylan Patel. And yeah, so hopefully this starts off the tone of the panel. All right. So I guess I just have a question to everyone here on the panel, which is so for us normies, right, that are used to begging for one or eight GPUs. How do these big deals get closed? Does a VP walk over to, you know, NeoCloud? It's like here's some scaling laws, infinite money. How do these multi-billion dollar deals actually get negotiated?
I don't know. I feel like right now things are pretty crazy and people who have plugged in two GPUs in their life get a huge deal. But how it used to be done at least last year is pretty easy. You build good stuff and you deliver to your customers and you prove that you can deliver quality compute and then you just keep doing it over and over and the deals kind of get bigger. I don't really think there's any magic to it except for delivering really high quality compute, supporting your customers, and being a great partner. Like any other business. Pretty unexciting answer but there isn't really that much magic to it, right? We've been building GPUs for a long time. We've been scaling aggressively and we managed to do it at high quality and that's why our customers trust us. We have our most recent deal with Meta which is significantly bigger than the deal we had with Meta two years ago. It keeps building as you deliver quality to your customers.
I think on the logistical side there's at least three moving trains. There's data center and energy capacity, there's the hardware accelerators themselves, you know, how quickly can you get them, and then there's a customer and how fast they want it. So those three things kind of have to align and each deal is slightly different in that way.
Yeah. But like my sense is there's still some sort of phase shift, right? Like if I'm asking for a GPU, no one cares. If you're asking for 64, okay, you wait in a queue. You ask for 100 million, that's when people are like okay well what are you doing exactly and why, what is the business justification. So at least for that last part, the business justification, has scaling laws really been one of the fundamental ways in which people do cost projection?
Doing cost projection, I mean I don't know, right now the appetite for compute is insatiable. Scaling laws and no scaling laws, it's kind of like all the compute that we can possibly build gets eaten up. And a lot of that is, a lot of our demand is not really around training and scaling on that side but from inference and it's just usage driven. I would say that most of our demand is really usage driven more than like oh we need more compute to train big models. And that purely comes from AI actually being useful and people using it.
Yeah, that makes sense. Inference is presumably significantly easier to project because you're like I have this many users, this many queries, simple Excel math can make that make sense. But trying to predict, if you look at training compute only, trying to predict how much training compute are we going to need in two years, I think everyone is going to be wrong. And so far the answer has been more. But if you try to predict two years ago how compute was going to look today, I was wrong because then reinforcement learning came in, it changed the shape, it changed the networking, now we also need a bunch of CPU compute. So the best strategy there is to be flexible, be able to build flexible data centers, be able to be as late binding as possible with what you put in them, how you configure them. And that's kind of how we do our planning. We know that we're going to need a lot of capacity. We know that we're going to put roughly these shapes in them. And then if that's going to be a big rack scale or not, we want to bind the things as late as possible. How much is air, how much is liquid, because I expect we have reinforcement learning now, someone is going to come up with some new paradigm that's going to change things and change how the compute shape needs to be. So yeah, if you can be flexible, if you can be ready when the customers need it, that's how you get the customers.
A lot of these deals are also just heavy vibes. Because you go look at OpenAI signing a hundred billion dollar deal. It's like their revenue is like 10 and they don't have that much money, right? But you believe, as a company, whether you're CoreWeave or Oracle or Crusoe, whoever you are, they're all involved in OpenAI clusters. You believe that they're going to be able to have the money that they don't have today. And then because these timelines for power for data centers are so far out and you don't know the exact mix of networking and CPUs and GPUs, you can't really price it. So then you kind of just have an imaginary number that is in there. Like, oh, $13 billion a year per gigawatt, that's reasonable for today, but what happens with the hardware in the future, we don't know. So OpenAI is signing deals in 28, 29. And a lot of it is like, well, here's a contract, but at any point we're going to change the networking, we're going to change the GPUs that we're going to put in it. We have no clue. Is it our own chip? Is it AMD? Is it Nvidia? Is it TPUs? Who knows, right? A lot of this is just vibes, I think, too.
Yeah. There's a lot of flexibility. The further out you go you need to build more flexibility because you don't know exactly the timeline for the next chips, you don't know what network architecture, like are we talking RoCE versus InfiniBand. So a lot of the contracts end up being flexible the further out you go.
I mean higher up stack, it's important to listen to your customers and listen to every single person in the ecosystem, every single startup in the ecosystem to kind of hear what are they inventing next. Exactly like okay a new frontier lab is forming up right now in stealth, they're inventing a new type of RL that will dramatically change how we should actually design the ROCm software but also possibly different hardware elements in the next generation of the chips and that's something that we take into consideration as well. So I think it's just listening a ton to everyone. And I like the part about delivering too because then you could actually snowball your deal.
I mean so speaking of that, a big theme here in a lot of your answers has been there's this insatiable demand. There's a lot of need for predictability of being able to get compute. I'm old enough to remember when there was three cloud vendors that were eating the world and then in the last couple of years we've had this remarkable rise of NeoClouds. And whenever I've asked how come, people give me this handwavy answer of like well we're just focused on GPUs. And I'm wondering if you all can give me a bit more specific and understanding of the nuances here and how come there was this big wedge in the market that you all created.
In the past, AWS was dominating the market, right? They were the biggest player. In GPUs, they aren't. They're not even top three, it seems to be. And a lot of this is driven by the fact that they made regular cloud computing super efficient through their custom NICs, Elastic Fabric, ENA and EFA. They made it through their custom CPUs. They make their own SSDs. There's all these things they do super custom optimizing for database workloads, storage workloads, and standard EC2 sort of CPU type stuff. GPU workloads are very different. They run across many nodes. The optimizations you have on networking are very different. Security versus latency versus speed. All these trade-offs are different and the cost is very different. And then also the software that Amazon delivers is irrelevant relative to what happens with a GPU cluster. And now GPUs are 10 to 50 times less reliable than a CPU. They break all the time and you need to figure out how to fix it. And it's a reality that your training and inference just has to deal with stuff breaking and your cloud has to constantly be able to fix it. And so there's all of this over-optimization and large infrastructure and team optimizing for the wrong problem for AI, which has completely different vectors and that's what some of the folks on the stage are really focused on.
Yeah, exactly. I'm going to tie into what you just said. I agree with everything Dylan said. A lot of the technologies they built for many years are a poor fit for AI. Like the proprietary networking scales poorly especially when you get above a thousand GPUs. And the next one which is really important from a software stack perspective which is very hard for legacy hyperscalers to break out of is what I call the blackbox model where they don't look at infrastructure kind of full stack. And they build to prevent failures that things shouldn't fail. Your CPU VM on Amazon, it will almost never crash, you live migrate, all that. That's not really what you want to build when you build an AI supercomputer. You don't want to give up half of your InfiniBand bandwidth to make sure you never have a failure. You'd rather have that failure and then handle it gracefully. And their stacks with the hypervisors in the middle and completely opaque into the customer scheduler, they're really poorly built to handle a world where you kind of want to embrace failure and handle it gracefully. Everything falls apart. It's a bad customer experience because now you as a customer have to go and find out which GPU is bad, report it back to the cloud. And by building a stack from the ground up which is focused around these use cases and can do the lifecycle fault detection optimizations throughout the entire stack, you can build a customer experience that is very hard for them to retrofit on top of how they built their things.
Yeah. And I would just piggyback a bit. A lot of what you were talking about of over-optimizing, I think it extends to the network. You don't need three or four redundant links. You don't need three or four backup power options. All of that adds to the cost. Especially with training, a little bit of downtime is okay if you're going to get a significantly reduced cost. We also go and build data centers in places that no one really ever thought to before because they're so hard to get to or you can't get the network there. And so all of those things add up. And then there's also the speed element. So you probably have people here that just raised a seed round of $10 million or $100 million and they're going to go spend a lot of that on compute and they dial up a big cloud company and they're saying, well, we serve Fortune 500, Fortune 100, get in line. Whereas if you come to us or these guys, we'll get really creative with you. And you can mix in some vibes that Dylan was talking about and you can get deals done and up and running really quickly compared to some of the incumbents.
I think something Nathan said that's really important to highlight is that speed is the moat. It's something Anushia at AMD says all the time, speed is the moat. And the thing about all the hyperscalers, Google, Microsoft, Amazon, is they just have all these organizations that care about the power being renewable and they have a very specific spec for the data center. And they only want to be in tier one markets, right? Like Virginia. I love Virginia apparently. But that's not where CoreWeave and Crusoe are really building all these clusters. They're in the middle of nowhere of Texas or Wyoming. Of course they have data centers in the typical markets as well. But also they'll take former mining places and convert it to GPU capacity. And what's important is these hyperscalers are optimized around oh if I save 2% on my cost of construction or power that's a huge savings for me. And that made sense in CPU and storage where most of the cost was power, network, power and the data center and so on. Hardware was split evenly with that. Whereas with GPUs, they're so expensive. Because AMD needs to beat Nvidia down. And once they do, GPUs will get cheap enough, but right now GPUs and the networking that Nvidia sells is like 80% of the cost. So if all of a sudden your power costs 20% more, because you're using maybe not the most efficient power delivery method or power generation or you're generating it on site or your data center costs more to build because you're just trying to go as fast as possible, it ends up not affecting you. Let's say that 20% of the cost goes up by 20%. Okay, that's just a few percent in cost increase, but I got the GPUs online many months sooner. CoreWeave and Crusoe had GB200 up way before Amazon. And that's a testament to them going speed versus Amazon's hyper-optimizing around their specific architecture they've already built and cost. And of course these guys have good cost too. I don't mean to say they're expensive.
Just, this is my experience with reliability. I run PyTorch programs. If it's hanging, I do control C and reboot the machine. Sometimes I blow my power outlet in my apartment because I have 2490s at home, which is unwise to have a single power outlet. So what happens here? Give me a picture, I'm in a massive data center, things fail. Is there a control plane? Are you lugging around a monitor with a keyboard and laptop and monitoring the state? What does that actually look like?
Exactly. The customer calls a data center guy and I'll plug into...
So I mean I think everyone implements this a bit differently. So we have our customers train in either Kubernetes, train inference Kubernetes or Slurm. So we have a full stack solution for both that's kind of where we limit today. Then we have some tools for serverless RL and so on through our Weights and Biases products. But the experience I want to give you, and when you came to the speed as well, speed is not only how quickly can we get a customer GPU, but it's also how quickly can they be up and running and be productive. Time to first productive training is a metric we measure that a customer gets access to a cluster be it 64 GPUs or 60,000 GPUs, the first day I want them to be able to run a training job without fighting the infrastructure, without having a bunch of interruptions. So what that means is we spent the last five years building automations around that, trying to predict failures before they happen and that comes by looking at telemetry from all parts of the stack. Simple things like BERT and more complex things like looking at temperature trends in the Tj limit in the GPUs and graphing that with inlet temperature into the system to track thermal paste degradation. Using our entire datasets of hundreds of thousands of Hoppers and over 100,000 Blackwells to predict when failures happen before they happen. Take those nodes out of service from a customer and replace them. And that's all logical. In what you see in your Slurm cluster as an example, those are all backed by a pool of compute that is say 20% higher than what you actually have. So that we always have spares and we can switch them out logically. So whenever there is a fault in our case, you never have to talk to a CoreWeave person, we automatically detect it. We replace the node, you have a new fresh node, your job can restart and it's up and running. And preferably I want to detect them ahead of time. And then once from you as a customer the node is kind of out of production what happens to it then? Then it can be a bunch of different things. Automation goes in and checks that this is a known issue, something simple like in Nvidia world an XID79 GPU basically fried itself like a power regulator failure, then we would automatically go and RMA it. That means a data technician will go and unrack the server, carry it over to the corner where ODM or OEM sits like a Dell in a corner, take the server put it over there and then there's a ticket says this GPU is bad and here's the proof now you need to replace it. Then they go unscrew it and they put a new GPU screw it back. Then we carry our data center, carry the server back, plug it in, plug all the eight InfiniBand links back in. In terms of InfiniBand in multi-plane RoCE that's like 32 links. So there's a lot of cables and then we just start a full burning process again because then we need to check now did the data center technician actually connect all the 32 cables correctly. There's a lot of manual steps in that. So you kind of need to burn in again, make sure everything is good and only then does it go back in a pool that can be given to a customer.
So Peter told you how it's done. You still have to do it faster than him.
So he seems very confident.
All right. So switching gears a bit, one thing I was very curious about is your relationship to the computer architects, different hardware vendors. So for instance, you look at workloads like DeepSeek, they're very memory hungry. AMD introduces more VRM, larger L3 caches. What does that relationship look like when NeoClouds are looking at what customers are doing, how hardware is changing? When do you tell them please do this, stop doing this, more of this? How does that collaboration work?
I think we just talk to a bunch of different groups and researchers at the different frontier labs but also at startups and get a sense of what workloads they're running, what workloads they're about to run and we try to mimic that as much as possible with different similar models or some kind of blueprint model. And so I think a lot of it's around testing and making sure that we also have the customers' trust so that we can actually get those blueprint models ahead of time and then be ready at day zero. And then for us it's also about open sourcing a lot of our software so that people can audit it and run it on their own as well. And be able to see that okay this actually works on our chip and this next generation of chip as well.
Yeah, I think there's also models come out of nowhere. DeepSeek came out and it was slightly too big to fit on an H100. So everyone wanted to run it. A lot of people had H100s, they couldn't do it. They did fit on an AMD MI300, they just had a lot more memory. So suddenly the demand for that skyrocketed. So yeah, as the models get bigger, you could tell better than we can how you think about how much memory to spec out for each hardware generation, but the fast pace of the industry is very unpredictable.
Yeah, I think it's super tough to predict and the flexibility comes in there, right? And there haven't been that many great industry tools to understand which accelerator is best for the best thing. That's one of the problems our customers see. Now there are some industry tools coming out, there's InferenceMax from Dylan that tries to help us understand which accelerator is best at this job and that's something that I think a lot of us have done mostly internally so I'm very happy to see that coming up in the open market.
I think a lot of these investments that clouds do, there's two poles. One is customer wants something now and they go deploy it. So a lot of times they want Nvidia but now something like DeepSeek rolls around now they want AMD because that fits. But then the flip side is this is a five-year, six-year asset. And no one is going to be running DeepSeek 670B model in a year. There's going to be something better probably from DeepSeek themselves or they're going to run something from Alibaba or whoever. There's just better models. So the reason that in that point in time why AMD was better is not something you can index your entire six-year investment in. But it is important to understand where hardware works better versus not. And so we launched InferenceMax which is an open source nightly benchmark of multiple models. It sweeps across all the parallelism configs and batch size, concurrency. And you get your interactivity, your throughput per GPU, and then it also shows you TCO per cost per million tokens based on SGLang and vLLM. And what's interesting is it's not just a one, it's actually a very complicated problem. It's not Nvidia's better or AMD is better. It really is what kind of model architecture are you running? AMD is better in certain cases, maybe not even per GPU but definitely once you factor in the pricing because Nvidia murders you on pricing as we all know. And then the flip side is sometimes Nvidia is better. And so that oscillates back and forth where the cost per million tokens which is often what many people in this crowd care about at the end of the day is what's my cost per million tokens on the open source software like SGLang and vLLM, it will flip back and forth. And so it's really important to recognize this when you're making your decision on to rent.
I guess outside of pricing, what are the margins like let's say for Nvidia and AMD respectively today.
So as far as Nvidia selling a Blackwell GPU they mark it up roughly four and a half x which is what it cost them to make it. So if it cost $5,000 to make they sell it for $30,000 roughly. AMD's is generous. They're only like two, two and a half x. That's pretty nice. What's something interesting is in the cloud market though, Nvidia because they spawned all of these clouds they made investments in many clouds, CoreWeave of course, but together and many other folks they've invested in. There's over 100 NeoClouds with Nvidia GPUs and so the clouds generally tended to have a more competitive market and their margins weren't always the best for small scale deals. Obviously the big scale deals there's only a few clouds that can do them like Crusoe and CoreWeave, but for the small scale deals where it's 100 GPUs, there's over 100 GPU clouds that can do this. And so the margins are really bad. Whereas AMD a year ago, there was almost no clouds offering AMD. But now there's I think six or seven different clouds offering AMD now. And so there's a competitive market again. And when it was just one cloud or two clouds offering AMD, their margins were really high. And so that didn't always flow through to the end customer. And it kind of sucked. But then AMD did a good job of getting Crusoe and many others to also TensorWave and many others to deploy AMD GPUs. So that cost savings for the cloud also gets passed on to the user.
