Dylan Patel - Only inference will keep AI clouds alive in three years

Yeah, absolutely. Thanks for having me.

Thank you. Thank you so much.

Yeah. I mean, so to get the same level of intelligence, the cost has collapsed drastically. Like tokens getting cheaper. That's because you can get the same level of intelligence with smaller and smaller models, which then means you can serve many, many, many more tokens per H100. So while the cost per token falls, it doesn't fall enough to make… it falls less than the amount of tokens that come out. And so therefore you get this increase in, you know, sort of total dollars out of a GPU. And with the total dollars out of the GPU going up, then the price of the compute also goes up. So cost of compute is not falling. It's actually increasing at least for an existing GPU, but generation on generation. Of course, the new GPU keeps getting, at least in terms of performance to how much it costs, it's getting cheaper. So, you know, sort of there's two ways to look at it. But that same GPU is going to be worth more a year from now because the models are worth so much more a year from now. They can put out so much more intelligence per GPU. So it's a good business to be in, building a neocloud. Yeah, it's a good business to be running stuff on top of compute.

I mean, I think, I think, you know, the optimization world is very, very complex. While there's a lot of open source software out there, whether it be VLLM, SGLang, you know, LLM-D, all these things, Dynamo, sort of AMD's ATOM, all these stacks exist. There's actually so many knobs that you can turn and dial in, and so many custom things that people are building on top, such as custom speculative decoding and so on and so forth, that really optimize your cost so you can take VLLM out of the box and not tune the hyperparameters. Or you can, you know, tune all these flags and add custom speculative decoding. And the amount of tokens you get out is, is like a multiple x difference. And so ultimately, you know, there's a lot of optimization to be done. And everyone sort of has to do it. Some people do it themselves. Some people will look to others to help them do it, but ultimately there's a lot of optimizations to be done. So we think agents are going to take over the optimization game. Agents are definitely becoming more and more involved in optimization. So at SemiAnalysis we have InferenceX, which is the largest open source benchmarking and performance optimization library out there. So you can take our containers, download them and then run them and they're fairly optimized. There's obviously a lot more to be done. Otherwise, you know, companies wouldn't have a business. But, you know, you can… a lot of our optimization, we actually just kick off agents that test various configurations and do the whole sweep, right, of hyperparameters, right? So whether it's, you know, sweeping on, you know, different batch sizes, different hyperparameters of the inference engine, different ways to do offload and caching and all these things. And so it takes time. Obviously the agents will try something, test the performance and then loop back. And so that'll happen over and over. You have to pay a lot of special attention to your harness. But yeah there's a lot… a lot can be done there.

Yeah. So a lot of people think an H100 is an H100. Blackwell is a Blackwell. That's not really true. There's so much that you can put on top. There's performance differences based on the software that's on top. And then also, you know, uptime and time to program, right? So you could get an H100 server that's bare metal and you can do some stuff with it, especially if you're skilled. You can do a lot more, but it's very different if you have the full life cycle reliability. What's the networking software and security look like? What's the cooling look like? What's the reliability look like? What type of network did you even deploy? You know, reliability is a big deal here because I think, reliability varies hugely between different neoclouds. Whether or not they have a managed Slurm or managed Kubernetes is a very big deal because it makes it a lot easier. Because the moment you get a cluster, if you don't have to install all this stuff versus having, you know, versus having a managed Slurm, managed Kubernetes, it's just so much easier and faster to run your workloads and do work. And, you know, does that neocloud give you spare nodes or do they just not give you spare nodes, right? I.e. if a node goes down, does it take them, how long does it take them to get that replaced and get you your nodes, right? So all these things are embedded in a cost. So we see a huge difference in pricing between the gold and platinum tier neoclouds and the silver and bronze and the underperformed. The price range is huge. So the software layer really helps increase the margins for the players, ultimately.

Yeah. So in our testing of a bunch of clouds we went through and we just found so many vulnerabilities, many of these vulnerabilities, you could just tell them, tell a model, hey, look for vulnerabilities and it would find vulnerabilities. It'll start looking into other people's storage. It'll start looking into other people's nodes, somehow got access to other people's Slurm. Obviously there's stuff we did on top of that, right? So if the models can find it or we can find it with various, you know, there's a lot of containers or different aspects of the software where there's a vulnerability, it's known and it's disclosed and the vendor hasn't updated it. And it's like, okay, a huge chunk of neoclouds suck at security. And that's really bad for their customers because it's not just, you know, it's their data is going to be vulnerable to anyone on that cluster. Anyone can go in and just start extracting data from the people on the cluster. And we've found instances where even it was like, we got a cluster and we found the vulnerability. We looked, and we're like, what the hell? There's like, national security firms on this. Literally a country's national security apparatus. Multiple agencies were on that same in that same data center we just had access to were like, this is really bad.

So neoclouds are really bad, and, you know, some of these neoclouds are the sovereign AI, right? And so, hey, every country is going to have a sovereign AI cloud and probably the security apparatus, the national security, the government, healthcare, all these like controlled and regulated industries are going to use that cloud. Not maybe, you know, the international ones. And then they end up with, you know, all these security vulnerabilities. So it's something that's really dangerous. And, we're trying to shed more light on it and show people, you know, what's wrong. We've been open sourcing some software, and we're going to, you know, just a little bit so far, but we're going to open source a lot more software on, you know, to make sure you're up to date on everything and things like that. Right. And then that's just like the table stakes. There's actually so much more security in terms of how do you set up your keys and make sure those are secure and your storage. And so anyways, like it's a Wild West out there.

Yeah. The ClusterMAX CLI is a small fraction of what we've been doing. But we were like, you know, let's open source at least something. And so we thought the CLI made a lot of sense, just, you know, check, check. Everything's up to date.

There's a lot of other stuff that we're going to open source over time, in ClusterMAX, but also a lot we're just going to keep proprietary because we do ultimately need to, you know, you have a business.