Hi listeners, welcome back to No Priors. Today I'm here with Dylan Patel, the chief analyst at Semi Analysis, a leading source for anyone interested in chips and AI infrastructure. We talk about open source models, the bottlenecks to building a data center the size of Manhattan, geopolitics, and poker as a tell for entrepreneurship. Welcome, Dylan. Dylan, thank you so much for being here.
Thank you for having me. I've been really looking forward to this conversation. And you're such a deep thinker about the space. And then also it's very odd you clearly have the Samsung watch.
Yeah. I got the Bling, the laptop, the fold. Yeah. Yeah. Tell me more.
So part of the origin story is that I was moderating forums when I was a child and my dad's first Android phone was the Droid, right? And for some reason, I was obsessed with messing with it, rooting it, underclocking it, improving the battery life, all these things because when we went on a road trip, there's nothing to do besides mess around on this phone. So, I posted so much about Android that I became a moderator r/android on Reddit and many other subreddits related to hardware and Nvidia and Intel and all this stuff. But because of that, I've just always had Android. Now, I've had work iPhones before, but I just really love Android. If you're going to like technology, I'm not someone who pushes it, but get the best stuff. So, I have the ultra Samsung watch, which I think looks cool, and the foldy phone, right? It's fun. It's obviously different and weird.
No, no iMessage is attractive. What does it dominate at? What is it better at? Besides the openness of the hackability, I don't even hack that much stuff anymore, right? It's like what do you use your phone for?
I think the main thing is like you can have Slack and an email up on two different parts of your phone. I think that's probably the main thing. Or like you can actually use a spreadsheet on a folding phone. You cannot use a spreadsheet on a regular phone. And that's not even an Android thing. Like Apple's folding phone next year will be able to do that just fine. And I'll have no argument then. Yeah. But I just like it. You know, people have their preferences. People are creatures of habit.
You got to look at the GPU purchasing forecast. Yes. On a sheet on your phone. Yes, I do. I do. No. Like it's like someone's telling you numbers. You're like, 'Wait, this is like slightly different than my number.' Right. Like Okay. So, we have a week of big rumored announcements coming up. Tell me your reaction to the OpenAI open source model.
In theory, it's going to be amazing, right? Like I assume this is releasing after it's released or Yes. So, that's okay. The open source model is amazing, guys. Like, I think the world is going to be really shocked and excited. It's the first time America's had the best open source model in six months, nine months, a year. Llama 3.1 405b was the last time we had the best model and then Mistral took over for a little bit if I recall correctly and then the Chinese labs have been dominating for the last like six, nine months right so it'll be interesting it'll also be funny because like the open source model probably won't be the best for just regular chat because it is like more reasoning focused and all these things but it'll be really good at code and so I'm excited for that yeah like tool use although that's like going to be confusing like how do you use the tools if you don't have access to open tool use stuff, but the model is trained to do so. That'll be interesting for people to figure out. I think the last thing is like the way they're rolling it out is really interesting. They accidentally leaked all the weights, but no one in the open source has figured out how to actually run inference on it because there's just some weird stuff in the model with the architecture like for bit and like the biases and all this other stuff. But what's interesting is other companies drop the model weights and say go make your own inference implementation. But OpenAI is like actually dropping the model weights and like all these custom kernels for people to implement in inference. So everyone has a very optimized inference stack day one.
And they work with partners on it too. Yeah. Working with partners on this. But this is very interesting because like when DeepSeek drops it's like well Together and Fireworks are like yeah we're the best at inference because we have all these like people who are really good at low-level coding. Whether it be like Fireworks with all their like former PyTorch Meta people or Together with like you know Tri Dao and all the you know Danfu and all these like super cracked like kernel people they have like higher performance right but in this case like Open releasing a lot of this stuff so it's interesting for the inference providers too like how do they differentiate now.
Yeah, I mean my premise on this is in the end a lot of the model optimization performance layer is open-source and it's a commodity. And it will end up being like a fight at the infrastructure level actually. And so you know all of these inference providers like as you mentioned you know Fireworks and Together and Base10 and such they compete on both dimensions and the question is what's going to matter in the long term.
Why would these model level software optimizations all be open? They haven't been open so far and the advancements are so fast right like oh well I think that a bunch of them have been partially open and I think OpenAI is also pushing for them to be open as Right. And so I think there's a lot of force in the ecosystem to open source from both like the NVIDIA level up and from the model providers down. Right. And so I think today these providers all fight on that dimension. Yeah. And they also fight on the infrastructure dimension. And I think infrastructure is going to end up being a bigger differentiator.
That makes sense. You can't open source your actual infrastructure, right? You just have to have the network and you have to run it, right? Yeah. Yeah, that makes a lot of sense. Although like I see today the inference providers have such a wide variance right like the ones you mentioned are on the leading edge especially like Together and Fireworks I think are on the leading edge of like their own custom stacks all the way down to like there's a lot of people who just take the out-of-box open source software.
Yeah, I think there's no market for that. But those guys have just Yeah, I agree. There's no market. It's like commoditized. They have really really way worse margins than the people who are very optimized. When you see Nvidia trying to open source all this stuff around Dynamo and OpenAI and all these other people are trying to open source stuff, but the level of optimizations is also like really really large like caching between turns and caching tool use calls and all these other things. And it's not just like a single server problem like it's like the DeepSeek implementation of inference is like 160 GPUs or something like that like that's over $10 million of hardware and then that's just one replica and then you'll have a lot of replicas and you share the caching servers between them. So like seems like just the orchestration of that but also the infrastructure of that it's a very large amount of infrastructure. I don't know that's interesting thought that that would be completely commoditized optimization layer.
Well, I think that there's optimization at the single node level and then there's like the system software where you can like orchestrate this and I think that owning the abstractions for it and having people use your tools and more sophisticated teams to do that optimization is like very ugly distributed systems problem. I think that will matter.
Okay. Yeah, I could agree with that. I could agree with that single node is not necessarily Yeah, I agree. let's move a you know out and a layer down like what does having access to an American open-source model mean or just more and more powerful like open source AI models mean for the application ecosystem I mean I know like a lot of people and some enterprises are really iffy about like using like the best open source model they're like worried it's like there's nothing wrong with them today there's nothing in them today right you know there's the worry that one day they will how do you check.
I mean you don't like you can just vibes it out like they're like competing with each other to just released as fast as possible, right? Like like DeepSeek and Moonshot and all these other you know Alibaba, etc. Like they're competing to release as fast as they can with each other. The Alibaba teams in Singapore like I don't think that they're like putting Trojan horses in these models, right? And like there's some interesting papers that Anthropic did on like you know trying to embed some stuff in models and ended up like being detectable pretty easily. again like I don't know how to, you know, I'm not too much into that space of interpretability and like eval but I just don't think that they are right. It's just a vibes thing. But some people are worried that they could be or they're just like iffy like oh I don't want to use a Chinese model. It's like well fine but now you're going to go use a service that is backed by a Chinese model which is fine like you know like but they you know they're fine with that. They just don't want to directly use the model. I don't know. I think it's interesting for some enterprises who are still stuck on Llama, but it's mostly just really interesting because it continues to move the commodity bar up now with this tier being open source. And sure, like probably won't be like drastically better than Kimi, but Kimi is so big, it's so difficult to run, like people aren't running it, whereas the OpenAI model is like relatively small, so you can run it without being like gigabrain at infrastructure.
you end up with that commoditizing so much more of the closed source API market. I don't know. I think that's just going to be great for adoption, right? Like yeah, one of my hopes is for our companies that are doing more with reasoning, it is like they're still blocked on cost and latency. So this is something that I've found very interesting is that when we've been trying to build a lot of alternative data sources for token usage who's using what tokens what models where etc why and it's very clear that people aren't actually using the reasoning models that much in API like Anthropic has eclipsed OpenAI in API revenue and their API revenue is primarily not thinking it's Claude 4 but it's not in the thinking mode you know code is code being the biggest use case that's skyrocketing and the same applies to like OpenAI and DeepSeek and from what we see querying big users and other ways of like scraping alternative data because the latency issues because the cost issues especially right the cost is just ridiculous.
You're exactly so I guess my view is you're not allowed to have a tech podcast without saying the words Jevons paradox now and I think like I think the behavior is going to be like we see a lot of people use reasoning because it's so much cheaper to run if you take out a big piece of the margin layer and you make it smaller and so I think like we have a lot of companies that are at scale who are using it, but it's so expensive that they restrain themselves. For a long time, OpenAI was charging more per token for the reasoning model, right, o1 and o3, than they were for GPT-4o, even though the architecture is like basically the same. It's just the weights are different. And there's like some reason for it to be a little bit more expensive per token because the context length is on average longer. But in general, like it made no sense for it to be like was it like 4x the cost per token? That didn't make any sense. And then finally, they like cut it. But for a long time, not only was it like way more tokens outputted, it was also a way higher price per token even though and they were just taking that as margin because they could, right? Because they had the only thing out there.
Yeah. And then, you know, DeepSeek dropped and Anthropic and Google and others started releasing models and it like, you know, commoditized quite a bit, but this is going to just like kneecap like like take cut everyone off at the hip, right? And bring margins down again. So, who has an API business, you mean? Yeah. Yeah. For API for models that aren't like super leading edge. What do you think evolves in the sort of Neocloud layer over time?
It's funny. Every day we still find a new Neocloud. Like we have like 200 now and still every day we find new ones, right? Like should they all exist? Obviously not, right? Like so to some extent it depends on what the Neocloud business is, right? Like today there is quite a bit of differentiation between the Neoclouds. It's not just like buy a GPU, put it in a data center. Otherwise, you wouldn't have some Neoclouds with like horrible utilization rate and you wouldn't have some Neoclouds who are like completely sold out on four, five, six-year contracts, right? Like CoreWeave, for example, who doesn't even quote most startups or they just give them a stupid quote because they just like I don't want your business or like they want a long-term contract, right? Which a lot of people don't want to sign. And so like there's quite a bit of differentiation in financial performance of these Neoclouds, time to deploy, reliability, the software they're putting on top, right? Like many of them can't even install Slurm for you. It's like what are you doing? like and you should have some sort of like so very low level hardware management. Yeah. Yeah. It's like very and it's like to some extent from the investor side we see a lot more debt and equity flowing in from the commercial real estate folks as commercial real estate has been really poor over the last couple years, few years. They've been starting to pour money into cloud space and obviously the return profile is quite different because it's like a short-lived asset versus like a longer lived asset. But at the end of the day like these companies they're okay with a 10 15% return on equity, right? and over time that falling that is not okay for venture capital, right? And yet a lot of these Neoclouds are backed by venture capital. So a lot of these companies will fail either because it no longer makes sense for them to continue to get venture funding or they end up getting out competed because they just can't get their utilization up unlike you know some other clouds right like the CoreWeaves and Crusoes and such of the world right so there's sort of like a rock and a hard place for a 100 of these Neoclouds and there's many of them who are like oh no I purchased these GPUs I have a loan it cost me this much and because my utilization is here I'm like burning cash, right? And they should at the very least not be burning cash, right? And so some of them are like, you know, they're desperate to sell the remaining GPUs, so they go out to like, you know, companies and give them insanely low deals. There's some startups who I really commend because they've like really figured out how to get the desperate Neoclouds to give them GPUs, but those Neoclouds are going to go bankrupt at some point because their cash flow is worse than their debt payment. But at the end of the day, like there's going to be a lot of consolidation. There is going to be differentiation, right? There's a lot of software today. But we have this like thing called ClusterMax where we review all the Neoclouds and major clouds and it's like actually some of these Neoclouds are better than Amazon and Google and Microsoft in terms of software in terms of uptime and availability or however you uptime availability reliability network performance like there's just a variety of things that they don't have all the old baggage but the vast majority are worse and we measure across like a bunch of different metrics including the ones I mentioned and security and so on and so forth but our vision of like ClusterMax is that it starts at like a really low stage today which is like does the cloud work and how long does it take the user to like get a workload running because you have Slurm installed or you have Kubernetes installed and you know your network performance is good and your reliability is good and it's secure right like these are like table stakes like what we consider gold or platinum tier today will be just like table stakes in like you know six months a year a couple years there'll be a whole layer of like software on top and then it's like do Neoclouds build this software Right. And some of them are right like Together Nebius are offering inference services on top. Right? So they're saying, 'Hey, we actually want to provide an API endpoint, not just rent GPUs.' And CoreWeave rumored by The Information to be attempting to buy Fireworks for the same reason, right? Like do you move up or do you just slide down into like I'm making commercial real estate returns or you have to go crazy, right? Like Crusoe is like we're going to build gigawatt data centers, right? Like okay, there's no competition there. There's like a few companies doing that, right? So it's very different. So you either have to go like really really big or you need to move into the software layer or you just make commercial real estate or you go bankrupt, right? Like these are the paths for all Neoclouds.
