Everyone, I'm happy to have Dylan here. The first time I saw one of his videos was an interview when he talked about NeoClouds and why they have the right to exist. That was super interesting for me because the theme of today is all about what are the new infrastructure primitives now that agents are here. You articulated that really well at that point why Neoclouds are different from hyperscalers and why they should exist. So can you share any of that with us?
Sure. In the age of AI, the hyperscalers were a bit slow to move, right? Google, Amazon, Microsoft, a bit slow to move into AI, and so a whole new crop of companies popped up. There was a new low bar, right? There's no need for a lot of the complex software that Amazon, Microsoft, Google had built up, and a lot of this in fact slowed down AI. They had custom networks that were actually not that great for AI; they were more focused around reliability and storage traffic rather than doing an all-reduce across a network or something like this. So there's a lot of things that these large cloud companies, these hyperscalers, had that NeoClouds could just skip, right? And then build focused, optimized solutions and deliver much lower cost because they have much lower overhead, right? There's not 20,000 Google PMs sitting in a meeting at these NeoClouds, right? Although some of them have started hiring Google PMs and slowed down. The Neoclouds, they just moved fast on energy. They move fast on bringing up GPU clusters, and so they were able to carve out a market, right? And then those are the early ones, and then since then there have been a lot more copycats or followers, many who haven't succeeded, many who are succeeding, right? It really is a battle of who's the most competent.
And so are all the, because there's like, I think you said there's like 200 of these neoclouds or give or take, right? And so are there differentiations between them that you see? Are some just copying the sort of software stack that these first ones did? Are they doing something else? Anything that you've seen that have been successful and less successful across these NeoClouds?
Yeah, so there's a variety of things that separate them, right? So we have something called Cluster Max that ranks all the neoclouds. But we test for various things such as observability, reliability, networking, security, orchestration, all these sorts of things are different, right? Like does someone test for their GPUs to be working properly when the GPUs are idle for a user, right? That's active health check or a passive health check. Hey, are the fans spinning at the right speed or is it drawing the right power? Is there something wrong with the node? Is the network flapping or is there anything wrong with the network? Is the performance there, right? Like there's all these sorts of checks and tests because GPUs are unreliable. There's all sorts of stuff around the types of software you're on top of the GPU, right? Many people started off doing just bare metal, right? So when you go look at like, hey, Microsoft's first contracts with CoreWeave, they were all bare metal, right? You just SSH in and Microsoft would set up their own stuff. But as we've stepped forward, there's a lot more people want, right? People wanted Slurm installed. Okay, that's simple. People wanted Kubernetes installed. That's a little bit more difficult, but still pretty simple. People wanted Slurm on Kubernetes installed because it's easier to push out jobs and things like that, right? And now you're starting to see people do things like managed array services and all these things for RL. And so there is a class of NeoClouds that are building these things and there's a class of Neoclouds that are like, I don't care about that. I'm just going to build GPUs and rent them out at bare metal. And there is a difference in cost, right? Like the clouds that have the good software also tend to charge more. And it sort of returns back to the old school model where Google, Microsoft, Amazon had good software and they charge more.
Right. And so you see a lot of these cloud companies start trying to launch inference services and all these other things.
So similarly in that vein, I'll just get to the CPU ones where historically we have customers that ask for this with like OpenClaw that happened. So there's a bunch of people that are like, 'Oh, I need my sandbox or CPU box running for a very, very long time. Can you give me like a $5 Hetzner type, you know, offering?' And I'm like, 'Well, no, because that is a bare metal machine that has no incurs.' But when you have something that's a larger software offering, then there is more cost. So it's hard to compete with that. So I would assume also on the Neoclouds that the ones that are bare metal, as you mentioned, have a lower cost of sale than the software ones. Moving on, I was just curious about why the right to exist they had because it's sort of an analogized with what we do. But the real question is you brought this up, which is directionally what we're thinking about, is that CPUs are the new bottleneck. Like every investor, partner, everyone that I talked to historically was just like GPU, GPU, GPU, and now you had this big report about CPUs and I'm like, okay, yes, thank you. And so you reported like this year will be a bottleneck. So just give us a high-level TDR. Why is CPU now a bottleneck and what do you see?
Yeah, I mean for the first few years of AI, CPU was a real big lagger, right? Sure, you used it for some storage, you used it for some checkpointing, used it for pre-processing your data and pre-training. But it was pretty light. And then inference was also like the models just weren't good enough to be agentic. You couldn't, you know, go step by step by step. So there was no capability for a model to go and do actions and string them together. So then it wasn't really, it was like you would send a string and it sends a string back, right? Simple inference. And so there was not much need for CPUs. But then over the last few years, or last couple years, it's not even like last couple years, right? You know, Q* started, OpenAI had that and they had all that drama, but then finally o1 preview launched I want to say like 15, 16 months ago, feels like much longer ago.
Feels like a long time ago, ages, yeah.
But o1 was the first model sort of there, and we sort of had an entire flood of models, right? Before people would do simple things like check the output of a model with regex or something simple just to see if it was doing structured outputs properly, you know, for function calling and things like this, right? But as we've stepped forward, the checks on the model are much, much larger and they've been integrated fully into training, right? With reinforcement learning, right? Instead of just doing regex, you're doing like various classifiers. Instead of doing classifiers, you're doing like code unit tests and compilation. Instead of doing that, you're running agentic flows where it's actually calling databases and stuff, or you're interfacing with some environment that is heavy on CPU like a physics simulation or biology simulation. You just kept stepping into more and more complicated things where the model outputs to checks it, right? This environment, reinforcement learning environment, and then goes back and trains on it. And this loop has gotten tighter and tighter and tighter over the last couple years. And then more recently, over the last, let's call it really the last six months, you know, code agent revenue has gone from a couple billion to north of 10 billion in like a very short amount of time. And the horizon of these has also increased dramatically, right? Like Claude Code can work for like six, seven hours and it's like fine, right? And so in that time frame, it's doing all sorts of calls to databases, it's doing calls to, you know, at least we use a lot of cron servers, but you know, whatever it is, right? There's anything and everything it can go ping, check it out, scrape, and just work agentically on its own. And that's also requiring a lot of CPUs. So over the last six months, that's ballooned as well in addition to the reinforcement learning training loop getting tighter and tighter. And so over the last six months, we've seen the entire cloud market run out of CPUs, right? I don't know if you folks have interacted with GitHub a lot recently, but it's really unstable.
I think you're the third person to mention that today.
Yeah. So we've been checking GitHub's stats on like how often is it down, how often does it fail to commit, like whatever, right? It's terrible. And that's because Microsoft sold all their CPUs that they had spare to other people, right? Either internal use for their lab, but you know, not really, more so like external labs. They've signed deals with Anthropic and OpenAI. And so they just have like no CPUs left, right? And we've seen the same at many other firms, right? So whereas before you would have like many GPU servers per CPU server, and so you'd be like 100 megawatts of GPUs would be served by even like one megawatt or less of CPUs, nowadays the ratio is getting much, much closer, both for RL training and for inference, agentic inference. So then you've just seen everyone run out of CPUs. Amazon's volumes on CPU servers they're installing have 3x year on year, right? From this year versus last year. And so there's just no capacity anywhere and that's causing a lot of instability. Not just in GitHub but probably other places too.
I mean, a lot of infrastructure today, we talked about this, like every day it's normal to see that some infra provider, be it GitHub or being like an actual, not to name, like they go down, like it's become quite a constant thing. And so could be lack of CPUs but also scale of the workloads and whatnot.
Could be that everyone's infra code is vibe coded.
Yeah, it could be that everyone's super code is vibe coded.
Yeah, could be that bit as well. I don't think it's all but there could be parts of that that is there. And so what I see which was interesting as well is the number of CPU workloads or the number of that are running on us. So Daytona has basically three use cases. So it's like code and command execution. So if like a Cloud Code type of thing needs to run on a CPU. We have computer use cases which we see actually growing really fast. We've now launched today, we announced our Windows sandbox which runs on a CPU. So if you need an agent to do like on legacy software which is like all finance, all support, all that's all there. But also to your point like reinforcement learning, we have a bunch of people that usually use like Kubernetes now they would use us. But the interesting thing is the size and scale of these loads are insanely large and they're growing insanely large and we're like the smallest cloud in the world. So my interest to the question is like if we as a very, very small company have this much, what does that do at the large scale, right? And we've had, I'm curious if you have any insight in this, but just RL, let alone the long-running agents which is on their own, we see customers coming in, one of them ran a million, one single customer yesterday in the span of six hours like 1 million of these CPU workloads. So that's one customer. And so like how many of these customers are doing RL going to need this? I don't know if you have any insights on this but I'm just curious.
I mean, there are some pretty shocking metrics like a million vCPUs sounds insane. But some of the scale of the workloads and contracts people are signing are even like more ridiculous than that, right?
I'm sure that's because we're really small.
Yeah. So I guess when you look at like the Anthropics and OpenAIs of the world, they have fully just eaten the entire capacity of multiple clouds, right? A lot of the impetus for the recent Amazon and OpenAI deal was yes, OpenAI wanted money. Yes, they needed compute, but they also just went to Amazon and were like, give us your CPUs.
And before OpenAI's stack was pretty much only on x86 CPUs, but Amazon had tons of ARM CPUs and so they ported the whole thing, right? It's like anywhere, I'll port my codebase to get access, right? To access to CPUs. And so this is the level of engineering people are willing to go to, right? Because usually devs are too lazy to do anything and they'll just go get capacity elsewhere, but there just isn't capacity anywhere else.
Yeah. So I mean that's interesting. We're all x86. That's all we have. We don't have ARM at the moment. But then like there's those two which I know but also like Nvidia has their like CPU, other people building out their CPUs. There's also differences between these CPUs. Are there just going to be like these general purpose CPUs? There's other like you probably know more about this than I do. I'm just like super curious on the types of CPUs. Like now there's so many, there used to be just like x86 and ARM, that was it like for the most part. Now there's different types of CPUs. Is it just because everyone's run out or are they like better at some things than others? Are there anything?
I mean, usually what happens like when there's a gold rush is the person with a broken pickaxe can also sell their pickaxes.
Yeah, I mean, the CPU market is like quite dynamic, right? You know, it was mostly Intel, AMD. I assume you mostly used Intel and AMD CPUs. Both of them have said they're like fully sold out. They've sent price increase notices to their customers. So they're not even competing with each other anymore. They're just like, how many can I make and sell? And likewise like Amazon had their Graviton CPUs. They've had what, five, six generations now. And Nvidia has their Grace and Vera CPUs. And no one really deployed Grace standalone CPU boxes. There was like a couple small deployments that Nvidia did for PR with people, but in reality, there was like very few standalone CPU deployments that Nvidia was able to do.
But now, well, just they're just not as good, right?
