Senior AI Analyst Exposes the Next $10 Trillion Monopoly in AI

No one yet has really brought a chip to market that was designed specifically for LLMs.

It takes hundreds of millions of dollars now for a chip startup when maybe back in the day you used to do several rounds of just like a couple million dollars to prove out your little proof of concept. And then if they get access to 100 megawatts, they're going to have to ask themselves, how can I get as much revenue as possible out of this 100 megawatt?

Thank you for having me.

Yes. Yes. I like the idea of starting with systems when we're talking about AI because that's really what it's all about these days. It's rack scale all the way out to of course like full data center clusters. So you know we're talking about the compute at the heart of these systems is ultimately GPUs and that's kind of what gets the airtime or AI accelerators but it's actually a full system. So if so stepping back the question is you know how did we get here and why are people buying a full system from one vendor and not commoditizing it. So I think if you start and you look obviously the workload that we all care about today is LLM inference. And so if you're running at the frontier and you know you're wanting to use the latest and greatest you know OpenAI model or Anthropic model these are models that are like two trillion or more parameters. And what that means is even as you quantize these and try to run them without using as much memory as possible, you might still need something like a terabyte or two terabytes or several terabytes of memory just for the model. Because with these models, as we saw with the scaling laws over, you know, late 2022, 2023, 2024, is the bigger the models and the more compute you have, the better answers you get. And so, you know, way back in the day, we might have taken an AI model and put it just on one GPU.

But if you start to think about like today's frontier models, you need maybe terabytes of memory and yet a single GPU only can have, for example, like 288 gigabytes of HBM on it to store these model weights. And so, quickly you say, well, wait a minute, one model can't fit on a single GPU's memory. So what do you have to do? Oh, you need several GPUs. So you might say, okay, take that model and split it across, you know, four GPUs or eight GPUs. And that's how we started to work our way into needing like, oh, this isn't just a single chip anymore. This is a whole system. And that's how we got our way to, for example, like the Grace Blackwell NVL72 rack. You know, it's a rack with 72 GPUs in it.

We've gotten to a place that we are talking about AI systems even just for inference even if you were only to have one rack you've got GPUs, CPUs, networking, power and cooling. And so back to the question of, you know, why isn't the industry commoditizing it, you know, I think a big piece of where we are at today is Nvidia did a really good job of getting to this rack scale first where they said hey we can help design the whole system and make sure that it works across 72 GPUs and 36 CPUs and that they can all talk to each other. And oh by the way, it also takes software and compilers so that you know your model developer can write your model and actually run it across all this. And so, you know, even though the industry always wants commoditization and competition in multiple suppliers, you know, Nvidia did a good job of frankly getting there first and making this complicated system turnkey so that the software developer it's as easy as, you know, writing their model, deploying it, and it quote unquote, you know, just works.

Yeah, I think so. We are not yet in the era where people are trying to squeeze down their costs as much. We've still been in the era of I need to, it's speed to market. We saw that with Elon Musk and xAI where he's been able to stand up data centers very quickly using Nvidia systems. And I don't think Elon's goal was to say how can we do this as cheaply as possible, but more of just how can we stand up this compute and get productive tokens out of it as quickly as possible. So even though a company can go out and potentially pull the components from many different vendors, there's just a desire to buy it and stand up quickly. And then you know to your other point of there is definitely simplicity there and we see you know even like AMD with their Helios rack that they're bringing to market same kind of thing. They see that pull from customers that say just I want to be able to buy the full rack. I want to be able to get it and stand it up as quickly as possible and get software running as quickly as possible.

Yeah, that is a tough and very interesting question because you know to your point now if you're a startup you can't necessarily just come in and say oh yeah I'm going to build a better GPU, better AI ASIC. People are going to say okay great now what do I do with that right? Are you going to make your end customers have to piece together the whole system? We just said that they want to stand it up as quickly as possible and deploy it as quickly as possible or are you as the startup now going to have to take on building the rest of the system? I think there's probably different approaches here. And by the way, this is why we see that it takes, you know, hundreds of millions of dollars now for a chip startup when maybe back in the day you used to do several rounds of just like a couple million dollars to sort of prove out your little proof of concept. And so yes, can a startup compete here? Yes, but they have to be very smart. I think it could also be death by a thousand paper cuts if you're trying to invent a new AI accelerator and you're trying to invent, you know, your own proprietary networking and maybe your own proprietary cooling. You know, and actually for example, Cerebras, you know, they took a very interesting approach where they're an AI accelerator startup, they've gone public, they make these wafer scale engines is what they called them. And which by the way for those listeners who aren't as familiar, instead of just making a chip, instead of taking a whole wafer of chips and dicing them up and making these individual chips like two GPUs and a CPU that you package, they said hey why don't we just leave it at the wafer scale and have all these chips that are on there and actually make the whole wafer our compute engine. But they had to invent now how do you make them all communicate? How do you cool all of that? How do you deliver power to all of that? And they had to basically invent all of that and that's going to take a lot of time. So I think any other AI ASIC who followed that could have an example like Cerebras to look at and say wow they did very innovative technical things here but they had to invent everything and that's you know timely or time-consuming and expensive. How should we maybe do this differently?

Sure. So at the end of the day when you just think of like a simple talking to an AI sort of chatbot, you know, you write a paragraph of explanation, you know, hey I want you to go research this and give me an answer, and the first thing that the AI system under the hood, the model, needs to do is read through the whole prompt that you gave it. And how this works is this is the prefill stage. And what's happening is in parallel, sort of one of the key innovations with the transformer model is this idea of attention where you work through everything in the long, you know, let's just use English for example, the long English paragraph and you ask, you know, which of these words are sort of connected to other words in the sentence and that way you can sort of piece together the context of the whole paragraph and what each word is referring to. And so you can do all of this in parallel. You can go through and if there's 100 words, you can look at all 100 words in parallel and compare them to all the other 100 words and do some of these calculations to figure out like are they related or not. That's prefill. You're ultimately doing a bunch of linear algebra, a bunch of matrix multiplication. Now when it's time to give you an answer, you know, especially if you remember back to like when ChatGPT first launched, it was a lot slower. You know, you essentially would start to see this stuff kind of come out word by word. And that's because you know every word that I am about to say depends on the words that I just said. And then when I say that word and I say the next word, it also depends on those other words. So basically this is decode. And this is where you're predicting the next tokens or the next words if we're talking about English. And that is sequential. You can't do that in parallel. And people who are running this inference at scale started to realize, hey, wait a minute. Prefill. I've got all these GPUs that are just doing all of this parallel computation and it actually doesn't require a huge amount of memory. And so I've paid for this high bandwidth memory that's very expensive by the way and getting more expensive by the day. And in prefill that HBM is actually just sitting there being underutilized. Hm. Interesting. Okay. Now I've got these other GPUs that are running decode and they're actually not fully utilizing all of their flops, all of their units of compute. Because actually decode is what's called memory bound which is it's a lot of hey I made a prediction and I need to get all the weights and all the whatever I need from the KV cache and then I need to do a little calculation and then I need to go back and forth and back and forth you know one at a time and you're not doing things in parallel and you're really just waiting on memory. And so you know at the highest level even Nvidia led the way with this saying wait a minute, we've got a bunch of GPUs that are being heavily utilized for compute and their memory is being underutilized. And then we have a bunch of during the prefill phase. Then we've got a bunch of GPUs in the decode phase that are basically underutilizing their compute and just totally utilizing all their memory. And so naturally, you know, any engineer is going to look at that and say like, huh, well, maybe we should disaggregate these and maybe the prefill should run on systems that have lots of compute but don't necessarily need all that memory. And maybe on the decode side, we should really emphasize memory bandwidth and how quickly information can get shuttled around. And maybe it doesn't even need quite as much compute or we should essentially overindex on the memory part. And so naturally right there we started to move into a world where Nvidia shipped for their systems a software layer called Dynamo which helps orchestrate this and would let you take Nvidia GPUs and it would orchestrate across. And that actually gave rise to the Groq and the Cerebras, these AI ASIC startups who actually were started even before transformer-based LLM was sort of the defining workload of our era, but they had made architectural choices where they used a lot of this really fast on-chip memory called SRAM. And it's actually where you use transistors to store the memory instead of DRAM, which is capacitors and transistors, which is what HBM is made of. We won't go way down into those memory details, but basically these early startups had made a bet on having really high memory bandwidth. And once the workload got separated into prefill and decode, they could actually sort of raise their hand and go, oh, wait. We're actually really good at decode. And in fact, we can go even faster than GPUs. And this kind of took us from the inference era where everything was on GPUs to saying, oh, wait a minute. What if you could slot in one of these ASICs that are heavily built on SRAM that can maybe unlock a thousand tokens a second whereas maybe a GPU when it was running decode just the way that GPUs are more general purpose designed and their memory hierarchy decisions maybe they could only run at like...

Yeah. Yeah. Yeah. There's like so many different nuances here. So if we take a look at the hyperscalers when they're deploying, you know, tens of billions of dollars of GPUs and as they're starting, which of course early on GPUs very general purpose, very flexible, you know, gives you the freedom to change your workloads. But let's say you're in OpenAI and you're like no these are our specific models that we know that we are iterating on like the biggest the frontier one and the medium size one and the small one. You start to say, hey, we should really sort of cater to this workload's needs and therefore it would make sense to deal with the complexities that you pointed out of having to split up prefill, decode, orchestrate that. It might be worth it and you know maybe that unlocks for example being able to sell an ultra premium tier where you get really really fast inference and maybe there's going to be a small subset of users who would pay you know 10x more to get tokens that are 5x faster. So you know I think there are certain model labs and hyperscalers that are at scale that are saying the complexity is totally worth it. We're willing to deal with different SKUs, if you will, like different chips. Now on the other hand, I think there's going to be tons of enterprises and arguably the long tail of consumers who of course they want inference and they're thinking about cost and they're thinking about speed, but they aren't going to want to manage all of that complexity necessarily. So for them, you know, I don't think where we're at today, they're not going to necessarily need that complexity. Now, of course, everyone always wants faster inference. So if there's a way for them to get that outcome without having to deal with all the complexity, of course, they're going to want it. I for one, when I'm trying to think about this space and ask like, oh, is it just one size GPU fits all or is it going to be many SKUs, you know, many different chips? Where's this going to go in the end? I do look to CPUs. Like when you look at any CPU vendor or any cloud, you know, like Google Cloud, they don't just deploy one CPU even for, you know, for customers who want to rent them from their cloud. They still have sort of a portfolio that say like, hey, this one has a lot of memory in case you're running a database. Oh, this one's actually pretty vanilla and it's nice and cheap if you're just running like an API server. And you'll see there's still like sort of shapes of we got this family of chip, this family of chip. I do think there's a world where we could get to having different, you know, a couple different shapes of AI accelerators. Now where we are, what we led into is we said, okay, we had this training era and then we had this inference era. Those are all on GPUs. And then we've got this next era where it's like, hey, let's have a GPU plus this specific decode chip that's, you know, made a good memory hierarchy trade-off so we could have high interactivity is what they call it, which is like really fast tokens. So where we are today though is those are multi-vendor. So it was like, you know, Nvidia plus Groq, of course, Nvidia acquired Groq, so they're trying to bring that all in their house. You know, someone else plus Cerebras, AMD plus Cerebras or Trainium plus Cerebras or whatever. And now my thought is that we will ultimately go to those SKUs living inside the same silicon vendor. Because it starts to get complicated when your route to market is to depend on another company, right? Like oh I sell a GPU and they sell a decode specific thing. It does feel like that is working right now. There's definitely demand for it.

Right. Right. Totally. I do think you know zooming way out when you look at the semiconductor industry in the long run there always seems to be like three or four vendors in a certain thing whether you look at like wafer fab equipment, foundries, it's complicated but we're getting back to maybe having two or three CPU vendors, GPU vendors. We are in an era of course as happens whenever there's like drastic innovation where a ton of competitors have popped up so I think it would not be crazy to zoom out and say like okay in the grand arc maybe only three or four people will shake out and therefore there'll be some sort of consolidation. I personally think this could look, this isn't as simple as just like oh Nvidia buys them, AMD buys them even though we're seeing some of that. Of course there's regulatory things to talk about there, but I actually think if you look up and down the stack, there's actually a lot of competition. For example, at the Neocloud layer, a Neocloud might be incentivized to say like, hey, I can tell that the customer just they want the right silicon for the right job, but they also don't want to deal with the complexity. I could buy a bunch of different silicon and I could deal with all the complexity and I could just try to give them, I could sell them tokens as a service and try to give them the speed or the cost or whatever that they're looking for and I could take that onus on and differentiate from other Neoclouds. So, I do think there's still routes to market where startups can come in today and they could just say like, hey, I am the best decode solution. You should try me. Or maybe I'm a prefill solution and yet be able to figure out a way to get to customers and to let the end developer let's say to not have to deal with all that complexity. Therefore you might just keep pulling on that and play it forward and say like oh wow could an AI ASIC startup ever merge with a Neocloud. I don't know maybe you know like maybe it won't just be Nvidia or AMD buying all these.

Yeah. Yeah.

Yeah. Yeah. So for listeners, by Neocloud we mean, you know, a cloud company that started by just renting GPUs. And I'll just talk through my hesitation when I first saw the idea of Neoclouds and maybe I am a fair proxy for some investors. Maybe they thought this way which was, you know, ultimately I saw, okay, well let's back up. So who's renting from these Neoclouds? Well it turns out a lot of it is hyperscalers which is ultimately driven by demand from the biggest model labs. And so the question is like, well wait a minute, you're saying that, you know, OpenAI is running on GPUs that Microsoft is renting from some Neocloud. Well doesn't Microsoft just have their own data centers? And the answer is well of course they do and they're trying to build more but at the end of the day we are limited by power, access to power. We are limited. There's like financial reasons to go to rent versus buy. Maybe even go off balance sheet as the capex has, you know, increased year over year over year. And so there's very legitimate reasons why even someone as sophisticated as Microsoft Azure might say like actually I want to rent GPUs from someone and I'm also building other data centers, but it's like maybe it's a stopgap. So looking at that I said, okay, so you're going to have a company that's a Neocloud that maybe they have access to power and maybe they have a good way to raise capital against assets like this, for example, like Bitcoin miners historically have had this experience, and maybe they're shifting into GPUs. And so I looked at it and I thought, huh, okay. So they were doing Bitcoin miners. Now GPUs are hot. So they're going to do GPUs, but they're just going to rent them to Azure. But Azure is also just standing up their own data center. So like how is this sustainable? They've got one huge customer concentration, literally maybe one. But to their credit that's how you get the financing is you say like yeah I've got Microsoft who's going to rent these from me and so people will lend against that and sort of believe in that. And so that's part of why I missed it as these companies were popping up. There's a legitimate need in the marketplace for people who have access to power, who can get financing, who can manage this, and who can move quickly, to stand up GPUs, to run models on them, to rent it out as bare metal or maybe at a higher level of abstraction. And even though they have serious customer concentration risk, so does everyone else in the semiconductor industry right now, right? Like who's Nvidia's end customers? It's like even Nvidia, the biggest, baddest, you know, they're selling a lot of GPUs to a certain set of customers.

Totally. That is the name of the game. If you look at any component supplier like even in the interconnect space or the switching space, like you look at like Credo who made these active electrical cables, really awesome invention, and same thing they're selling it to like even when they went public they're selling it to a handful of customers. And so the name of the game too for this era is like watching how this unfolds and seeing like how do some of those people who win one big customer ultimately win a couple more or how do they build out more of a portfolio and sort of reduce a little bit of that risk. But it just is the way that the industry works right now.

Maybe a little bit of everything. I mean, at the end of the day, this is risky business because the investments might be $40 billion this year and $80 billion next year and a hundred billion the year after that. So, obviously for these clouds, I think a lot of the existing players weren't necessarily, it's not that they didn't believe that the future that we're in now would manifest. I think it's all about timing. And I do think that they're probably not incentivized necessarily to, you know, just sprint out and stand up all these data centers. Obviously, it's a huge investment. I think that the Neoclouds were able to move faster, take on more risk. Like they can kind of go for broke here. It's like, all right, cool. Let's convert. Let's buy a bunch of GPUs and let's get moving. And so yeah, I just don't think that that kind of like the innovator's dilemma. It's like they're already the big three clouds. They already have all the customers. I think they're probably looking at each other, the other big CSPs probably like AWS and Google and Azure all looking at each other and just saying like, well, are we standing up? Are we all believing that this future is coming and investing in GPUs sort of at the same rate as each other? But that could still not actually be enough supply to meet demand. Now I do think that demand actually went higher faster than everyone had expected especially once we got reasoning models and then now of course in the agentic age. And so therefore I do think that even if they all looked around and said like yeah we think if we grow supply like this is a little risky and it feels like these numbers are really big but we think the demand will be there, but I do think demand has skyrocketed and gives an opportunity for neoclouds to come in and say like yeah we'll fill that demand. I think there was also maybe like a difference in interest level. You heard some of the hyperscalers make comments about how these bare metal GPU instances are low margin and sort of commodity and therefore they don't want to support them as much whereas the neoclouds read them as revenue and therefore good.

Yeah. Yeah. Yeah. Yeah. You make a fair point. I mean anyone whose business was in renting CPUs or selling services on top of CPUs, the cost structure is much better than GPUs. And so, yeah, you could see, you know, CFOs saying like, wait a minute, we're going to spend a ton of money and our margins are going to go down. Even if our margin dollars go up, there's still conversations to be had that might make you slow down or hesitate a little bit.

Yeah. Yeah. Yeah. Yeah. Yeah. It's very interesting. I mean, I'm not sure there will always be need for a hundred Neoclouds, but I do think that it's real demand and it will always be. I don't think that the big three clouds will always meet the needs of people, you know, indefinitely. Not only that, so like okay, we've got particular workloads, you know, now where it's like, yeah, I want really high interactivity or medium interactivity. But there's always going to be innovation. Like the World Labs company with Fei-Fei Li, you know, they're coming out with world models. Who's going to make the bet there? Like what if those don't run to the exact shape that all these big clouds have invested in? And I do think that there will always be, you know, new workloads or new demands or whatever that are popping into existence where a Neocloud is going to pop up quickly and say like I can meet your need and I can innovate there. And I think that will always exist. Again, it may not be enough sort of cutting edge frontier demand for a hundred neoclouds to hop on it and try to offer something different, but I definitely think there'll always be a need for these more nimble, smaller essentially like GPU-as-a-rental companies that can innovate a lot closer to where the frontier of like model labs and AI enabled software companies are innovating.

Yeah, you know, I definitely was the type of person where right away I was like, okay, this is different. This feels funny. I need to dig in and understand it and try to understand both sides. I could definitely see why it looks like circular financing. And you could even call it as much, like call it what you will. I think going back to thinking through demand and supply and the cost of capital here and just knowing that, you know, it is a fact that there's just insatiable demand especially again now with agentic where literally the cost, the barrier to entry for software development has gone as close to zero as possible. I mean, I've got a son who made like a 70,000 line video game this summer and he didn't write those lines by himself, right? He used Codex to do it and like that's amazing and it's unreal and I think dude just wait till he's in high school and in college and beyond. Like he's going to use AI so much more intelligently than me. He's going to use way more tokens than me. Like I can't even believe what it's going to be like in the future. And so, you know, you can just look in every industry and see people doing like lots of people writing software doing interesting things that they couldn't do before. So like the demand totally real, the supply very fixed. Ultimately at the end of the day you might look at like someone like a TSMC and there's just only so many wafers that come out, only so much CoWoS capacity. But even as GPUs get built, so you know even if we can increase the amount of GPUs that get built, the question is you know who has the capital to buy them because today we might be talking you know $5, $10 million a rack or more. Like who has that kind of money laying around? And so there is this cost of capital, this financing thing that comes into play where it's like okay, you know if people want, if customers are just like I just want inference, I want it as fast as possible, I want it as soon as possible, please make it happen, like who in the supply chain has the money to invest in standing up all these data centers and running them and getting the inference? And then you know oh it could be neoclouds. Okay well can they, do they have access to the capital that they need to, you know, make whatever tens of billions of dollars investment or even a few billion dollars...

investment. And you know, the answer is a lot of these might be early companies or they were Bitcoin miners or whatever. So, they might have some access to capital, but if you're Nvidia and you're staying on the sidelines and you have all this money and they're the world's biggest company, you're saying demand is incredible and supply is what it is, but we've got some supply in the market, but it's now it's not just building it. It's like getting it powered up, financed, stood up. Of course, it makes sense from their perspective to say if we can help get this stood up more. If we can, the neoclouds like banks aren't so sure they want to lend to them, but if Microsoft or Google or AWS says, I'll be the offtaker. So, you know, remember I'm the, you know, it's not don't think about the Neo cloud, think about me when you're lending against it. And if Nvidia could also come in and say like, "Hey, we want to help make this happen. Like, can we put our brand behind it? Backstop it, whatever. Can we just make this happen?" I can see why Nvidia would want to do that. Now, does that mean also that they benefit from it? Of course, totally. You know, it's customers and maybe even if there's some sort of, you know, revenue sharing, it's a new source of revenue for Nvidia. But,