AI Semiconductor Landscape feat. Dylan Patel | BG2 w/ Bill Gurley & Brad Gerstner

Is scaling dead? Then why is Mark Zuckerberg building a 2-gigawatt data center in Louisiana? Why is Amazon building these multi-gigawatt data centers? Why is Google, why is Microsoft building multiple gigawatt data centers plus buying billions and billions of dollars of fiber to connect them together? Because they think, hey, I need to win on scale, so let me just connect all the data centers together with super high bandwidth so then I can make them act like one data center towards one job. This whole 'is scaling over' narrative falls on its face when you see what the people who know the best are spending on.

So when I was eight, my Xbox broke, and I have immigrant parents. I grew up in rural Georgia, so I didn't have much to do besides be a nerd. I couldn't tell them I broke my Xbox. I had to open it up, short the temperature sensor, and fix it. That was the way to fix it. I didn't know what I was doing at the time, but then I stayed on those forums and I became a forum warrior. You know, you see those people in the comments always yelling at you, Brad. That was me as a child. It was just arguing with people online as a child and then being passionate. As soon as I started making money, I was reading earnings from semiconductor companies and investing in them with my internship money. Yeah, reading technical stuff as well, of course, and then working a little bit.

Yeah, so today we are a semiconductor research firm, AI research firm. We service companies. Our biggest customers are all hyperscalers, the largest semiconductor companies, private equity as well as hedge funds. And we sell data around where every data center in the world is, what the power is in each quarter, how the build-outs are going. We sell around fabs. We track all 1,500 fabs in the world. For your purposes, only 50 of them matter, but we track all 1,500 fabs around the world. Same thing with the supply chain of whether it be cables or servers or boards or transformer substation equipment. We try and track all of this on a very number-driven basis as well as forecasting. And then we do consulting around those areas.

So I would say if you ignored Google, it would be over 98%. But then when you bring Google into the mix, it's actually more like 70%, because Google is really that large of a percentage of AI workloads, especially production workloads. By production, you mean in-house workloads for Google? Production as in things that are making money. Things that are making money, they're actually probably, it's probably even less than 70%, right? Because you think about Google Search and Google Ads are two of the largest AI-driven businesses in the world. The only things that are even comparable are like TikTok and Meta's. And those Google workloads, I think it's important just to kind of frame this, those are running on Google's proprietary chips. They're non-LLM workloads. Correct. So Google's production workloads for non-LLM and LLM run on their internal silicon. And I think one of the interesting things is yes, everyone will say Google dropped the ball on Transformers and LLMs. How did OpenAI do GPT and not Google? But Google was running Transformers even in their search workload since 2018, 2019. The advent of BERT, which was one of the most well-known, most popular Transformers before we got to the GPT madness, has been in their production search workloads for years. So they run Transformers on their own in their search and ads business as well.

So I like to think of it as a three-headed dragon. I would say every semiconductor company in the world sucks at software except for Nvidia. So there's software, there's of course hardware. People don't realize that Nvidia is actually just much better at hardware than most people. They get to the newest technologies first and fastest because they drive like crazy towards hitting certain production goals and targets. They get chips out faster than other people from thought design to deployed. And then the networking side of things, they bought Mellanox and they've driven really hard with the networking side of things. So those three things kind of combine to make a three-headed dragon that no other semiconductor company can do alone.

Right. So when you look broadly at the GPU, no one buys one chip for running an AI workload. Models have far exceeded that. You look at today's leading-edge models like GPT-4, which was over a trillion parameters. A trillion parameters is over a terabyte of memory. You can't get a chip with that capacity. A chip can't have enough performance to serve that model even if it had enough memory capacity. So therefore, you must tie together many chips together. And so what's interesting is that Nvidia has seen that and built an architecture that has many chips networked together really well, called NVLink. But funnily enough, and the thing that many ignore, is that Google actually did this alongside Broadcom. And they did it before Nvidia. Today everyone's freaking out about, or not freaking out, but everyone's very excited about Nvidia's Blackwell system. It is a rack of GPUs that is the purchased unit. It's not one server, it's not one chip, it's a rack. This rack weighs three tons and it has thousands and thousands of cables and all these things that Jensen will probably tell you, extremely complex. Interestingly, Google did something very similar in 2018 with the TPU. Now they couldn't do it alone. They know the software, they know what the compute element needs to be, but they can't do a lot of the other difficult things like package design, like networking. And so they had to work with other vendors like Broadcom to do this. And because Google had such a unified vision of where AI models were headed, they actually were able to build this system architecture that was optimized for AI. Whereas at the time, Nvidia was like, well, how big do we go? I'm sure they could have tried to scale up bigger, but what they saw is the primary workloads didn't require scaling to that degree. Now everyone sort of sees this and they're running towards it. But Nvidia's got Blackwell coming now. Competitors like AMD and others have to make an acquisition recently to help them get into the system design. Because building a chip is one thing, but building many chips that connect together, cooling them appropriately, networking them together, making sure that it's reliable at that scale is a whole host of problems that semiconductor companies don't have the engineers for.

I would say for differentiating, Nvidia has primarily focused on supply chain things, which might sound like, oh well, they're just ordering stuff. No, no, no. You have to work deeply with the supply chain to build the next generation technology so that you can bring it to market before anyone else does. Because if Nvidia stands still, they will be eaten up. They're sort of the Andy Grove 'only the paranoid will survive.' Jensen is probably the most paranoid man in the world. He's known for many years since before the LLM craze, all of his biggest customers were building AI chips. Before the LLM craze, his main competitors were like, oh, we should make GPUs. And yet he stays on top because he's bringing to market technologies at volume that no one else can. And so whether it be in networking, whether it be in optics, whether it be in water cooling, whether it be in all sorts of other power delivery, all these things, he's bringing to market technologies that no one else has. And he has to work with the supply chain and teach those supply chain companies, and they're helping obviously, they have their own capabilities to build things that don't exist today. And Nvidia is trying to do this on an annual cadence now. That's incredible. Blackwell, Blackwell Ultra, Rubin, Rubin Ultra. They're going so fast, they're driving so many changes every year. Of course people are going to be like, oh no, there's some delays in Blackwell. Yeah, of course, look how hard you're driving the supply chain.

Because it's only two to three years ahead. Well, the funny thing is, a lot of people at Nvidia will say Jensen doesn't plan more than a year or a year and a half out, because they change things and they'll deploy them out that fast. Every other semiconductor company takes years to deploy, make architecture changes.

Yeah, so the main thing for Nvidia is, hey, this workload is this big. It's well over $100 billion of spend for the biggest customers. They have multiple customers that are spending billions of dollars. I can hire enough engineers to figure out how to run my model on other hardware. Now maybe I can't figure out how to train on other hardware, but I can figure out how to run it for inference on other hardware. So Nvidia's moat in inference is actually a lot smaller on software, but it's a lot bigger on TCO. They just have the best hardware. Now what does the best hardware mean? It means capital cost, it means operation cost, and it means performance. Performance TCO. Yes. And Nvidia's whole moat here is if they stand still, their performance TCO doesn't grow. But interestingly, they are. Like with Blackwell, not only is it way, way, way faster, anywhere from 10 to 15 times on really large models for inference because they've optimized it for very large language models, they've also decided, hey, we're going to cut our margin too somewhat because I'm competing with Amazon's chip and TPU and AMD and all these things. They've decided to cut their margin too. So between all these things, they've decided that they need to push performance TCO not 2x every two years, you know, Moore's Law. They've decided they need to push performance TCO 5x maybe every year. At least that's what Blackwell is, and we'll see what Rubin does. But 5x plus in a single year for performance TCO is an insane pace. And then you stack on top, hey, AI models are actually getting a lot better for the same size. The cost for delivering LLMs is tanking, which is going to induce demand.

So I think a lot of people in the investor community call CUDA, which is just one layer for all of Nvidia software, there's a lot of layers of software. But for simplicity's sake, regarding networking or what runs on switches or what runs on all sorts of things, fleet management stuff, all sorts of stuff that Nvidia makes, we'll just call it CUDA for simplicity's sake. But all of this software is stupendously difficult to replicate. In fact, no one else has deployments to do that besides the hyperscalers. And 800,000 GPUs is like a Microsoft inference cluster. It's not a training cluster. So when you talk about, hey, what is the difficulty here? On training, this is users constantly experimenting. Researchers saying, hey, let's try this, let's try that. I don't have time to optimize and hand-wring out the performance. I rely on Nvidia's performance to be quite good with existing software stacks or very little effort. But when I go to inference, Microsoft is deploying five, six models across how many billions of revenue? So all of OpenAI's revenue plus what, $10 billion of inference revenue? Yeah, so they have $10 billion of revenue here and they're deploying five models, right? GPT-4, 4o, 4o mini, and now the reasoning models, o1. So they're deploying very few models and those change what, every six months? So every six months they get a new model and they deploy that. So within that timeframe, you can hand-wring out the performance. And so Microsoft has deployed GPT-style models on other competitors' hardware, such as AMD, and some of their own, but mostly AMD. And so they can wring that out with software because they can spend hundreds of engineer hours or thousands of engineer hours on working this out because it's such a unified sort of workload.

Nvidia has been pushing non-AI workloads for accelerators for a very long time. Professional visualization, Pixar uses a ton of GPUs to make every movie. All these semiconductor engineering applications, all these things do use GPUs. I would say they are a drop in the bucket compared to AI. The other aspect I would say is, and this is sort of a bit contentious with your chart, I think, but IBM mainframes sell more volume and revenue every single cycle. So yes, no one in the Bay uses mainframes or talks about mainframes, but they're still growing. And so I would say the same applies to CPUs, to classic workloads. Just because AI is here doesn't mean web serving is going to slow down or databasing is going to slow down. Now what does happen is that line is like this and the AI line is like this. And furthermore, when you talk about, hey, these applications, they're now AI, you know, Excel with Copilot or Word with Copilot, whatever, they're still going to have all of those classic operations. You don't get rid of what you used to have. Southwest doesn't stop booking flights, they just run AI analytics on top of their flights to maybe do pricing better or whatever. So I would say that still happens. But there is an element of replacement that is sort of misunderstood, which is given how much people are deploying, how tight the supply chains for data centers are, data centers take longer, they're longer time supply chains unfortunately. Which is why you see things like what Elon's doing. But when you think about that, well, how can I get power then? So you can do what CoreWeave is doing and go to crypto mining companies and just clear them out and put a bunch of GPUs in them, retrofit the data center, put GPUs on them like they're doing in Texas. Or you can do what some of these other folks are doing which is, hey, well, my depreciation for CPU servers has gone from three years to six years in just a handful of years. Why? Because Intel's progress has been this. So in reality, the old Intel CPU is not that much better. But all of a sudden over the last couple years, AMD's burst onto the scene, ARM CPUs have burst onto the scene, Intel's started to right the ship. Now I can upgrade. The plurality of Amazon CPUs in their data centers are 24-core Intel CPUs that were manufactured from 2015 to 2020, more or less the same architecture. This 24-core CPU, I can buy 128 or 192-core CPU now today where each CPU core is higher performance. And well, if I just replace like six servers with one, I basically invented power out of thin air. I mean, in effect, because these old servers which are six-plus years old, they can just be deprecated. And so with CapEx of new servers, I can replace these old servers. And now every time I do that, I can throw another AI server in there. So this is sort of the, yes, there is some replacement, I still need more total capacity, but that total capacity can be served by fewer machines maybe if I buy new ones. And generally the market is not going to shrink, it's still going to grow just nowhere close to what AI is. And AI is causing this behavior of I need to replace so I can get power.

So pre-training scaling laws are pretty simple. You get more compute and then I throw it at a model and it'll get better. Now what is that? That breaks out into two axes: data and parameters. The bigger the model, the more data, the better. And there's actually an optimal ratio. Google published a paper called Chinchilla which says the optimal ratio of data to parameter, model size. And that's the scaling thing. Now what happens when the data runs out? Well, I don't really get much more data, but I keep growing the size of the model because my budget for compute keeps growing. This is a bit not fair though. We have barely, barely, barely tapped video data. So there is a significant amount of data that's not tapped. It's just video data is so much more information than written data. And so therefore you're throwing that away. But I think that's part of the misunderstanding there. But more importantly, text is the most efficient domain. Humans generally, yes, a picture paints a thousand words, but if I write 100 words, you can probably figure out FAS. And the transcripts of most of those videos were already in there already. But regardless, data is a big axis. Now the problem is this is only pre-training. Training a model is more than just the pre-training. There's many elements of it. And so people have been talking about, hey, time compute. Yeah, that's important. You can community-scale models if you figure out how to make them think and recursively be like, oh, that's not right, let me think this way. Much like you don't hire an intern and say, hey, what's the answer to X, or you don't hire a PhD and say, hey, what's the answer to X. You're like, go work on this and then they come back and bring something to you. So inference time compute is important. But really what's more important is as we continue to get more and more compute, can we improve models if data is run out? And the answer is you can create data out of thin air almost, in certain domains. And so this is the whole debate around scaling laws. How can we create data? And so what is Ilya's company doing most likely? What is Mira's company doing most likely? Mira Murati, CTO of OpenAI. What are all these companies focused on? OpenAI, they have Noam Brown who's sort of one of the big reasoning people on road shows just going and speaking everywhere basically. What are they doing? They're saying, hey, we can still improve these models. Yes, spending compute at inference time is important, but what do we do at training time? Because you can't just tell a model 'think more' and it gets better. You have to do a lot of training time. And so what that is, is I take the model, I take an objective function I have. What is the square root of 81? Now if I asked many people what's the square root of 81, many could answer, but I bet many more people could answer if they thought about it more. But you say, hey, let's have the existing model do that. Let's have it just run every possible, not possible, many permutations of this. Start off with say five, and then anytime it's unsure, branch into multiple. So you start out, you have hundreds of quote-unquote rollouts or trajectories, generated data. Most of this is garbage. You prune it down to, hey, only these paths got to the right answer. Okay, now I feed that and that is now new training data. And so I do this with every possible area where I can do functional verification. Functional verification, i.e., hey, this code compiles, hey, this unit test that I have in my code base, how can I generate the solution, how can I generate the function. And you do this over and over and over again across many, many, many different domains where you can functionally prove it's real. You generate all this data, you throw away the vast, vast majority of it, but you now have some chains of thought that you can train the model on, which then it will learn how to do that more effectively and it generalizes outside of it. And so this is the whole domain. Now when you talk about scaling laws, its point of diminishing returns is kind of not proven yet, by the way. Because it's more so, hey, the scaling laws are a log-log axis. I.e., it takes 10x more investment to get the next iteration. Well, 10x more investment, going from $30 million to $300 million, $300 million to $3 billion is relevant. But when Sam wants to go from $3 billion to $30 billion, it's a little difficult to raise that money. That's why the most recent rounds are a bit like, oh crap, we can't spend $30 billion on the next run. And so the question is, well, that's just one axis. Where have we gone on synthetic data? Oh, we're still very early days. We've spent tens of millions of dollars maybe on synthetic data.

Yeah, I think one of the interesting things about AI is at first in 2022, 2023, with the release of diffusion models, with the release of text models, people were like, oh wow, artists are the ones that are the most out of luck, not technical jobs. Actually, these things suck at technical jobs. But with this new axis of synthetic data and test time compute, actually where the areas where we can teach the model, we can't teach it what good art is because we have no way to functionally prove what good art is. We can teach it to write really good software. We can teach it how to do mathematical proofs. We can teach it how to engineer systems because there are, while there are trade-offs and this is not just a one-zero thing, especially on engineering systems, this is something you can functionally verify is this works or not, or this is correct, the output. And then the model can iterate more often.

So the scaling goes back to what's the optimal ratio, how do we continue to grow. Just blindly growing parameter count when we don't have any more data, or the data is very hard to get at because it's video data, wouldn't give you so many gains. And then there's also the axis of it's a log chart, you need 10x more to get the next jump. So when you look at both of those, oh crap, I need to invest 10x more and I might not get the full gain because I don't have the data. But the data generation side, we are so early days with this. So the point is I'm still going to squeak out enough gain that it's a positive return, particularly when you look at the competitive dynamic, our models versus our competitor models. So it's a rational decision to go from 100,000 to 200,000 or 300,000 even if the big one-time gain in pre-training is behind us, or rather it's logarithmically more expensive to do that gain. Correct. So it's still there, the gain is still there, but the whole 'Orion has failed' narrative around OpenAI's model, and they didn't release Orion, they released o1 which is sort of a different axis. It's partially because of these data issues, but it's partially because they did not scale 10x. Because scaling 10x from four to this is actually, I think this is Gavin's point.

I think they have never said the Chinchilla scaling laws were what delivers us AGI. They've had scaling, scaling is you need a lot of compute. And guess what, if you have to generate a ton of data and throw away most of it because only some of the paths are good, you're spending a ton of compute at train time. And this is sort of the axis that is, we may actually see models improve faster in the next six months to a year than we saw them improve in the last year because there's this new axis of synthetic data generation and the amount of compute we can throw at it. We're still right here in the scaling law. We haven't pushed it to billions of dollars spent on synthetic data generation, functional verification, reasoning training. We've only spent millions, tens of millions of dollars. So what happens when we scale that up? So there is a new axis of spending money. And then there's of course test time compute as well, i.e., spending time at inference to get better and better. So it's possible, and in fact many people at these labs believe that the next year of gains or the next six months of gains will be faster because they've unlocked this new axis through a new methodology. And it still scales, right? Because this requires stupendous amounts of compute. You're generating so much more data than exists on the web and then you're throwing away most of it. But you're generating so much that you have to run the model constantly.

Yeah, so I think that goes back to the point around what can we functionally verify is true or not. What can I grade and it's not subjective. What class can you take in college and you get the thing back and you're like, oh this is BS, or dang I messed that up. There's certain classes where you can have determinism of grading the output. Exactly. So if it can be functionally verified, amazing. If it has to be judged, there's sort of two ways to judge an output. There is without using humans, this is sort of the whole Scale AI. What were they initially doing? They were using a ton of humans to create good data, labeled data. But now humans don't scale for this level of data. Humans post on the internet every day and we've already tapped that out more or less. So what are domains that work? These are domains where, hey, in Google, when they push data to any of their services, they have tons of unit tests. These unit tests make sure everything's working well. Why can't I have the LLM just generate a ton of outputs and then use those unit tests to grade those outputs? Because it's pass or fail. And then you can also grade these outputs in other ways, like oh it takes this long to run versus this long to run. So then you have various. There's other areas such as, hey, image generation. Well, actually it's harder to say which image looks more beautiful to you versus me. I might like sunsets and flowers and you might like the beach. You can't really argue what is good there. So there's no functional verification there, it's only subjective. So the objective nature of this is where. So where do we have objective grading? We have that in code, we have that in math, we have that in engineering. And while these can be complex, engineering is not just 'this is the best solution,' it's 'hey, given all the resources we've had and given all these trade-offs, we think this is the best trade-off.' That's usually what engineering ends up being. Well, I can still look at all these axes. Whereas in subjective things, like hey, what's the best way to write this email or what's the best way to negotiate with this person, that's difficult. That's not something that is objective.

Yeah, so I think when you look at the Street's estimates for CapEx, they're all far too low based on a few factors. So we track every data center in the world and it's insane how much, especially Microsoft and now Meta and Amazon and many others, are spending on data center capacity. And as that power comes online, which you can track pretty easily if you look at all of the different regulatory filings and satellite imagery, all these things that we do, you can see that hey, they're going to have this much data center capacity. What are you going to fill in there? It turns out you have to fill it up. You can make some estimates around how much power is each GPU, all-in, everything. Satya said he's going to slow down a little bit, but they've signed deals for next year rentals in some of these cases. Part of the reason he said is he expects his cloud revenue in the first half of next year to accelerate because he said we're going to have a lot more data center capacity and we're currently capacity constrained. So again, going back to the 'is scaling dead' question, then why is Mark Zuckerberg building a 2-gigawatt data center in Louisiana? Why is Amazon building these multi-gigawatt data centers? Why is Google, why is Microsoft building multiple gigawatt data centers plus buying billions and billions of dollars of fiber to connect them together? Because they think, hey, I need to win on scale, so let me just connect all the data centers together with super high bandwidth so then I can make them act like one data center towards one job. So this whole 'is scaling over' narrative falls on its face when you see what the people who know the best are spending on.

So when you think about training a neural network, it is doing a forwards pass and a backwards pass. Forwards pass is generating the data basically, and it's half as much compute as the backwards pass which is updating the weights. When you look at this new paradigm of synthetic data generation, grading the outputs, and then training the model, you're going to do many, many, many forward passes before you do a backwards pass. What is serving a user? That's also just a forwards pass. So it turns out that there is a lot of inference in training. In fact, there's more inference in training than there is updating the model weights because you have to generate hundreds of possibilities and then you only train on a couple of them. So that paradigm is very relevant. The other paradigm I would say that is very relevant is when you're training a model, do you necessarily need to be co-located for every single aspect of it? And the answer is, depends on what you're doing. If you're in the pre-training paradigm, then maybe you don't. You need it to be co-located. You need everything to be in one spot. Why did Microsoft in Q1 and Q2 sign these massive fiber deals? And why are they building multiple similar-size data centers in Wisconsin and Atlanta and Texas and Arizona? Why are they doing that? Because they already see the research there for being able to split the workload more appropriately. Which is, hey, this data center, it's not serving users, it's running inference. It's just running inference and then throwing away most of the output because some of the output is good. And because I'm grading it. And they're doing this while they're also updating the model in other areas. So the whole paradigm of pre-training is not something down. It's just logarithmically more expensive for each generation, for each incremental improvement. People are finding other ways to, but there's other ways to not just continue this, but hey, I don't need a logarithmic increase in spend to get the next generation of improvement. In fact, through this reasoning training and inference, I can get that logarithmic improvement in the model without ever spending that. Now I'm going to do both, because each model jump has unlocked huge value.

Yeah, so I think there's a couple things that are not fair. Cisco's revenue, a lot of it was funded through private credit investments into building out telecom infrastructure. When we look at Nvidia's revenue sources, very little of it is private credit. And in some cases, yes, it's private credit like CoreWeave, but CoreWeave is just backstopped by Microsoft. There is significant amounts of difference in what is the source of the capital. The other thing is at the peak of, especially when you inflation-adjust it, the private capital entering the space was much larger than it is today. As much as people say the venture markets are going crazy, throwing these huge valuations at all these companies, and we were just talking about this before the show, but the venture markets, the private markets have not even tapped in. Guess what, private market money like in the Middle East and these sovereign wealth funds, it's not coming in yet, has barely come in. Why wouldn't there be a lot more spend from them as well? And so there is a significant amount of the difference of capital. The source is positive cash flows of the most profitable companies that have ever lived or ever existed in humanity versus credit speculatively spent. So I think that is a big aspect that also gives it a bit of a knob. These companies that are profitable will be a bit more rational. I think Corporate America is investing more in AI and with more conviction than they did even in the internet wave.

Yeah, so pre-training may be slowing down or it's too expensive, but there's these other aspects of synthetic data generation and inference time compute. Inference time compute is, on the surface, sounds amazing. I don't need to spend more training a model. But when you think...

Think about it for a second. This is actually very, very... this is not the way you want to scale. You only do that because you have to. Think about it, GPT-4 was trained with hundreds of millions of dollars and it's generating billions of dollars of revenue. Hundreds of millions of dollars to train GPT-4 and it's generating billions of dollars of revenue. So when you say like, hey, Microsoft's capex is nuts, sure, but their spend on GPT-4 was very reasonable relative to the ROI they're getting out of it right now.

When you say, hey, I want the next gain, if I just spend a large amount of capital and train a better model, awesome. But if I don't have to spend that large amount of capital and I deploy this better model at the time of revenue generation rather than ahead of time when I'm training the model, this also sounds awesome. But this comes with this big trade-off. When you're running reasoning, you're having the model generate a lot and then the answer is only a portion of that.

Today when you open up ChatGPT, use GPT-4o, you say something, you get a response. You send something, you get a response. All of the stuff that's being generated is being sent to you. Now you're having this reasoning phase. OpenAI doesn't want to show you, but there's some open source Chinese models like Alibaba and DeepSeek, they've released some open source models which are not quite as good as OpenAI of course, but they show you what that reasoning looks like if you want to. It generates tons of things, it sometimes switches between Chinese and English, whatever it is, it's thinking. It's like, should I do it this way? Should I break it down in these steps? And then it comes out with an answer.

Now on the surface, awesome, I didn't have to spend any more on R&D or capital. They don't treat training models as R&D on a financial basis, but they don't have this R&D ahead of time. You get it at spend time. But think about what that means. One simple thing that we've done a lot of tests on is, hey, generate me this code, make this function. I describe the function in a few hundred words, I get back a response that's a thousand words, awesome, and I'm paying per token.

When I do this with o1 or any other reasoning model, I'm sending the same response, a few hundred tokens, I'm paying for that, I'm getting the same response roughly, a thousand tokens. But in the middle there was 10,000 tokens of it thinking. What does that 10,000 tokens of thinking actually mean? It means the model's spinning out 10 times as many tokens. Well, if Microsoft's generating $10 billion of inference revenue and their margins on that are good, they've stated this, they're anywhere from 50 to 70% depending on how you count the OpenAI profit share, anywhere from 50 to 70% gross margins, their cost for that is a few billion dollars for $10 billion of revenue.

Now obviously the better model gets to charge more, so o1 does charge a lot more. But you're now increasing your cost from, hey, I outputted a thousand tokens to I outputted 11,000 tokens. I've 10x'd my spend to generate now not the same thing, right? It's higher quality. And that's only part of it. That's deceptively simple. It's not just 10x, because if you go look at o1, despite it being the same model architecture as GPT-4o, it actually costs significantly more per token as well.

And that's because of this chart that we're looking at here. This chart shows, hey, what is GPT-4o? If I'm generating a thousand tokens, and that's what GPT-4 on the bottom right is, of Llama 405B, this is an open model so it's easier to simulate the exact metrics of it. If I'm doing that, I'm keeping my users' experience of the model constant, i.e., the number of tokens they're getting at the speed. When I ask it a question, it generates the code, whatever it is, I can group together many users' requests. I can group together over 256 users' requests on one server for Llama 405B on an Nvidia server, a $300,000 server or so.

When I do this with o1, because it's doing that thinking phase of 10,000, this is basically the whole context length thing. Context length is not free. Context length or sequence length means that it has to calculate attention, attention mechanism, i.e., it spends a lot of memory on generating this KV cache and reading this KV cache constantly. Now the maximum batch size, i.e., concurrent users I can have, is a fraction of that, one-fourth to one-fifth the number of users can currently use this server.

So not only do I need to generate 10x as many tokens, each token that's generated is four to five times less users. So the cost increase is stupendous. When you think about a single user, cost increase for a single token to be generated is four to five times, but then I'm generating 10x as many tokens. So you could argue the cost increase is 50x for an o1 style model on input to output.

Well, there's good news and bad news here, Brad, which I think is what Dylan's telling us. If you're just selling Nvidia hardware and they remain the architecture and this is our scaling path, you're going to consume way more of it.

Correct. Remember they're also paying for 10x as many tokens. So it's actually, the consumer is paying 50x more per query. But they're getting value out of it because now all of a sudden it can pass certain benchmarks like SWE-bench, software engineering benchmark, which is just a benchmark for generating decent code. There's front-end web development, what do you pay front-end web developers, what do you pay back-end developers versus, hey, what if they use o1? How much more code can they output? Yes, the queries are expensive but they're nothing close to the human.

And so each level of productivity gain I get, each level of capabilities jump is a whole new class of tasks that it can do, and therefore I can charge for that. So this is the whole like, yes I spend a lot more to get the same output, but you're not getting the same output with this model.

So I would say the o1 style model is so early days, people don't even get it. o1 is like they just cracked the code and they're doing it. But guess what, right now on some of the anonymous benchmarks, there's LMSYS which is like an arena where different LLMs get to compete and people vote on them. There's a Google model that is doing reasoning right now and it's not released yet but it's going to be released soon enough. And Anthropic is going to release a reasoning model. These people are going to one up each other.

And also they've spent so little compute on reasoning right now in terms of training time and they see a very clear path to spending a lot more, i.e., jumping up the scaling laws. Oh, I only spent $10 million, well wait, that means I can jump up two to three logarithms in scaling like that because I've already got the compute. I can go from 10 to 100 million, billion to 10 billion on reasoning in such a quick succession. And so the performance improvements we'll get out of these models is humongous in the coming six months to a year in certain benchmarks where you have functional verifiers.

Yeah, so today, o1 is stupendously expensive. You drop down to 4o, it's a lot cheaper. You jump down to 4o mini, it's so cheap. Why? Because now with 4o mini I'm competing against Llama and I'm competing against DeepSeek, I'm competing against Mistral, I'm competing against Alibaba and I'm competing against tons of... you think those are market clearing prices? I think... and in addition, there is also the problem of inferencing a small model is quite easy. I can run Llama 70B on one AMD GPU, I can run Llama 70B on one Nvidia GPU, and soon enough there will be on one set of Amazon's new Trainium. I can sort of run this model on a single chip.

This is a quite a bit easier problem than running this complex reasoning or this very large model. And so there is that difference. There's also the fact that there's literally 15 different companies out there offering API inference on Llama and Alibaba and DeepSeek and Mistral, these different models. You're talking about Cerebras and Groq and Fireworks and all these others, Together, all the companies that aren't using their own hardware. Now of course Groq and Cerebras are doing their own hardware and doing this as well, but the margins here are bad.

We had this whole thing about the inference race to the bottom when Mistral released their Mixtral model which was very revolutionary late last year because it was such a level of performance that didn't exist in the open source that it drove pricing down so fast. Because everyone's competing for API, what am I as an API provider providing you that, why don't you switch from mine to his? Because it's pretty fungible, I'm still getting the same tokens on the same model. And so the margins for these guys is much lower.

So Microsoft's earning 50 to 70% gross margins on OpenAI models and that's with the profit share they get or the share that they give. Or Anthropic similarly in their most recent round they were showing like 70% gross margins. But that's because they have this model. You step down to here, no one uses this model, a lot less people use it from OpenAI or Anthropic because they can just take the weights from Meta Llama, put it on their own server, or vice versa, go to one of the many competitive API providers, some of them being venture funded, some of them with money.

So there's all this competition here. So not only are you saying I'm taking a step back and it's an easier problem, and so therefore if the model's 10x smaller it's like 15x cheaper to run, and on top of that I'm removing that gross margin. And so it's not 15x cheaper to run, it's 30x cheaper to run. And so this is sort of the beauty of, well, is everything commodity? No, but there is a huge chase to, if you're deploying in services, this is great for you. You have to have the best model or you're no one if you're one of the labs.

And so you see a lot of struggles for the companies that were trying to build the best models but failing. And arguably not only do you have to have the best model, you actually have to have an enterprise or a consumer willing to pay for the best model. Because at the end of the day, the best model implies that somebody's willing to pay you these high margins, and that's either an enterprise or a consumer. So I think you're quickly narrowing down to just a handful of folks who will be able to compete in that market on the model side.