Well, first I want to welcome everyone here. I'm thrilled to see all of you guys here participate in our launch event. This has been an exciting week for us at Arya. We finally can talk publicly about the details of what we're doing. And so I want to thank our customers, our partners, our investors, our board members, everybody is here. And we look forward to showing you more about what we've been building today. I'm going to invite to the stage Gavin Baker. Gavin, it's good to see you.
Thank you for having me, Mansour.
Of course. Well, first, thank you. I know you flew here for this event. So really appreciate that. And you just joined our board. So I want to welcome you to our board and look forward to working with you. So what I wanted to do here is explore the economics of AI factories. You've talked extensively about MFU, model flop utilization. You've talked extensively about token efficiency. You've stated that for AI factories to be successful, they need to be the lowest cost producers of tokens. Can you break this down for us?
Yeah, absolutely. And I actually think it's really interesting and it's awesome for everyone in this room. But at no point in my career as an investor in tech has a big winner been because they were the low-cost producer of anything. It's not like Apple is worth three trillion because they're the low-cost producer of smartphones. Nvidia is not worth five trillion or whatever it is today because they're the low-cost producer of accelerators or silicon. They're just the best. AI is fundamentally different in that at some level the amount of intelligence you get out of an answer is determined by two things: the token efficiency, how many tokens you need to get that answer. Other people call that intelligence density. We'll see which term wins. I like token efficiency. And then cost per token. So for a given level of intelligence, the person who can get there with the fewest tokens and the lowest cost per token is going to win. Because this is something that I think I started actually in aggregates, rock pits, and in any extraction industry, oil and gas, metals and mining, the low-cost producer always wins. And we haven't had that in tech. And if you were the low-cost producer of intelligence, you can choose between actually generating more tokens and having a higher quality product, or you can choose to have higher margins and invest that money in user acquisition, customer service, all these things. Now, token efficiency, that is fundamentally the job of the labs. Full stop. But everyone in this room presumably is here for lowest cost per token. And there is a trade-off, as Dylan referenced. You know, if you can make really, really fast tokens, maybe cost is less important. But strangely enough, the way you get the lowest cost per token is you spend a lot on everything around the accelerator, which makes perfect sense from first principles. If these data centers are AI factories, you know, in my days following manufacturing industries, factory utilization is a key input. In airlines, it's how you know, are the seats full. And the truth is that GPUs, TPUs, Traniums, they are rarely running at full utilization. So we have spent hundreds of billions of dollars on these AI factories and they are running at low levels of utilization. And the network is one of the biggest culprits in this.
So that's why I invested and I'm very excited.
Yes. Well, that's indeed actually if you think about the utilization of these accelerators, it's around 30 to 45% maybe for training, even lower for distributed inference. And so like you pay for all the flops in an accelerator and you're only getting, you know, a very small fraction towards the job. And the larger the cluster, the more inefficient it is.
Absolutely. And indeed the network is connecting all of it. And as the world, maybe you can comment on this, as the world moves from training to more inference at scale and especially distributed inference, right, in these high interactivity use cases that Dylan mentioned, you know, the network becomes even more of a bottleneck.
Absolutely. Yeah. I mean especially with, you know, the expert parallelism and the disaggregation of prefill and decode.
Indeed. I mean, the network is more important than ever. And just if you take your utilization from 40 to 80, you've dropped your cost per token in half. And you've literally, you have dramatically increased your margins. And it's crazy. And on top of all of this, not only have we spent hundreds of billions of dollars on these AI factories that are sub-optimized, often because of networking, but we are increasingly in a watt-constrained world. So, you know, seven states have bills to make it illegal to build data centers. And so, you know, if you have a data center in the ground, it is crazy not to run those GPUs as close to 100%—GPUs or TPUs, XPUs, XPUs is the word. XPUs. I see Hassan from Broadcom. So, XPUs, not GPUs. It's crazy not to run them as close to 100% utilization as you can. And then you can make all sorts of trade-offs between interactivity and latency, throughput, and hit the right target for your workload and your users.
