Hi, good afternoon everybody. Nice to have you here. My name is Christine. I'm an MD at NScale and I'm very excited to be moderating this discussion. Now before we get to know our panelists, just join me in zooming out just for a minute because if you think about the last decade, we have really measured infrastructure using metrics that mostly mattered to engineers. Now AI is changing all of that. Today there is one metric that directly connects infrastructure to the end user: time to first token, or TTFT.
Now, for those of you who don't know NScale, just a couple of words on us. We are building the physical and digital foundation that enables AI at scale. And we are a vertically integrated AI hyperscaler that own and operate layers of that stack. Now in this discussion what we would like to focus on is very much connected to how we think about AI as an integrated system because performance can never be determined by a single layer in isolation and that system is what Jensen Wang often refers to as the AI five layer cake. You have energy chips, infrastructure, models, and applications. And every layer depends on the one beneath it. And every layer impacts the experience and the performance on the layer above it. Now, we're fortunate to have two leaders with us that are defining those layers of the cake. So, let's get to meet our panelists. Rod, let's start with you. Can you say a few words around your role at Nvidia and also how Nvidia works with partners like NScale and others within the five layer cake?
Sure. So I'm Rod Evans. I'm the European vice president for AI cloud infrastructure at Nvidia. And my whole mission is to make sure that we can provide cloud services out there to all of our customers globally. So working with people like Vast on the infrastructure level for storage and data and with NScale as one of our leading service providers helps to build up some of those layers of the cake, certainly on the bottom three layers of the five layer cake.
Jeff over to you same to you. Could you introduce what you do at Vast and also how Vast works with partners within the five layer cake.
Sure. So I'm Jeff Denworth. I'm a co-founder of VAST Data. VAST has built the kind of the data services layer that's used by most of the world's largest AI clouds today. The company powers somewhere between four to 5 million GPUs around the world, largely made by the NVIDIA Corporation. And yeah, we've been helping organizations build very scalable services platforms. And you know if you think about the five layer cake you have energy and you have processors and you have the compute layer then you have models and applications and VAST sits at the compute layer building basically a platform designed for storage and database services as well as a compute runtime that's used for things like RAG pipelines and agentic pipelines that customers can build on top of NVIDIA hardware using the models of their choice. So, we're the gooey center of the cake, I think, that a lot of people really like to use.
Good way of putting it. I like that.
Everybody got that. Thank you. So, time to first token as we hear and we heard that earlier today as well is suddenly everywhere and it's impacted by the layers of the cake that we just touched upon. So, if you start with your VOD, what has actually changed? Why is time to first token the metric that everybody's talking about and not just focusing on cost per token utilization and return on investment?
So if you look at the sheer investment required to bring these things to marketplace in terms of the data center, the storage, the GPUs, everything that goes with it, then you really need to make sure you're making that infrastructure actually create things that have value, tokens. So it really becomes super important now when you if you're adding a 20 megawatt 10,000 GPU system you're a shade under $2 billion to put that infrastructure in place. Now when you've got that sort of investment sitting there I don't think anybody can sit there with $2 billion not earning any money. So the critical thing you have to do is make sure you get this integrated as quickly as possible and you think of the whole picture. So it isn't just about the GPU, it isn't just about the storage. It isn't just about any aspect of it. It's a really tight integration between all the layers to make sure you're maximizing that investment to get the return back to your business or businesses as quickly as possible.
It's all happening in sync. In other words,
Absolutely. And it's becoming more and more that way as we start to build systems now where you have a lot of GPU sharing with memory between the GPUs. It becomes super important that you have a well thought through infrastructure to make it happen.
So if you want to build on that, Jeff, from your vantage point, what do you think has changed over the last say 12 to 24 months?
First of all, I think the economics have changed, right? If I go back two years ago, people were selling GPU by the hour. And now if you can sell tokens as a service, you can basically take that same asset and sell it for two times as much, right? And so then that takes you down a path of building inference services that ultimately can drive a much better margin profile for organizations. And, to me, NVIDIA is a very exotic animal in that there are things that Jensen pushes that are far in advance of what the market actually sees and he's been talking about token factories for like three years now. And now we're actually seeing them get built where service providers are only selling tokens. And the interesting thing about his pursuits and his thrust into the market is he's pushing for absolute lowest cost of token delivery and that requires full stack optimization all the way from how the power gets deployed up to applications get built on models and NVIDIA is an amazing organization in my book just because they're way ahead of the market and always building for the next two to three years of demand out and that just involves depressing token costs while also giving the best possible service experience. And so yeah, you got to get cost down, but the way you get cost down is by bringing the time to first token down. You just need less machinery waiting for an inference operation to start and then complete.
So as we're hearing it's a whole stack metric and Rod, the floor is set by compute. So if we take the latest updates from NVIDIA's sites, what is so important about those updates and what do you actually change at the hardware level and how do those changes actually affect how latency and ultimately throughput utilization and cost per token are impacted.
So if you look at the latest announcement NVIDIA announced, we've announced improvements all the way along the stack from our new Vera Rubin generation. We've got dedicated product for inferencing and the requirements are also changing. If you look at large language model training or large language models, most of the mathematical operations are matrix mathematics. Now matrix mathematics just needs lots of raw grunt to make it work. Raw horsepower, raw flops. When you get down to inference it's much more looking in terms of memory bandwidth and you get different types of architectures. So we've launched a new architecture that brings in a CPU layer. We bring in a dedicated inferencing chip with the LPX platform. We have Vera Rubin providing the full stack for the large horsepower type applications. And we've done integration improvements all the way through the stack. Got new networking technology. With our next generation system, there's no cables in the back between all the GPUs. It's cableless because that's much more efficient and requires less infrastructure to make it work. But more significantly about all of this is we're really trying to build out this concept of the token factory and build it out there so you have every single aspect of it tightly integrated with the applications, the models, the whole nine yards to make it into the most efficient system you can get in terms of getting that token out there much quicker.
I think it's also fair to say that storage is now being built into the computer as well. And maybe you want to expand on that.
Yeah, adding storage into the layer is absolutely fundamental because if your GPU is sitting there idle, it's not doing any work. And the biggest inhibiting factor today on getting GPU systems and AI systems into production is typically the data infrastructure. By that, do I have data in the right format? Have I cleansed it the right way? Can I get the right performance to keep the GPUs busy? And it's really fundamental that you get a really well architected referenceable data platform to make that work such as provided by VAST Data which is very heavily integrated with the architecture we're putting forward.
