Hello, I'm John Furrier with theCUBE. We are here in Seattle for the exclusive interview around Reinvent Preview and all the news coming out. So we had Matt Garman, the CEO of AWS. Welcome back to theCUBE.
Thanks for inviting me back to this great studio here.
Yeah, actually thank you. Thank you for coming to Seattle. Welcome.
So on the keynote announcement, there's a lot going on with AI infrastructure. You guys are leading in the CapEx. There's a big sovereign conversation with global ecosystem, a lot of national conversations. But AI factory seems to be dominating the conversation and all the discussions about what's powering it. And I was saying on LinkedIn and X, AI factories are like the cloud. It's got a similar thing to it. So a lot of the trends are pointing to large scale compute, accelerated computing, XPUs. This is the hottest area right now in build out of, I call AI factory areas, not just data centers, they're like full-blown systems. What's your vision on how the infrastructure is changing and evolving?
Yeah, I think, look, it's a fun time to be in the cloud for sure. And I agree there's always kind of a new name for what the cloud or things are, but we are seeing that particularly for the largest customers building some of these AI systems, they are needing massive, massive areas of compute. We've talked about our project Rainier, which is together with Anthropic, 500,000 Tranium two chips all in basically a same data center campus. And we used to say some phrases like the rack is the new computer and the data center is the new computer. Now we kind of say the campus is the new computer because it really is... These are massive compute installations and for the very largest customers. But I think the cool part about that is the innovation that comes back to every customer out there who's interested in going and building models. And not every customer is going to go spend tens of billions of dollars to go build a frontier model, but every customer wants to be able to take advantage of AI. Every customer wants to be able to go and build. And some of the announcements that we have at Reinvent are really targeted at how do we help customers be able to go get that value out of all of this spend that's going in there. And as you mentioned, AWS is spending a large amount of capital to go and build out data centers all around the world to land compute infrastructure, to land network infrastructure, to land storage infrastructure, so that customers can keep seamlessly scaling. And we're landing different new types of compute, we're landing new types of clusters, and we're landing new services to help them take advantage of all this technology that's coming out.
One of the things in following you guys since the beginning is just the growth of just how you guys grew, but now as this next level hits, the conversations keep coming back to the chips, the silicon. Is there any economics that have changed? Because your core competency that you guys have built has been building large scale systems. We've been covering the new region expansions, now you've got big data centers, you've done multiple deals where you're going out and building more AWS data centers. That's a core competency. Is that going to be a service? And how does that service look? Because it's almost like reminds... I won't say outpost, but remember the outpost was like, okay, we'll put it at the edge, we'll put AWS out there. These large data centers that you're building have a distributed component to them. And with AI, tie that together because these are essentially you're building out the infrastructure and how do you expand that out when the software stacks are changing? Energy is bounding everything. You've got software stacks around the agentic coming.
Yeah. Look, I think that there's going to be a small number of customers who want to have these dedicated AI factories, as we call them, where they can go and we'll put large amounts of compute next to where they're going to be. These are going to be sovereign nations. These are going to be US governments and secret agencies. These are going to be really, really, really large customers. However, I think that the vast majority of customers are going to continue to take advantage of this technology inside of an AWS cloud. They're going to go to US East or they're going to go to EU West or wherever their region is and take advantage of these technologies there. And for us, that investment in silicon is one of the things that it can be invisible to customers under the covers, but what our job is to deliver better performance at lower costs and at better latencies so that customers can keep getting value in their applications. And I think sometimes that gets lost. And it's why we invest in our own custom silicon because we think customers want a choice. We think customers are going to want to use the best infrastructure for the workloads that they have. And at Reinvent, we announced the GA of Tranium 3, really excited about Tranium 3. It's the most powerful AI system available out there today. And as we've talked about before, we're kind of bad at naming, but it turns out that Tranium 3 is the world's best inference platform. And so these are no longer chips, they're no longer servers. A Tranium 3 Ultra server is actually two racks of computers that are all kind of networked together to give you this really, really large instance that is a super powerful AI system. And that is incredibly exciting and gives great differentiated performance for customers and we're excited about that roadmap. We think Tranium 3 is going to be a blowout success and is today more than 50% of the tokens that are served through Bedrock are already running on Tranium. So Tranium already runs more than half of our tokens all going through Bedrock. And we think Tranium 3 is a real step change function in what's going to be possible there. We also pre-announced that we're already a hard at work at Tranium 4, which you might expect, but we announced that to the audience today and super excited about that as well. I think Tranium 3 is about a 4X increase in the amount of compute compared to Tranium 2. Tranium 4 is like an 8X jump again in the amount of compute that's available. It's a further jump in memory bandwidth so that you can really kind of streamline the throughput of inference through these platforms and training through these platforms. Really excited about that. The team has been just incredible at turning that innovation into value for our customers. And that's what I'm most excited about.
You mentioned tokens, pumping out a lot of tokens. One of the things when I ask people on the queue, 'Hey, what's an AI factory?' It's quite the buzz where it's become a thing. And it's really more than a data center. It's really a lot more. And some answers are data in, tokens out. Some say data in, intelligence out. So the theme is bringing AI into physical things, whether it's workflows, digitally, or physical AI. So the question is that as you see Amazon bringing AI to things because the way the models are ruling, you need all that horsepower. What's under the hood? Can you explain, because everyone thinks, oh, Nvidia is doing it alone. And they're not. They work with you guys very closely. So talk about what you guys are building that's going to deliver that token value. You're doing a lot internally with Anthropic and you're publicly talking about it. So everyone wants more tokens.
Yeah. Well, and it's not just tokens. And I think the key thing, and actually, if you take a step back, it's fun to talk about like an AI factory. 99.999% of customers will never purchase an AI factory. They'd really just want to use the cloud in their environment and build inference into their applications. And so it's a fun thing to talk about, but most customers are not going to purchase that. What they're going to buy though is what they... And actually when you get down and you talk to an enterprise, what they realize is their differentiated value is data, right? It's the IP that they have inside of their company. It's what makes insurance company A different from insurance company B or bank A different from bank B or retailer A different from retailer B. They have this data and it's about their workflows. It's about their customers. It's about the IP that goes into their business. And when I talk to a lot of customers, they say, 'Look, AI is awesome. It's like incredible models that are enabling content summarization. They're enabling workflows. They're enabling, pulling together a bunch of information.' And this is an interesting kind of observation is they say, 'Look, I wish I had a model that deeply understood my company. It deeply understood all of my data and everything like that.' Now, obviously all of the model companies out there in the world don't have access to this private set of data. And this is the tricky part. And so a lot of companies are like, 'I'd love to train my own data.' Because then those tokens that are outputted would know me. They're not generic tokens. They're tokens that know my company, but I don't want to spend $10 million to go build, and I don't have the skills to build a frontier model. So we actually sat back and said, how do we get the customers get more value out of those tokens? How do they get more value for their enterprise? It's not just tokens, right? So generic tokens are actually useless. They're just bits of data if they're not useful and integrated into your company. And so part of the announcement we had at Reinvent was around our new set of Nova models, which are fantastic. They're really, really great and a leap forward in what's possible, particularly from a price performance, a latency perspective and capabilities, really impressive with regards to Nova 2 Light and Nova 2 Pro. Very excited about those. What we also announced though is this program called Nova Forge. And Nova Forge, what it does is it introduces the first ever idea of open training models. And so there's open weights models today where people can kind of take a completed model and then do a bunch of post training on it and play with the weights and try to customize their model. What we saw is what happens is those models actually forget all of the knowledge they got originally, right? And so you actually, customers, they try and they get somewhat better results, but can't really customize it. What we're doing with Nova and Nova Forge is we're allowing customers to insert their proprietary data earlier in the training process. And then we're having a proprietary set of Amazon curated data that we will mix with that data to kind of finish training the model. So to the end of the day, the customer has effectively a privately built frontier model that deeply knows their enterprise data and is at the frontier of capability. And then they have that for their own use. And then you get back to now the tokens are super valuable to them and they know their company, they deliver results and we're very excited about this.
This is a huge deal, I think, because one of the things, you look at the models when they first came out, the frontier standard was defined by large scale, multi-step reasoning, persistence, memory and all those things. Unpack that because what this means is you're essentially... It's almost like an AI development kit if you think about it in a kind of weird way. But talk about the impact of that because the question I would have is, okay, is that secure on my premise? Is that going to be on the cloud and does other people see the data or you're going to give me your Nova. It's kind of half-baked and I integrate my stuff in and then I have a persistent brain basically.
That's right. So the idea that we have is that every single startup and enterprise can have their own custom frontier model and it's only for them. And so once you take your data through a bunch of tools and recipes that we make available, mix it together with the proprietary kind of curated Amazon data set of all the things that we've put together so that the model knows how to do reasoning and knows how to do intelligence, et cetera, but together with your data and then it's secure inside of your VPC. No one has access to it. None of that data goes back to the Nova team. All of that data and your own version of your model stays inside of your VPC. And so you know it's secure, you can run it from there, you can upload it to Bedrock and run it in a serverless way so you can actually get all of the benefits of Bedrock running this model serviceably, but no one else has access to it, including us. And this is why customers for the last 20 years have trusted Amazon to keep their data safe.
Talk about what the alternative, I mean, because what you're kind of getting at is that the heavy lift to build the frontier model is a lot of work. Costs a lot of money and you're going to open that up and democratize that and enable that. And on the other hand, I could see a significant performance enhancement because there's a lot of kind of knowledge in the data. Does that fill up on the performance side too?
Hugely. So we have a couple of really great beta customers who've been working with us, take Reddit as an example. They were super interested in this idea where they could infuse their data into Nova so that it really deeply understood what they think about and what values the Reddit community has and how that tone should be. They've seen much, much, much better results than any commercial model or anything that they've seen out on the market.
So you're basically saying you think AI factories will be these large scale data centers, whether it's in the cloud or servicing kind of cloud architecture and that the enterprises will move to these highly efficient frontier models that'll embed their data estate into it.
That's right. Exactly it.
And what does that mean? Give an example of what that would look like because this could change the economics of the enterprise.
Yeah. So I'll use this exact example where somebody like Reddit was trying to struggling with, how do I get these models to deeply understand my own data? Instead, what they did is they took a checkpoint of a... And we basically said, look, we can train one of these large frontier models one time and then let everybody else kind of ride on top of that and benefit from that capability. And so what we do is we take a checkpoint, allow them to mix their data in, our data gets mixed in. Now they have a copy of the model for a very, very small fraction of the cost of training their own model. And they have this custom model that they can use for however they want. They can use that model to distill smaller models. They can use that model to train their own kind of internal workflows. They can use that model to go build interesting experiences for customers. We've talked to banks, we've talked to manufacturing companies, we've talked to pharmaceutical companies. Everybody is quite excited about this and I think this is really what people actually want to do.
I think it was two years ago I asked you here in this room if you would work with OpenAI at that time, they were not under contract with you guys, but now you all have worked with them. We also talked about how models will work with each other. So what you're kind of getting at now is if I'm an enterprise, I get my model, the frontier capability, I then embed my data estate in there, all my intellectual property, proprietary data, and now I can then use other models in Bedrock or other places.
Well, you can always use other models, but you can use your own kind of custom model in conjunction with those other models. So of course there's going to be lots of use cases where you're still going to want to use Claude Sonnet model as an example. Or there's other use cases where you're going to want to use open source models, whether they come from OpenAI or they come from Quinn or they come from Llama or from Meta. You're going to want to combine a bunch of these things together for the best. Some of them will be cheaper and more performant on certain use cases, but you have your own custom model for the workflows that really matter for your data and your kind of core line of business applications. And we think it's going to be a really great capability for a large set of customers.
Well, give you props, Matt. You got it right on the future two years ago, what we talked about actually happened. I want to ask you on that thought, okay, I love this direction, but now we're talking about distributed computing at the edge. If I walk into a retail outlet, I got wireless systems, they're connected to the internet using cloud services. You've got this idea of a hyper convergence with wireless. You guys have Prometheus, a lot of work going on there. There's a lot of telecom activity happening. What does the AI factory cloud vision look like when you say, how do I bring all those smart models across the network of a company because I might want to have an edge computer vision and have custom models come to the edge? Because it's a hybrid distributed environment. What's your vision on how this progresses? Because you're going to see models come out faster now. I want to train at the edge too.
Yeah. Look, I think there's a lot of interesting kind of unsolved problems there at the edge. I think frankly, I kind of differ a little bit of where I think the world is going right now. What I'll tell you is I think, and the more I see this, I see actually things moving more towards the cloud as opposed to away from the edge. And kind of how I think about it is because the edge is compute constrained, because the edge is power constrained, you don't actually want to use all the models there. You're not going to have access to the data at the edge. You're going to have little bits of inference. You may have your Alexa devices be able to kind of recognize wake words, but there's so many things you're going to want those models to do, you're not going to put them at the edge. And so the edge inference is like a small amount of little inference there where you want that to be smart, you want that to be small. But the vast majority of that hybrid world is actually trending towards the cloud because the models are bigger, the data set is bigger, the knowledge that you want to get out of those is bigger. And so that's actually a trend I see, which is not things moving out to the edge. I think it's more-
So it translate into the Amazon speak of cloud early days. It's like an API call in your mind. Is that how you see it? Because it is calling a model.
And that's exactly right. And so in the API call world, and I'll just use the Alexa device as an example, the Wake word kind of is a local kind of AI engine that can recognize that you said Alexa. And then when you ask it, 'Hey, summarize this or do a conversation.' It sends API requests effectively back to the cloud where these smart sets of a bunch of different models can answer questions, respond to you, have a conversation, summarize data, et cetera, make recommendations.
