We're not just a distributor of other people's IP. We own our own IP.
But how do you change the hearts and the minds of the broader US demographic with regards to artificial intelligence mythos? I think it's rumored to be the first 10 trillion parameter model.
It's better to have your own chips and demand than not having your own chips.
Where's the next big bottleneck?
The next big bottleneck will be largely around...
Is there some line in the sand, some benchmark you would determine Gemini is no longer safe to release publicly?
We have more demand than we can possibly meet from all the other AI labs.
Thomas, what keeps you up at night?
All right, Thomas, thank you for joining me today. We're at the Google Cloud Campus and I really appreciate your time today.
I am super excited to talk to you. I have a bunch of questions for you.
The first question I've been thinking about so much lately is TPU capacity. When you look at the other frontier labs like Anthropic and OpenAI, all they talk about is being compute constrained.
But then you see Google over here who you know you have the full stack, you have your own chips, but you're not only serving your own inference, you're training, you're selling inference, you are also allowing some of your competitors to build on top of your own chips, you're also selling your own chips. How do you have so much capacity or how do you think about it whereas the other frontier labs can't seem to get enough?
If you think about what percentage of the world are we monetizing and in some places we monetize the tokens as the token and the chip, in other places we monetize the token somebody else's model but using a chip underneath it. And part of the reason is we go back many many years and we do long-term planning. And so when we saw this AI moment coming we looked at a number of different factors to ensure we were not physically constrained. We diversified how many energy sources we had. We locked in real estate so we could build data centers. We changed how we manufacture data centers. We don't build them in construction. We shifted a lot more to manufacturing because manufacturing can always do faster than you can do with construction. We reduced cycle time to deploy machines. All of that was stuff we've done and that helps us capacity wise. And then on the silicon side, we've always worked with Nvidia as a partner, but we've also wanted to build our own silicon and we've done it for I think it's like 11th year now or 12th year. Eighth generation TPU will be announced at our event.
Yeah, we're going to talk about that.
And so that's something we've got real art of doing over and over and over and delivering that advantage over and over. And it's something we've delivered. And the interesting thing is we see demand now from not just from the AI labs but from other segments. You'll see Citadel for example in capital markets talking about how they using our TPUs. You'll see department of energy and high performance computing customers talk about it. So we're also seeing TPUs becoming more general purpose infrastructure not just for AI algorithms.
And so when you're looking at monetizing TPUs across all of the different avenues that you have to allocate your compute, how does it compare? Can you maybe if you want to share specific numbers, great. But if you're looking at just comparing and contrasting selling a TPU versus allowing Anthropic or OpenAI to serve their inference through your infrastructure versus your own Gemini models, how do those different avenues compare?
We balance investments across all these, you know, and we make great margins no matter which way we're selling it because we own our own IP. We're not just a distributor of other people's IP. I think that's helped us and you've seen us improve both topline and operating margin. We've also moved TPUs now as for example when you look at capital markets. One of the things we find that's very interesting is that algorithmic trading was done using numerical computation which was largely done on traditional compute that's been constrained by the Moore's law. The incremental improvement you see generation to generation is getting slower and so many of the top firms have seen the huge improvements they can get by shifting to inference. Instead of doing computation using numerical techniques if you shift to inference you can write the improvements you're seeing in inference time and as they've come in they want our machines in their venues where it's closer to where the exchanges are for example.
So we've started taking TPUs and making available in other people's data center for some of our key customers and that's a slightly different business model. And we have you know in macro I would say diversification improves product because you see requirements from many places. Diversification in monetization also helps us grow. I mean when we deal with supply chain vendors for example because we are using these chips not just for our own needs but also offering them a market they say well Google's demand is a sum total of a much larger pool and as a result we get favorable terms.
I want to stick on this point for a moment longer. If compute demand is infinite, I mean even just for the R&D side, why not just hoard the compute? Why not just keep it? And to put a finer point on it, if AGI is really the goal that all of the AI labs are heading toward and whoever hits it first and is able to scale it out wins, it seems like keeping the capacity, keeping it for yourself, keeping it for your own models is actually quite beneficial. What am I missing?
You have to make money to fund all of this.
Google makes a ton of money, but you have to keep generating cash flow to do it.
And this is one more lever for us to generate sufficient cash flow. And the amount we allocate to other people is balanced always against our own needs and our own capital requirements. And you know, no matter which lab you're in, venture capital cannot fund you indefinitely.
Yeah. Yeah. And as compute costs grow, if you're running a loss leader business where you're losing money, and you're not making enough money from inference and other techniques to cover the cost of training, as that gap gets wider, the number of sources you can go to gets smaller.
I've been talking about how Google is in such a unique position. They have the cash cows, they have the chips, they have the models. Does your Gemini team ever come to you and say like we don't have enough? I know I'm really stick on this point. It's just so wild for me to hear where these other companies are just not able to keep up.
There's always demand for these things and I think for the next 10 years there will always be more demand than supply and that's a good place to be in if you have your own chip. If you don't, you're reselling other people's stuff. And in a capacity constrained environment, your unit economics get more expensive. And in our case, because we control the chip, the unit economics remain attractive. So that is going to be an advantage for us because we own the silicon.
So if you look at the whole pie of your TPUs of your compute infrastructure, can you talk a little bit about what the split is between training, inference, selling TPUs, serving inference for other labs?
Broadbrush, I think we don't talk about the details, so I'm not going to go through every element.
But broadbrush, if you look in macro, cloud is about half of Alphabet's capital and it's growing because it's growing much faster as you know. So that's like a split and then on our side a significant part of our growth is coming from Gemini and our models and so you can use that as a rough approximation.
Okay. You mentioned data centers and building out data centers. Can you explain what the difference is between construction and manufacturing when you're talking about data centers?
Yeah, it's just what is the unit at which you're deploying capacity. So you could take for example a rack of machines, assemble it in a data center. You could take an entire row of machines and deploy it in a data center. The higher the grain at which you can deploy, the more you can preconstruct it, pre-test it in a central location, which means you're much faster in deployment.
When you're planning deployment of a new data center, I think you're probably more aware than anybody there's a pretty negative sentiment about data centers in the US specifically. I think it's 20% favorability. How do you think about that and how do you think the broader AI industry can start to change the opinion, change the sentiment around artificial intelligence and specifically deploying data centers which gives the US a strategic advantage? I tend to be very optimistic about AI in general. So, how do you think about that?
What people are really concerned about on data centers is a couple of things. Number one, you know, will energy costs in my state or my county go up?
Second, will there be sufficient employment in the local community in which the data center operates? And so, there's a couple of things that we're doing. First one is we're investing in behind the meter technology where we're not taking energy off the grid and we cross connect to the grid if the state wants it so that in case there's short supply on the grid our energy can fuel the grid. We're investing in alternate forms of energy because we think that the traditional mode of generate and distribute is not necessarily the only way that energy supply will come into the market. And so one of the things we're looking at is can you reduce the unit cost of energy with new forms of energy delivery created by the demand for AI but then can serve the broader market. Third, we take a lot of attention to making sure the unit of energy we're consuming, what's called PUE, that we have the best in the industry. Meaning, if you need 100 megawatts of compute, how little incremental megawatts do you need from the energy source so you're not wasting energy? And we're by far the most efficient at that in the world. And there's a thousand things that go into that on thermodynamic exchange, how we do heating, all of that. Lastly, we're investing in the communities in which we're in. And to avoid the communities feeling like Google's deploying in one giant location, we distribute it in many places so that no individual state feels like we're becoming a big burden on their resources. And we've had a great track record. I travel to many of our data centers. And when you go to the local economy and see the kids in the school systems and you see the employees who operate our data centers who are super important to us, how much economic development we bring to those rural communities where, you know, we think that's part of our responsibility.
That's fantastic. What about the broader sentiment and not the local community? Because when you go in, you create jobs, you're investing, you're not using the electricity and driving up the prices directly. That's all wonderful, but how do you actually change the hearts and the minds of the broader US demographic with regards to artificial intelligence?
That's going to be a process, you know, and I think it's finding places where you can apply the technology in a way that's good for society rather than just causing issues where people are worried about job displacement. I'll give you just a few examples of things.
You'll see in our keynote a company called Signal. They're a health insurer based in Germany. They're the largest health insurer in Germany. They today deploy a lot of agents built on Gemini Enterprise to help their teams do work. Now the really interesting thing was there was a lot of anxiety when we started the work with them that it would mean job displacement. They have not let go anybody and in fact what they found is the accuracy with which and the speed with which they can answer questions from people about am I eligible for this treatment or not. It's cut down from in some cases 23 minutes to research an answer now to less than a few seconds. And so that's improved efficiency. It's improved quality of their customer care and they've not let go a single job. We work for example with the American Society for Clinical Oncology. They're the large, the 51,000 members, every oncologist in the United States. They wanted an application where AI was helping a doctor sitting down to deal with a patient to understand standard of care guidelines which is this person's come in they have breast cancer what's the guideline it turns out they're also diabetic I can't prescribe chemo if they're diabetic of this kind there's a lot these rules are incredibly complicated in many cases they overlap and they wanted some help on providing answers. The answers have to be 100% correct. You can't have hallucination and we've helped them and it helps doctors take care of patients and the feedback from their membership has been incredibly rewarding to see. So there are lots of examples and we always say the most important thing for example is building a wealth advisor. So today, if you think of the average citizen, if you're a high net worth person, you can go to a private bank and you have wealth management professional advise you. If you're an average person who doesn't have those financial resources, you may not get high quality advice. Citigroup is building a wealth advisor. They're going to be showing it at the event which allows you to use our reasoning and task management capabilities in Gemini to advise people and also to help them take investment actions if they want. So these are examples of things that society is going to find beneficial. It takes time to get the balance between the AI is going to cause massive job displacement to also hear from this side of things. And that's part of the journey we're on as a society.
I do agree. I think job displacement in particular is something the general population in the US is extremely worried about. Let me ask you directly for your organization, Google Cloud. Now that you're seeing your engineers and other parts of your organization be more productive because of artificial intelligence, more automation, are you hiring? Are you letting go folks? Are you stable? Where are you on that front?
We're adding people for products and sales. We're hiring a lot of people in a go-to-market organization. We're hiring a lot for deployed engineers. And in places where we're building new product, we're adding capability. Like an example I will tell you, here's an example of what people don't see. A long time ago we said as models became more sophisticated in understanding code and second as models learn how to use computers to do tasks there are lots of things they can do amazingly well but one of the issues with understanding code is they can also find vulnerabilities in code.
And so enormous anxiety about cyber security vulnerabilities from some of the new models.
We're going to talk about that. Made a decision a long time ago to do three things. Number one, helping improve Gemini as a way to detect issues in code. And we have a lot of customers using it. Second, helping build a model that can repair code because if you're finding vulnerabilities very quickly, people may not be able to keep up. So, can a model assist you in fixing it? And we have new capability coming for that. When we acquired this company Wiz, you'll see us showing new capability with Wiz and it's really about continuous detection.
You know, people call it continuous red teaming. So, we're going to show three different types of agents. An agent that continually attacks you to make sure your vulnerabilities are being fixed and you don't get caught off guard, which you couldn't do before. An agent that prioritizes the issues that are being discovered so that you can get, okay, these are the ones I really need to fix and then a third one that helps you fix them.
I'm glad to hear you're still hiring. You're more productive and hiring. There are companies out there. I think Block is the big one. You know, Jack Dorsey put out this blog post. Block laid off half of their organization, blamed AI or pointed at AI as the reason. What do you think the difference is between like how Google sees this productivity increase and is still increasing employment versus Block who said, 'No, we're actually going to transform the company. We need half as many people and we're going to do things better.' Where's the discrepancy there?
Every company has demand for its products and services and each CEO makes their own decisions. We're seeing plenty of demand and so we're investing.
Let's talk about Nvidia for a moment. Jensen just did a podcast with Taresh and he talked about how Nvidia and their architecture is the cheapest on a per token basis overall total cost of ownership that's because of CUDA and NVLink networking tooling delivering better tokenomics. Do you agree with that assessment? Do you think Google is the best overall total cost of ownership? And if not how does Google catch up?
We have a lot of customers who say we are the best total cost of ownership.
Hey, I guess that was the answer, right?
I mean, the reality is if you're an AI lab, you choose the best platform. It's not just our own teams that use it. We have more demand than we can possibly meet from all the other AI labs. And so, I would just tell you that they would not be asking for TPU if we were much more expensive.
Is a big factor of what makes TPU special the speed? I've noticed the Gemini family of models is very fast and as a speed maxi myself, I very much appreciate the speed. And typically when you're looking at ASICS, they're specialized. They tend to be a lot faster than the generalized GPU. Like is that a selling point for a lot of AI labs or for your own customers or are they still saying quality all day?
Quality. Quality. It's a combination I would say three core elements because I think it's not the chip, it's the system. A TPU system for example 8 has 9600 chips. 8i is I think 1152 all on a single optical Torus network. So there's incredibly high bandwidth, super predictable latency across all the chips in a pod. And that gives you for example when you look at the speed with which we're able to take stuff out of the memory for processing and to put stuff back in memory it's extraordinarily efficient. Just to give you an example 8T the training chip can fit two petabytes of memory in a single system. Two petabytes is like 100 times the size of all the Library of Congress digitized.
