Google Cloud CEO: Anthropic, TPUs, Mythos, NVIDIA and more

We're not just a distributor of other people's IP. We own our own IP.

It's better to have your own chips and demand than not having your own chips.

The next big bottleneck will be largely around...

We have more demand than we can possibly meet from all the other AI labs.

Thanks for having me.

Sounds good.

Mhm.

If you think about what percentage of the world are we monetizing and in some places we monetize the tokens as the token and the chip, in other places we monetize the token somebody else's model but using a chip underneath it. And part of the reason is we go back many many years and we do long-term planning. And so when we saw this AI moment coming we looked at a number of different factors to ensure we were not physically constrained. We diversified how many energy sources we had. We locked in real estate so we could build data centers. We changed how we manufacture data centers. We don't build them in construction. We shifted a lot more to manufacturing because manufacturing can always do faster than you can do with construction. We reduced cycle time to deploy machines. All of that was stuff we've done and that helps us capacity wise. And then on the silicon side, we've always worked with Nvidia as a partner, but we've also wanted to build our own silicon and we've done it for I think it's like 11th year now or 12th year. Eighth generation TPU will be announced at our event.

And so that's something we've got real art of doing over and over and over and delivering that advantage over and over. And it's something we've delivered. And the interesting thing is we see demand now from not just from the AI labs but from other segments. You'll see Citadel for example in capital markets talking about how they using our TPUs. You'll see department of energy and high performance computing customers talk about it. So we're also seeing TPUs becoming more general purpose infrastructure not just for AI algorithms.

We balance investments across all these, you know, and we make great margins no matter which way we're selling it because we own our own IP. We're not just a distributor of other people's IP. I think that's helped us and you've seen us improve both topline and operating margin. We've also moved TPUs now as for example when you look at capital markets. One of the things we find that's very interesting is that algorithmic trading was done using numerical computation which was largely done on traditional compute that's been constrained by the Moore's law. The incremental improvement you see generation to generation is getting slower and so many of the top firms have seen the huge improvements they can get by shifting to inference. Instead of doing computation using numerical techniques if you shift to inference you can write the improvements you're seeing in inference time and as they've come in they want our machines in their venues where it's closer to where the exchanges are for example.

So we've started taking TPUs and making available in other people's data center for some of our key customers and that's a slightly different business model. And we have you know in macro I would say diversification improves product because you see requirements from many places. Diversification in monetization also helps us grow. I mean when we deal with supply chain vendors for example because we are using these chips not just for our own needs but also offering them a market they say well Google's demand is a sum total of a much larger pool and as a result we get favorable terms.

You have to make money to fund all of this.

And this is one more lever for us to generate sufficient cash flow. And the amount we allocate to other people is balanced always against our own needs and our own capital requirements. And you know, no matter which lab you're in, venture capital cannot fund you indefinitely.

There's always demand for these things and I think for the next 10 years there will always be more demand than supply and that's a good place to be in if you have your own chip. If you don't, you're reselling other people's stuff. And in a capacity constrained environment, your unit economics get more expensive. And in our case, because we control the chip, the unit economics remain attractive. So that is going to be an advantage for us because we own the silicon.

Broadbrush, I think we don't talk about the details, so I'm not going to go through every element.

But broadbrush, if you look in macro, cloud is about half of Alphabet's capital and it's growing because it's growing much faster as you know. So that's like a split and then on our side a significant part of our growth is coming from Gemini and our models and so you can use that as a rough approximation.

Yeah, it's just what is the unit at which you're deploying capacity. So you could take for example a rack of machines, assemble it in a data center. You could take an entire row of machines and deploy it in a data center. The higher the grain at which you can deploy, the more you can preconstruct it, pre-test it in a central location, which means you're much faster in deployment.

What people are really concerned about on data centers is a couple of things. Number one, you know, will energy costs in my state or my county go up?

Second, will there be sufficient employment in the local community in which the data center operates? And so, there's a couple of things that we're doing. First one is we're investing in behind the meter technology where we're not taking energy off the grid and we cross connect to the grid if the state wants it so that in case there's short supply on the grid our energy can fuel the grid. We're investing in alternate forms of energy because we think that the traditional mode of generate and distribute is not necessarily the only way that energy supply will come into the market. And so one of the things we're looking at is can you reduce the unit cost of energy with new forms of energy delivery created by the demand for AI but then can serve the broader market. Third, we take a lot of attention to making sure the unit of energy we're consuming, what's called PUE, that we have the best in the industry. Meaning, if you need 100 megawatts of compute, how little incremental megawatts do you need from the energy source so you're not wasting energy? And we're by far the most efficient at that in the world. And there's a thousand things that go into that on thermodynamic exchange, how we do heating, all of that. Lastly, we're investing in the communities in which we're in. And to avoid the communities feeling like Google's deploying in one giant location, we distribute it in many places so that no individual state feels like we're becoming a big burden on their resources. And we've had a great track record. I travel to many of our data centers. And when you go to the local economy and see the kids in the school systems and you see the employees who operate our data centers who are super important to us, how much economic development we bring to those rural communities where, you know, we think that's part of our responsibility.

That's going to be a process, you know, and I think it's finding places where you can apply the technology in a way that's good for society rather than just causing issues where people are worried about job displacement. I'll give you just a few examples of things.

You'll see in our keynote a company called Signal. They're a health insurer based in Germany. They're the largest health insurer in Germany. They today deploy a lot of agents built on Gemini Enterprise to help their teams do work. Now the really interesting thing was there was a lot of anxiety when we started the work with them that it would mean job displacement. They have not let go anybody and in fact what they found is the accuracy with which and the speed with which they can answer questions from people about am I eligible for this treatment or not. It's cut down from in some cases 23 minutes to research an answer now to less than a few seconds. And so that's improved efficiency. It's improved quality of their customer care and they've not let go a single job. We work for example with the American Society for Clinical Oncology. They're the large, the 51,000 members, every oncologist in the United States. They wanted an application where AI was helping a doctor sitting down to deal with a patient to understand standard of care guidelines which is this person's come in they have breast cancer what's the guideline it turns out they're also diabetic I can't prescribe chemo if they're diabetic of this kind there's a lot these rules are incredibly complicated in many cases they overlap and they wanted some help on providing answers. The answers have to be 100% correct. You can't have hallucination and we've helped them and it helps doctors take care of patients and the feedback from their membership has been incredibly rewarding to see. So there are lots of examples and we always say the most important thing for example is building a wealth advisor. So today, if you think of the average citizen, if you're a high net worth person, you can go to a private bank and you have wealth management professional advise you. If you're an average person who doesn't have those financial resources, you may not get high quality advice. Citigroup is building a wealth advisor. They're going to be showing it at the event which allows you to use our reasoning and task management capabilities in Gemini to advise people and also to help them take investment actions if they want. So these are examples of things that society is going to find beneficial. It takes time to get the balance between the AI is going to cause massive job displacement to also hear from this side of things. And that's part of the journey we're on as a society.

We're adding people for products and sales. We're hiring a lot of people in a go-to-market organization. We're hiring a lot for deployed engineers. And in places where we're building new product, we're adding capability. Like an example I will tell you, here's an example of what people don't see. A long time ago we said as models became more sophisticated in understanding code and second as models learn how to use computers to do tasks there are lots of things they can do amazingly well but one of the issues with understanding code is they can also find vulnerabilities in code.

We're going to talk about that. Made a decision a long time ago to do three things. Number one, helping improve Gemini as a way to detect issues in code. And we have a lot of customers using it. Second, helping build a model that can repair code because if you're finding vulnerabilities very quickly, people may not be able to keep up. So, can a model assist you in fixing it? And we have new capability coming for that. When we acquired this company Wiz, you'll see us showing new capability with Wiz and it's really about continuous detection.

You know, people call it continuous red teaming. So, we're going to show three different types of agents. An agent that continually attacks you to make sure your vulnerabilities are being fixed and you don't get caught off guard, which you couldn't do before. An agent that prioritizes the issues that are being discovered so that you can get, okay, these are the ones I really need to fix and then a third one that helps you fix them.

Every company has demand for its products and services and each CEO makes their own decisions. We're seeing plenty of demand and so we're investing.

We have a lot of customers who say we are the best total cost of ownership.

I mean, the reality is if you're an AI lab, you choose the best platform. It's not just our own teams that use it. We have more demand than we can possibly meet from all the other AI labs. And so, I would just tell you that they would not be asking for TPU if we were much more expensive.

Quality. Quality. It's a combination I would say three core elements because I think it's not the chip, it's the system. A TPU system for example 8 has 9600 chips. 8i is I think 1152 all on a single optical Torus network. So there's incredibly high bandwidth, super predictable latency across all the chips in a pod. And that gives you for example when you look at the speed with which we're able to take stuff out of the memory for processing and to put stuff back in memory it's extraordinarily efficient. Just to give you an example 8T the training chip can fit two petabytes of memory in a single system. Two petabytes is like 100 times the size of all the Library of Congress digitized.

And because it's the super low latency network, your throughput from memory into the chips themselves are extremely fast. Third, as if you look above that layer, from a programming stack point of view, you have a lot of tools that Google has built and given to the industry that's used for compiler optimization. For example, JAX, we've done great work with PyTorch, you know, XLA, Pathways, these are all technology that Google's built. And so you put all of that together and even if you look at inference VLM, there's a number of technologies that we super optimized. It's that whole stack that makes that TPU system so efficient and so powerful. And you see that measure through what we call goodput. Goodput is how much effective throughput are you seeing. We also made some decisions a long time ago like three four years ago for instance we saw that energy to your point earlier on energy was going to be short supply so we focused on optimizing the dollars per watt or tokens per watt and I think that's another element that you see a lot of people wanting.

Yes.

I would say the history we have across the different layers of the stack has compounded over time. When we did TensorFlow, we realized you needed a large scale distributed programming model for training and we built JAX for example that was something that was compounded on our history of learning from what people were trying with TensorFlow and needing a new distributed training model. Right? So some of these accumulate over time because we learn from what we're doing from prior and we're making new improvements. We're also incredibly attuned to the market listening to customers making decisions like people asked us why did you build 8i you know the inference chip it's because we've seen that as you eventually no matter how rich you are you cannot fund training without making money on inference and so you have to at least cover the cost of your training from a break even point of view over time you can't just always depend on venture capitalists to fund you. And so we said there's going to be a big demand for inference. We knew what the factors we needed to optimize for inference and you know frankly the demand for the inference the 8i has been way way more than we expected.

Ironwood was a mixture. It was used for training and inference. Yeah, I think people run for example inference there's a lot of diality to it during for like chat during the daytime people wake up and they ask a bunch of questions at night even some people still sleep and so at that time a lot of people were using it spot for inference like post training a lot of people were doing on spot instances at night so it's a general purpose chip with 8T is mostly for training. Some people are considering using it for inference. And 8i is primarily for inference, although people with smaller models also use it for training.

So you see that in the work we're doing with Gemini as much as in the silicon. So if you look at Gemini, we've seen sort of three phases if you will with the models. The first phase was where people were asking the model a set of questions and it was answering and you may iterate on it in a multi-turn but it was primarily kind of a search chatbot like experience. So our Gemini Enterprise product does provide the ability to do search and answer questions. It also added deep research to do deep analysis.

Then the second phase came along where people used to use diffusion models primarily to create content like images, audio, video and then with 2.5 Nano Banana we added media in was always true but media out became part of the main model. And so we saw people from creative, for example, WPP, a variety of CPG firms using Gemini Enterprise, our enterprise AI platform to create content. And there's all kinds of content creation now going on with it.

And then the models became really good at dealing with the abstractions of the world. And when I say abstraction of the world, if you go to a company, the model has to be hooked into a variety of different systems. That is to talk to your CRM system to ask answer questions about customers. It may have to look at your supply chain and planning systems. And as the models became really good at dealing with those and the ultimate abstraction is abstracting the rest of the world as a computer because if you can talk to a computer the computer can talk to everything because all these forms of software are just abstractions.

That's then led to this notion of agent. An agent is a module, let's call it, that you can delegate tasks to. The agent describes itself as a set of skills and it knows how to operate a set of tools, and then it can operate those, including a computer, and do tasks on your behalf. Now for us, that allows people, whether it's Xfinity using us to schedule and manage all their customer care, Walmart using us for a variety of things in their organization from planning to scheduling, Bosch in manufacturing using us, Merck has talked about how they're using us for research and patient care, from drug discovery out to delivering to patients, the whole cycle being automated. And that's the next phase of evolution. And so part of it is we're sort of co-designing as the skills of the model advance, we're able to broaden the set of things that can be done.

So if you look back, when you asked search a question, the first phase, there were a lot more input tokens than output tokens when you ask the model a set of questions because you would ask it a very complicated, long-described question, it would say this is the answer. Then when you came to content generation, you would give it a simple prompt, create a video that shows my dog wearing a superman cape and driving a car, and then it would take a while to generate the output tokens. So that generated a very different mix of tokens, of the type of tokens. Multimodality was one big thing, and then the volume of output tokens grew. Then you come along to agents. It informed chip design in three or four different ways. It informed us on how long you need to maintain stuff in memory. So for example, what kind of KV cache would you want? Because now you're delegating something that could run for 6, 7, 12 hours. You don't want to be shuffling things in and out as tokens because they get expensive. So that's one example. Second example, you want this system to operate a computer. By the way, that computer is a traditional classical compute machine.

So when people asked us how did that inform your chip effort, we not only work with Intel, we all have our own ARM chips, and so we built that because we saw general purpose compute usage coming from these tools. When you run for inference an agent that does many, many different steps, there are things about how you want to hold and pin objects in memory in the model so that the model runs things super efficiently because that can really optimize the cost of inference. There are many things we've done internally on how the chip can hold things in memory. And then because people wanted even a practical example, people want inference in many locations because they want to manage latency, unlike training where you can put it in a few big locations. So a practical example is TPU can be run in non-water-cooled mode so that you can put it in many more locations because air cooling is still the primary thing in most data centers. So there's a lot of thought that goes into these decisions. I'm just giving you three simple examples to illustrate.

Yes.

So, we look at the whole system all the time. A couple examples. We're announcing two new storage solutions next week. One is our managed Lustre solution. We've improved it to run 10 terabytes per second throughput. It is really designed for large-scale training. So you can cross-connect it to a giant cluster, and because you have large data sets, you can read them from the large-scale Lustre cluster now into a large training fleet for super-efficient scale. So that's one. Second thing we introduced is a new ultra-low latency inference storage system called Rapid Storage. And the idea is you can centrally keep the information you want to inference on in cloud storage, but you can mount it close to where, think of it as a forward proxy-like thing, wherever your inference chips are running. And so from your inference processor down to the storage system, Rapid Storage to fetch for inference, it's incredibly fast. It's 15 terabits per second. So you get ultra-low latency. You want to optimize all this stuff on a common network backbone. So we're introducing a new form of networking called Virgo which gives you ultra-low connectivity speed across a giant cluster. So there are many, many other parts to the stack we're also co-designing because of agents coming in, and the idea is to give people the most efficient cost structure to run agents with the best performance and quality.

The next big bottleneck will be largely around when consumers use virtual machines. You know, let's say I'm a consumer at home, I build an agent, and the agent is going to schedule travel for me, just hypothetically, if you're going on vacation, and you ask it to do a bunch of tasks like go look up eight travel sites which are exposed as tools, you know this common thing now people call MCPs or APIs, let's go find all the travel sites, let's say it's booking a trip to Europe or to Southeast Asia, run that calculation for me, the total cost, and tell me my budget. Consumers cannot afford to have VMs running forever. It's extremely expensive as you know. So people want to activate, deactivate VMs whenever a task gets done, and because these tools need local storage, these virtual machines can be oversubscribed, but you can also have local disk from which you read and write super efficiently, and so that's going to be a bottleneck because it's going to directly affect how widespread you can make this technology available because companies can pay for stuff obviously, the cheaper and more efficient they can use more stuff, but if you want to bring this to consumers, you know, for them it gets expensive very quickly. And if you want to reach everybody, you're going to have to engineer the cost structure of these things. And having again that ability to go across the layers from the agent down to Gemini down to the storage system and the compute systems, that allows us to co-design.

Yes.

Google's a platform company, you know, so when you're a platform company, different parts of your business compete with different players in the market. Some parts of your business may supply them and some part of the business may compete with them. And so we're determined to be best-in-class in the models and we're very proud of what we've done, not just with Gemini, the model, but also the whole tool chain that we're bringing around Gemini with our enterprise portfolio of tools. At the same time, you know, there are customers who want, for example, our TPUs and so Anthropic is an example of them. And it's just part of being a platform company. It's the same way that people ask us, how well do you optimize your model with Apple? Apple, for example, has signed a contract with us for the model, as you know. And so people go, isn't that competing with your Android platform and ecosystem? Yes. But that's part of being a platform company.

We have an executive team with Sundar and we discuss these and as any mature company we make those decisions. There's difficult calls every day. For instance, we have demand not just from Anthropic. So what percentage, even if you said there's X amount for Gemini and there's Y amount for the rest of the world, what amount do you give Anthropic versus the hundreds of other labs and other customers ask us for it? That's all complicated decisions that anybody has to make. What I'll tell you is this. It's better to have your own chips and demand than not having your own chips.

You'll see new stuff from us on Gemini with announcements coming both at Next and soon after. I think on the capability of the model, we're very proud of where Gemini is. I mean, it's been state-of-the-art for a long time. We have a new version of Gemini coming very, very soon and from all the benchmarks we've seen, we've been very confident on that as well.

We've had a capability to do disaggregated serving which allows us to scale very large dense models super well and that's been in place for a long time and so we're able, we would not design a model that we couldn't serve. And so we're very confident TPUs can serve the largest models in the world, and most importantly our serving stack we use for disaggregated serving by definition is the most efficient on TPU of all the model providers in the industry. So we're very confident we can serve the largest models, particularly the largest Gemini models.

We're not seeing that from the point of view of chip design or system design or lack of capacity or any of that.

We are seeing, I mean I'll give you two or three examples of things we are seeing. Historically, a lot of the data that was fed into models was unstructured data like text, audio, video, files, etc. Those continue to grow. But the reality of those is like there are many elements in an enterprise context which make them actually really simple to deal with. When you ask a question to an agent and you have the agent respond and tell you, tell me the citation or where did you derive this answer from, it's easy if it's in a document because you can just show a link to that document. Now, just imagine you ask the model a question, tell me how much inventory we'll need to meet demand for this product. That is going to translate to a query on a system like an SAP system or some kind of supply chain system that's dynamically going against a set of tables. So first being accurate on decomposing that query into which table is it getting it from and showing the response like where is the citation, like how do you get this, how do I know the answer that you gave me is correct is a much more complicated problem. And so because of the work we do in enterprise, we're able to feed Gemini a lot more cycles into our trajectory optimization harness with structured data, complex things like complex fields. Have you ever seen, when you talk about computer use and browser use, if you ever see an enterprise application with a thousand fields, drop-down lists, etc., there's no consumer app that would ever have that complexity. So being in this space also allows us to teach our Gemini systems some of those things and put it into the harness.

Today we have a lot of engineers using Jules, which is our internal coding harness, and that feedback has been going directly to DeepMind in the reinforcement loop and it's improving the quality of Gemini for coding every day and we have a lot of people in my organization using it.

So when we talk about software engineering productivity, we look at it slightly differently than is reported externally. So if you work in a company that builds products like Google does, the reality is that there are two or three examples of things that you find that are really important. Like a senior engineer writes much more compact code than a junior engineer. So we don't count how many lines of code as a measure because that's generally, a weaker engineer writes a lot more code to do the same task that a senior engineer does.

Yeah. So it's how much functions do we add that's important. The second thing, we've always had a tradition at Google that when you go to check in code you need peer review. Typically the peer review is done by senior managers, right, so they become the bottleneck. So we've introduced, and people are using Gemini, and we for example recently in Cloud introduced it to scan for security vulnerabilities in code. So it's not just that the tool is being used to generate code, we're also using it to inspect code and that helps us get, when the senior engineers come in for the review, a bunch of pre-work has been done. The third one is for the long term, in any real software company, the bulk of the time of the engineers where they find doing less productive work is debugging issues. So we built a version of Gemini and one of the things we're going to show next week is, you know what's the most complex computer in the world? The most complex computer in the world is a cloud. It makes a PC look like a toy. And so we've taken all of our cloud and exposed it as tools to the model. And so now we're using Gemini to troubleshoot incidents happening. And so that's also helped us improve the speed with which people can function and in turn improve the quality of the model itself. So there's a number of dimensions through which we look at the issue. But as productivity increases and you're shipping more features more quickly, I know lines of code is not the measurement, but it's certainly an output of this increased velocity.

That's a risk that we have to manage as an industry. People talk about I'm going to give you a prompt and the prompt's going to generate a block of code and you don't need to understand the code because you understand the prompt. In reality, for a complex system, the prompt will not explain all the potential behavior of the code. And so, for example, how do you deal with exceptions? And so that's something that I think every time you find this one area, like some time ago people said you won't need all these software engineers and then along comes the model and finds a lot of security vulnerabilities, and just when we need a ton of software engineers to work with models, like we're introducing a version of our model that can actually fix bugs, a fixed security vulnerability specifically, but you still need a human to use the tool and focus on it. Sometimes the industry over-rotates and so you say you don't need anybody just when you need it. And so we take a much longer-term view of things and so we're constantly looking at, for instance, do you need a supervisor model to look at code in a different way to actually review the code, and that's why when I said we still do peer review of the code and we're helping our senior engineers use the tool to do the reviews, and then the question comes will the tool be self-aware enough if it generated the code, will it find an issue with code that it generated because it's not self-aware of certain patterns. That's something we're looking at approaches to solve. And so our goal has always been to make sure we have the best model is to apply it at scale. And in my team alone, we have thousands of people using it every single day. I mean, if you walk just over there to the campus, you can see people like six different windows open. One in which they're coding, one in which they're compiling, one in which they're deploying and testing, and another one where they've got a background job running to run code review. I mean, there's a lot of people using the Jules tool harness and it's part of just evolving how work is getting done.

We're working through that and what would that line be. But you know our issue has been, so if Mythos finds a set of issues, what percent of those issues could be found with an open-source model? And the reason I mention open-source model is no matter how much you defend and you can say well I'll make sure closed-source models don't fall in adversaries' hands, open-source models for sure are going to be falling into adversaries' hands and they're just getting better. So sooner or later some part of this, it may not be all of the patterns, but some part of it can be detected. So what should you do in response? And we are unique because we're a hyperscaler, we're a model provider and we also have a cybersecurity organization, both our Mandiant team and Wiz. So we've done three practical things. If people are going to find issues using a model, you need to have a model help fix issues because they're going to find them way faster than humans can fix. So you need a model to help fix and so we're looking at something there. Second, if they're going to find issues with models, they're going to use the model and computer use to launch a large-scale attack. And so to defend that, using like I'll red team my system once a month is not going to be sufficient. So introducing agents that can do continuous red teaming and agents that can actually help fix, like for example it's one thing to fix the code. It's a second thing to find all the places the old code was running and remove it and then deploy the new code that's been patched and updated. Right? So that's the second piece. And the third piece is there's so much code out there. What do I start with? So again, that's another thing where we built tools to help people identify and prioritize what to fix.

No, we as Google use a ton of open-source and we contribute a ton of open-source. We're going to help the open-source community using our tools to actually go fix these things. I'm just pointing out the reality of where things are is that adversaries are going to use the model and the first place they're going to try and scan is popular open-source libraries because that gives them the maximum surface area to try and attack. And so those are all elements where we think it's important to go and address and fix and we're in process with the rest of the industry.

We're balancing so many things, making sure we have, to one part of your discussion, do we have right long-term plans for capital infrastructure for data centers, networks, enough of those lovely TPUs to go around? Second, are we constantly pushing the domain problems, the important problems? Three years ago when we said we should solve the problem of as AI gets better, cyber is going to be definitely an area that's affected. And when we made the offer to buy Wiz, people asked like why would you guys be doing that? When we look at our Gemini enterprise platform, just to give you an example, between January and now our token count has jumped from 10 billion a minute to 16 billion a minute. And the number of enterprise users of Gemini Enterprise has jumped by 40% sequentially. So, we're always looking at are we solving the right problems for customers and users and that's always the focus for us. And as long as we keep pushing aggressively on solving those problems and staying ahead of the market when the technology is evolving so quickly that when something happens, you got to have solutions before that occurs to the most part. And our teams have done an amazing job and we're super proud of what they've done and looking forward to the event.