CEOInterviews.AIJeff Dean & Noam Shazeer – 25 years at Google: from PageRank to AGI

I

Interviewer0:46

Today I have the honor of chatting with Jeff Dean and Noam Shazeer. Jeff is Google's Chief Scientist, and through his 25 years at the company, he has worked on basically the most important things for the current AI revolution. He has been the inventor or co-inventor of all the main architectures and techniques that are used for modern LLMs: from the Transformer itself, to Mixture of Experts, to Mesh Tensorflow, to many other things. And they are two of the three co-leads of Gemini at Google DeepMind. Awesome. Thanks so much for coming on.

J

Jeff Dean1:35

Thank you. Super excited to be here.

I

Interviewer1:39

Okay, first question. Both of you have been at Google for 25, or close to 25, years. At some point early on in the company, you probably understood how everything worked. When did that stop being the case? Do you feel like you still understand everything?

N

Noam Shazeer2:11

It was not the case that everyone at Google knew everything. It was just the case that Jeff knew everything because he had basically written everything.

J

Jeff Dean2:18

You're very kind. I think as companies grow, you kind of go through these phases. When I joined, we were 25 people, 26 people, something like that. So you eventually learned everyone's name, and even though we were growing, you kept track of all the people who were joining. At some point, you lose track of everyone's name in the company, but you still know everyone working on software engineering things. Then you lose track of all the names of people in the software engineering group, but you at least know all the different projects that everyone's working on. Then at some point, the company gets big enough that you get an email that Project Platypus is launching on Friday, and you're like, 'What the heck is Project Platypus?'

It's good to know what's going on in the company, even at a very high level, even if you don't know every last detail. And it's good to know lots of people throughout the company so that you can go ask someone for more details or figure out who to talk to. With one level of indirection, you can usually find the right person in the company if you have a good network of people that you've built up over time.

I

Interviewer3:29

How did Google recruit you, by the way?

J

Jeff Dean3:35

I kind of reached out to them, actually.

I

Interviewer3:35

And Noam, how did you get recruited?

N

Noam Shazeer3:44

I actually saw Google at a job fair in 1999, and I assumed that it was already this huge company, that there was no point in joining, because everyone I knew used Google. I guess that was because I was a grad student at Berkeley at the time. I guess I've dropped out of grad programs a few times. I saw this search engine, and figured I should apply to multiple places for a job. But then it turned out to be really fun, it looked like a bunch of smart people doing good stuff. They had this really nice crayon chart on the wall of the daily number of search queries that somebody had just been maintaining. It looked very exponential. I thought, 'These guys are going to be very successful, and it looks like they have a lot of good problems to work on.' So I was like, 'Okay, maybe I'll go work there for a little while and then have enough money to just go work on AI for as long as I want after that.'

I

Interviewer4:53

Yeah, yeah. In a way you did that, right?

N

Noam Shazeer4:53

Yeah, it totally worked out. A friend of mine told me that his New Year's resolution for 2000 was to live to see the year 3000, and that he was going to achieve this by inventing AI. I was like, 'Oh, that sounds like a good idea.' I didn't get the idea at the time that you could go do it at a big company. But I figured, 'Hey, a bunch of people seem to be making a ton of money at startups. Maybe I'll just make some money, and then I'll have enough to live on and just work on AI research for a long time.' But yeah, it actually turned out that Google was a terrific place to work on AI.

J

Jeff Dean5:51

One of the things I like about Google is our ambition has always been sort of something like that. And also you could see that what we were doing initially was in that direction, but you could do so much more in that direction.

I

Interviewer6:17

How has Moore's Law over the last two or three decades changed the kinds of considerations you have to take on board when you design new systems, when you figure out what projects are feasible? What are still the limitations? What are things you can now do that you obviously couldn't do before?

J

Jeff Dean6:34

I think of it as actually changing quite a bit in the last couple of decades. Two decades ago to one decade ago, it was awesome because you just wait, and like 18 months later, you get much faster hardware, and you don't have to do anything. And then more recently, I feel like the general-purpose CPU-based machine scaling has not been as good, like the fabrication process improvements have slowed down. But I think at the same time, we're seeing much more specialized computational devices, like machine learning accelerators, TPUs, and very ML-focused GPUs, more recently, are making it so that we can actually get really high performance and good efficiency out of the more modern kinds of computations we want to run that are different than a twisty pile of C++ code trying to run Microsoft Office or something.

N

Noam Shazeer7:43

It feels like the algorithms are following the hardware. Basically, what's happened is that at this point, arithmetic is very, very cheap, but the bottleneck is memory bandwidth. So you want algorithms that are N cubed operations and N squared bytes of data communication basically.

J

Jeff Dean8:22

Well, I would say that the pivot to hardware oriented around that was an important transition, because before that, we had CPUs and GPUs that were not especially well-suited for deep learning. And then we started to build TPUs at Google that were really just reduced-precision linear algebra machines, and then once you have that then you want to exploit it.

N

Noam Shazeer8:52

It seems like it's all about identifying opportunity costs. Like, okay, this is something like Larry Page, I think, used to always say: what's the biggest thing that you have that you're missing out on? In this case, I guess it was that you've got all of this chip area, and you're putting a very small number of arithmetic units on it. Fill the thing up with arithmetic units! You could have orders of magnitude more arithmetic getting done. Now, what else has to change? Okay, the algorithms and the data flow and everything else. And, oh, by the way, the arithmetic can be really low precision, so then you can squeeze even more multiplier units in.

I

Interviewer9:38

Noam, I want to follow up on what you said, that the algorithms have been following the hardware. If you imagine a counterfactual world where, suppose that the cost of memory had declined more than arithmetic, or just invert the dynamic you saw. Okay, data flow is extremely cheap, but arithmetic is expensive. What would that world look like?

N

Noam Shazeer10:17

Sure. I guess I joined Google Brain in 2012. I left Google for a few years, happened to go back for lunch to visit my wife, and we happened to sit down next to Jeff and the early Google Brain team. I thought, 'Wow, that's a smart group of people.' I think I said, 'You should think about deep neural nets. We're making some pretty good progress there.' 'That sounds fun.' Okay, so I jumped back in…

J

Jeff Dean10:42

I wooed him back, it was great.

N

Noam Shazeer10:47

…to join Jeff, that was like 2012. I seem to join Google every 12 years: I rejoined Google in 2000, 2012, and 2024. What's going to happen in 2036?

J

Jeff Dean11:13

I think one general trend is we're getting better at quantizing or having much more reduced precision models. We started with TPUv1, and we weren't even quite sure we could quantize and model for serving with eight-bit integers. But we sort of had some early evidence that seemed like it might be possible. So we're like, 'Great, let's build the whole chip around that.' And then over time, I think you've seen people able to use much lower precision for training as well. But also the inference precision has gone. People are now using INT4 or FP4, which sounded like, if you said to someone like we're going to use FP4, like a supercomputing floating point person 20 years ago, they'd be like, 'What? That's crazy. We like 64 bits in our floats.'

N

Noam Shazeer12:10

Yeah, just a 0-1. And then you have a sign bit for a group of bits or something.

J

Jeff Dean12:16

It really has to be a co-design thing because, if the algorithm designer doesn't realize that you can get greatly improved performance, throughput, with the lower precision, of course, the algorithm designer is going to say, 'Of course, I don't want low precision. That introduces risk.' And then it adds irritation. Then if you ask the chip designer, 'Okay, what do you want to build?' And then they'll ask the person who's writing the algorithms today, who's going to say, 'No, I don't like quantization. It's irritating.' So you actually have to have someone who can say, 'Look, it's going to be a little irritating, but your model is going to be three times faster, so you're going to have to deal.'

I

Interviewer13:18

Through your careers, at various times, you've worked on things that have an uncanny resemblance to what we're actually using now for generative AI. In 1990, Jeff, your senior thesis was about backpropagation. And in 2007 — this is the thing that I didn't realize until I was prepping for this episode — in 2007 you guys trained a two trillion token N-gram model for language modeling. Just walk me through when you were developing that model. Was this kind of thing in your head? What did you think you guys were doing at the time?

J

Jeff Dean14:12

I thought it would be fun to do something around neural nets. So, he and I decided I would implement a couple of different ways of parallelizing backpropagation training for neural nets in 1990. I called them something funny in my thesis, like 'pattern partitioning' or something. But really, I implemented a model parallelism and data parallelism on a 32-processor Hypercube machine. In one, you split all the examples into different batches, and every CPU has a copy of the model. In the other one, you pipeline a bunch of examples along to processors that have different parts of the model. I compared and contrasted them, and it was interesting. But it turned out we needed about a million times more compute before they really started to work for real problems, but then starting in the late 2008, 2009, 2010 timeframe, we started to have enough compute, thanks to Moore's law, to actually make neural nets work for real things. That was kind of when I re-entered, looking at neural nets. But prior to that, in 2007...

I

Interviewer15:36

Sorry, actually could I ask about this?

N

Noam Shazeer15:41

Oh yeah, sure. First of all, unlike other artifacts of academia, it's actually like four pages, and you can just read it. It was four pages and then 30 pages of C code. But it's just a well-produced artifact.

J

Jeff Dean16:15

Tell me about the N-gram model. We were working on machine translation. There was a DARPA contest for translating a couple of different languages to English, I think, Chinese to English and Arabic to English. The Google team had submitted an entry, and the way this works is you get 500 sentences on Monday, and you have to submit the answer on Friday. I saw the results of this, and we'd won the contest by a pretty substantial margin measured in Bleu score, which is a measure of translation quality. So I reached out to Franz, the head of this winning team. I'm like, 'This is great, when are we going to launch it?' And he's like, 'Oh, well, we can't launch this. It's not really high throughput, obviously. It was doing 100,000 disk seeks in a large language model that they sort of computed statistics over — I wouldn't say 'trained' really — for each word that it wanted to translate. Obviously, doing 100,000 disk seeks is not super speedy. But I said, 'Okay, well, let's dive into this.' So I spent about two or three months with them, designing an in-memory compressed representation of N-gram data. We were using — an N-gram is basically statistics for how often every N-word sequence occurs in a large corpus, so you basically have, in this case, we had 2 trillion words. Most N-gram models of the time were like, 'the' occurs 17 times in the web, or something. And so I built a data structure that would let you store all those in memory on 200 machines and then have sort of a batched API where you could say, 'Here are the 100,000 things I need to look up in this round for this word,' and we'd give you them all back in parallel. That enabled us to go from taking a night to translate a sentence to basically doing something in 100 milliseconds or something.

I

Interviewer18:47

There's this list of Jeff Dean facts, like Chuck Norris facts. For example, that 'for Jeff Dean, the speed of light is actually a lower bound.' And this is a perfect example: you had a system that took 12 hours to translate a sentence, and Jeff Dean decided to optimize it over a weekend. Just going from 12 hours to 100 milliseconds, I got to do the orders of magnitude there.

J

Jeff Dean19:11

All of these are very flattering. They're pretty funny. They're like an April Fool's joke gone awry by my colleagues.

I

Interviewer19:31

Obviously, in retrospect, this idea that you can develop a latent representation of the entire internet through just considering the relationships between words is like: yeah, this is large language models. This is Gemini. At the time, was it just a translation idea, or did you see that as being the beginning of a different kind of paradigm?

J

Jeff Dean19:50

We definitely saw it as language models. Noam has worked on a number of other things at Google, like spelling correction systems that use language models. That was like 2000, 2001, and I think it was all in-memory on one machine.

N

Noam Shazeer20:24

Yeah, I think it was one machine.

J

Jeff Dean20:29

His spelling correction system he built in 2001 was amazing. He sent out this demo link to the whole company. I just tried every butchered spelling of every few-word query I could get, like 'scrumbled uggs Bundict' — I remember that one, yeah yeah — instead of 'scrambled eggs benedict', and it just nailed it every time.

N

Noam Shazeer20:48

Yeah, I guess that was language modeling.

I

Interviewer20:52

But at the time, when you were developing these systems, did you have this sense of, 'look, you make these things more and more sophisticated, and eventually they will sweep the world and be artificial intelligence'?

N

Noam Shazeer21:20

I think at the time, a lot of people were excited about Bayesian networks. That seemed exciting. Definitely seeing those early neural language models, both the magic in that, 'okay, this is doing something extremely cool' and also, it just struck me as the best problem in the world in that for one, it is very, very simple to state: give me a probability distribution over the next word. Also, there's roughly infinite training data out there. There's this amazing ability to just learn from observations of the world. And then it's AI complete. If you can do a great job of that, then you can pretty much do anything.

I

Interviewer23:44

There's this interesting discussion in the history of science about whether ideas are just in the air and there's a sort of inevitability to big ideas, or whether they're sort of plucked out of the ether by specific individuals. How do you think about that in the context of the Transformer?

N

Noam Shazeer24:17

I think in some sense, it was in the air, and in some sense, you need some group to go do it. I like to think of a lot of ideas as being partially in the air, where there are a few different, maybe separate research ideas that one is squinting at when you're trying to solve a new problem. You draw on those for some inspiration, and then there's some aspect that is not solved, and you need to figure out how to solve that.

J

Jeff Dean25:11

When you're looking at a research area, you come up with this idea, and you have this feeling of, 'Holy shit, I can't believe that worked?' One thing I remember was in the early days of the Brain team. We were focused on 'let's see if we could build some infrastructure that lets us train really, really big neural nets'. At that time, we didn't have GPUs in our data centers; we just had CPUs. But we know how to make lots of CPUs work together. So we built a system that enabled us to train pretty large neural nets through both model and data parallelism. We had a system for unsupervised learning on 10 million randomly selected YouTube thumbnails. We got that working and training on 2,000 computers using 16,000 cores. After a little while, that model was actually able to build a representation at the highest level where one neuron would get excited by images of cats. It had never been told what a cat was, but it had seen enough examples of them in the training data of head-on facial views of cats that that neuron would turn on for that and not for much else. Similarly, you'd have other ones for human faces and backs of pedestrians, and this kind of thing. That was kind of cool because it's from unsupervised learning principles, building up these really high-level representations. Then we scaled that up to a much larger model. That neural net was probably 50x bigger than one that had been trained previously, and it got good results. So that sort of said to me, 'Hey, actually scaling up neural nets seems like, I thought it would be a good idea and it seems to be, so we should keep pushing on that.'

I

Interviewer27:29

These examples illustrate how these AI systems fit into what you were just mentioning: that Google is fundamentally a company that organizes information. AI, in this context, is finding relationships between information, between concepts, to help get ideas to you faster, information you want to you faster. Now we're moving with current AI models. Obviously, you can use BERT in Google Search and you can ask these questions. So how are you thinking about that? Is Google still an information retrieval company if you're building an AGI? An AGI can do information retrieval, but it can do many other things as well.

J

Jeff Dean28:30

I think we're an 'organize the world's information' company, and that's broader than information retrieval. Maybe: 'organizing and creating new information from some guidance you give it'. 'Can you help me write a letter to my veterinarian about my dog? It's got these symptoms,' and it'll draft that. Or, 'Can you feed in this video, and can you produce a summary of what's happening in the video every few minutes?' I think our multimodal capabilities are showing that we can deal with many different kinds of information: like weird lidar sensors on autonomous vehicles, or genomic information, or health information. And then, how do you extract and transform that into useful insights for people and make use of that in helping them do all kinds of things they want to do? Sometimes it's, 'I want to be entertained by chatting with a chatbot.' Sometimes it's, 'I want answers to this really complicated question, there is no single source to retrieve from.' You need to pull information from 100 web pages, figure out what's going on, and make an organized, synthesized version of that data. Then dealing with multimodal things or coding-related problems. I think it's super exciting what these models are capable of, and they're improving fast, so we're going to see a lot more.

N

Noam Shazeer30:14

A billion dollars is not cool anymore. What's cool is a trillion dollars. But a trillion dollars is not cool anymore. What's cool is a quadrillion dollars. Obviously the idea is not to just pile up some giant pile of money, but it's to create value in the world, and so much more value can be created when these systems can actually go and do something for you, write your code, or figure out problems that you wouldn't have been able to figure out yourself. To do that at scale, we're going to have to be very, very flexible and dynamic as we improve the capabilities of these models.

J

Jeff Dean31:12

But I also think there's value in seeing what we could achieve for end-users and then how can we work backwards from that to actually build systems that are able to do that. As one example: organizing information, that should mean any information in the world should be usable by anyone, regardless of what language they speak. And that I think we've done some amount of, but it's not nearly the full vision of, 'No matter what language you speak, out of thousands of languages, we can make any piece of content available to you and make it usable by you. Any video could be watched in any language.' I think that would be pretty awesome. We're not quite there yet, but that's definitely things we're working on.

I

Interviewer32:13

And then obviously language models have limited context right now, but they can really think. It's like dark magic, in-context learning. It can really think about what it's seeing. How do you think about what it would be like to merge something like Google Search and something like in-context learning?

J

Jeff Dean32:33

Yeah, I'll take a first stab at it because I've thought about this for a bit. One of the things you see with these models is they're quite good, but they do hallucinate and have factuality issues sometimes. Part of that is you've trained on, say, tens of trillions of tokens, and you've stirred all that together in your tens or hundreds of billions of parameters. But it's all a bit squishy because you've compressed it. Whereas information in the context window, in the input of the model, is really sharp and clear because we have this really nice attention mechanism in transformers. The model can pay attention to things, and it knows the exact text or the exact frames of the video or audio or whatever that it's processing. Right now, we have models that can deal with millions of tokens of context, which is quite a lot. It's hundreds of pages of PDF, or 50 research papers, or hours of video, or tens of hours of audio, or some combination of those things, which is pretty cool. But it would be really nice if the model could attend to trillions of tokens. Imagine if you could have all of the world's information in context, and the model could make use of that, with my permission, to help solve what it is I'm wanting it to do. But that's going to be a big computational challenge because the naive attention algorithm is quadratic. You can barely make it work on a fair bit of hardware for millions of tokens, but there's no hope of making that just naively go to trillions of tokens. So, we need a whole bunch of interesting algorithmic approximations to what you would really want: a way for the model to attend conceptually to lots and lots more tokens, trillions of tokens. Maybe we can put all of the Google code base in context for every Google developer, which would be pretty cool.

N

Noam Shazeer35:11

Models are quite memory-efficient at memorizing facts. You can probably memorize on the order of one fact or something per model parameter. Whereas if you have some token in context, there are lots of keys and values at every layer. It could be a kilobyte, a megabyte of memory per token. You take a word and you blow it up to 10 kilobytes or something. Yes. There's actually a lot of innovation going on around, okay, A, how do you minimize that? And B, what words do you need to have there? Are there better ways of accessing bits of that information?

I

Interviewer36:12

I want to talk more about the thing you mentioned about: look, Google is a company with lots of code and lots of examples. If you just think about that one use case and what that implies, so you've got the Google monorepo. Maybe you figure out the long context thing, you can put the whole thing in context, or you fine-tune on it. Why hasn't this been already done? You can imagine the amount of code that Google has proprietary access to, even if you're just using it internally to make your developers more efficient and productive.

J

Jeff Dean36:49

To be clear, we have actually already done further training on a Gemini model on our code base, and it's incredibly useful. Sundar, I think, has said that 25% of the characters that we're checking into our code base these days are generated by our AI-based coding models with kind of human oversight.

I

Interviewer37:28

How do you imagine, in the next year or two, based on the capabilities you see around the horizon, your own personal work? What will it be like to be a researcher at Google? You have a new idea or something. With the way in which you're interacting with these models in a year, what does that look like?

J

Jeff Dean37:43

Well, I assume we will have these models a lot better and hopefully be able to be much, much more productive. If you have an idea for a new system, you can say, 'I'd like to build a system that does X, Y, and Z,' and the model can generate a pretty reasonable first cut at that. From a research perspective, maybe you can say, 'I'd really like you to explore this kind of idea similar to the one in this paper, but maybe let's try making it convolutional or something.' If you could do that and have the system automatically generate a bunch of experimental code, and maybe you look at it and you're like, 'Yeah, that looks good, run that.' That seems like a nice dream direction to go in. It seems plausible in the next year or two years that you might make a lot of progress on that.

N

Noam Shazeer38:48

It seems under-hyped because you could have literally millions of extra employees, and you can immediately check their output, the employees can check each other's output, and you can have a lot of parallel work going on.

I

Interviewer39:14

I do want to play with this idea more because it seems like a big deal if you have something kind of like an autonomous software engineer, especially from the perspective of a researcher who's like, 'I want to build the system.' Okay, so let's just play with this idea. As somebody who has worked on developing transformative systems through your careers, the idea that instead of having to code something like whatever today's equivalent of MapReduce is or Tensorflow is, just like, 'Here's how I want a distributed AI library to look. Write it up for me.' Do you imagine you could be 10x more productive? 100x more productive?

J

Jeff Dean39:53

I was pretty impressed. I think it was on Reddit that I saw we have a new experimental model that someone gave a prompt like, 'Build me a SQL database from scratch, no external dependencies, and please do that in C.' From what the person said, it actually did a quite good job. It generated a SQL parser and a tokenizer and a query planning system and some storage format for the data on disk and actually was able to handle simple queries. From that prompt, which is like a paragraph of text or something, to get even an initial cut at that seems like a big boost in productivity for software developers. I think you might end up with other kinds of systems that maybe don't try to do that in a single shot, but have a lot of background processes that are exploring different ideas. You'll need ways of managing the workflow if you have a lot of these background activities happening.

I

Interviewer41:23

Can you talk more about that? What interface do you imagine we might need if you could literally have millions of employees you could spin up, hundreds of thousands of employees you could spin up on command, who are able to type incredibly fast, and who — it's almost like you go from 1930s trading of tickets or something to now modern Jane Street or something. You need some interface to keep track of all this that's going on, for the AIs to integrate into this big monorepo and leverage their own strengths, for humans to keep track of what's happening.

N

Noam Shazeer42:11

I think of it as a major issue. We have lots and lots of really, really brilliant machine learning researchers, and we want them to all work together and build AI. So actually, the parallelization among people might be similar to parallelization among machines. I think definitely it should be good for things that require a lot of exploration, like, 'Come up with the next breakthrough.' If you have a brilliant idea that is just certain to work in the ML domain, you can just try it. But if you have a lot of ideas, you can try them all.

I

Interviewer43:17

Suppose in the world today there are on the order of 10,000 AI researchers in this community coming up with a breakthrough. Probably more than that. There were 15,000 at NeurIPS last week. Wow. 100,000, I don't know. Yeah, maybe. Sorry. No, no, it's good to have the correct order of magnitude. The odds that this community every year comes up with a breakthrough on the scale of a Transformer is, let's say, 10%. Now suppose this community is a thousand times bigger, and it is, in some sense, like this sort of parallel search of better architectures, better techniques. How do you think about that?

J

Jeff Dean44:16

It's a good question, because I don't know that folks haven't been doing that as much. We definitely have lots of great ideas coming along. Everyone seems to want to run their experiment at maximum scale, but I think that's a human problem. It's very helpful to have a 1/1000th scale problem and then vet 100,000 ideas on that, and then scale up the ones that seem promising. So, one thing the world might not be taking advantage of enough is doing more small-scale experiments to filter ideas before scaling them up.

I

Interviewer46:11

There's a trend where Gemini 3 is coming up with all these different architectural ideas, trying them out, and you see what works, and you're constantly coming up with algorithmic progress that makes training the next one easier and easier. How far could you take that feedback loop?

J

Jeff Dean46:26

I think one thing people should be aware of is that the improvements from generation to generation of these models often are partially driven by hardware and larger scale, but equally and perhaps even more so driven by major algorithmic improvements and major changes in the model architecture, the training data mix, and so on, that really makes the model better per flop that is applied to the model, so I think that's a good realization. Then I think if we have automated exploration of ideas, we'll be able to vet a lot more ideas. Currently, we have a lot of brilliant machine learning researchers: looking at lots of ideas, winnowing ones that seem to work well at small scale, seeing if they work well at medium scale, bringing them into larger scale experiments, and then settling on adding a whole bunch of new and interesting things to the final model recipe. If we can do that 100 times faster through those machine learning researchers just gently steering a more automated search process, rather than hand-babysitting lots of experiments themselves, that's going to be really, really good. The one thing that doesn't speed up is experiments at the largest scale. You still end up doing these huge training runs that take months.

I

Interviewer48:19

There's this algorithmic side improvement that future AI can make. There's also the stuff you're working on. I'll let you describe it. But if you get into a situation where just from a software level, you can be making better and better chips in a matter of weeks and months, and better AIs can presumably do that better, how does this feedback loop not just end up in, Gemini 3 taking two years, then Gemini 4 is — or the equivalent level jump is now six months, and then three months, and then one month?

J

Jeff Dean49:12

I think if we could dramatically speed up the chip design process. As we were talking earlier, the current way in which you design a chip takes you roughly 18 months to go from 'we should build a chip' to something that you then hand over to TSMC and then TSMC takes four months to fab it, and then you get it back and you put it in your data centers. So that's a pretty lengthy cycle, and the fab time in there is a pretty small portion of it today. But if you could make that the dominant portion, so that instead of taking 12 to 18 months to design the chip with 150 people, you could do it in a month with a lot of AI assistance, then I think you could get perhaps much more exploration and more rapid design of something that you actually want to give to a fab. That would be great because you can shrink fab time, you can shrink the deployment time by designing the hardware in the right way, so that you just get the chips back and you just plug them into some system. And that will then enable a lot more specialization, it will enable a shorter timeframe for the hardware design so that you don't have to look out quite as far into what kind of ML algorithms would be interesting. Instead, it's like you're looking at six to nine months from now, what should it be? Rather than two, two and a half years. That would be pretty cool.

N

Noam Shazeer51:15

I do think that the fab time is three to five months. Okay, but that's how long training runs take anyways, right? So you could potentially do both at the same time. Potentially. Okay, so I guess you can't get sooner than three to five months. But the idea that you could get — but also, yeah, you're rapidly developing new algorithmic ideas. That can move fast. That can move fast, that can run on existing chips and explore lots of cool ideas.

I

Interviewer51:33

So, isn't that a situation in which you're... I think people sort of expect like, ah, there's going to be a sigmoid. Again, this is not a sure thing. But just like, is this a possibility? The idea that you have sort of an explosion of capabilities very rapidly towards the tail end of human intelligence that gets smarter and smarter at a more and more rapid rate?

J

Jeff Dean52:11

I think we're not there yet. The models can break down a problem internally in the model into a bunch of steps, can sort of puzzle together the solutions for those steps, and can often give you a solution to the entire problem that you're asking. But it isn't super reliable, and it's good at breaking things down into five to ten steps, not 100 to 1,000 steps. So if you could go from, yeah, 80% of the time it can give you a perfect answer to something that's ten steps long to something that 90% of the time can give you a perfect answer to something that's 100 to 1,000 steps of sub-problem long, that would be an amazing improvement in the capability of these models. We're not there yet, but I think that's what we're aspirationally trying to get to. We don't need new hardware for that, but we need better algorithms.

N

Noam Shazeer53:16

I think there's a lot of room for applying more compute at inference time. I guess the way I like to describe it is that even a giant language model, even if you're doing a trillion operations per token, which is more than most people are doing these days, operations cost something like 10 to the negative 18 dollars. And so you're getting a million tokens to the dollar. I mean compare that to a relatively cheap pastime: you go out and buy a paper book and read it, you're paying 10,000 tokens per dollar. So we're already 100x cheaper than reading a paperback. We're 10,000 times cheaper than talking to a customer support agent, or a million times or more cheaper than hiring a software engineer or talking to your doctor or lawyer. Can we add computation and make it smarter? I think a lot of the takeoff that we're going to see in the very near future is of this form. We've been exploiting and improving pre-training a lot in the past, and post-training, and those things will continue to improve. But taking more inference time compute, having the model think about a problem for a while, generate a bunch of different potential solutions. Maybe it does some searches on its own, gets some information back, consumes that information, and figures out, oh, now I would really like to know more about this thing. So now it iteratively explores how to best solve the high-level problem you pose to this system. And I think having a dial where you can make the model give you better answers with more inference time compute seems like we have a bunch of techniques now that can kind of do that. The more you crank up the dial, the more it costs you in terms of compute, but the better the answers get. That seems like a nice trade-off to have, because sometimes you want to think really hard because it's a super important problem.

I

Interviewer55:51

Is there a fundamental limit to taking inference time, like having some way in which you can just linearly scale up inference time compute? Or is this basically a problem that's sort of solved, and we know how to throw 100x compute, 1000x compute, and get correspondingly better results?

J

Jeff Dean56:33

We're working out the algorithms as we speak. So I believe we'll see better and better solutions to this as these many more than 10,000 researchers are hacking at it, many of them at Google. I think we do see some examples in our own experimental work of things where if you apply 10x more compute at inference time, you get much better quality of the answers than we're getting today. And so that's about designing new algorithms, trying new approaches, figuring out how best to spend that 10x instead of x to improve things.

I

Interviewer57:27

Does it look more like search, or does it look more like just keeping going in the linear direction for a longer time?

J

Jeff Dean57:37

I really like Rich Sutton's paper that he wrote about the Bitter Lesson and the Bitter Lesson effectively is this nice one-page paper but the essence of it is you can try lots of approaches, but the two techniques that are incredibly effective are learning and search. You can apply and scale those algorithmically. So I think we'll see a combination of both: the model will try a few different ways of solving this problem, and that one didn't work, but this one worked better. I'm going to explore that a bit more.

I

Interviewer58:22

How does this change your plans for future data center planning and so forth? Where can this kind of search be done asynchronously? Does it have to be online, offline? How does that change how big of a campus you need and those kinds of considerations?

J

Jeff Dean58:35

One general trend is it's clear that inference is going to be a much larger fraction of total compute than training over time. So we need to design data centers that are good at both training and inference, and we need to be able to dynamically allocate resources between the two.

N

Noam Shazeer58:43

In terms of inference, if you have a model that's already trained and you want to do inference on it, it is going to become a growing and important class of computation. Maybe you want to really crank up the amount of compute you use at inference time, and even more specialized solutions will make a lot of sense.

I

Interviewer59:21

Does that mean you can accommodate more asynchronous training? Training? Or inference? Or just that different data centers don't need to talk to each other, you can just have them do a bunch of...

N

Noam Shazeer59:37

I like to think of it based on whether the inference you're doing is latency-sensitive—like a user is actively waiting for it—or a background thing. Maybe I have some inference tasks I'm trying to run over a whole batch of data, but it's not for a particular user; I just want to extract some information. There are probably a bunch of things we don't have very much of right now, so we periodically make sure that all the things you think are good are good together. If not, understand why they're not playing nicely.

I

Interviewer1:10:26

Two questions. One: how often does it end up being the case that things don't stack up well together? Is it rare or does it happen all the time?

N

Noam Shazeer1:10:36

It happens 50% of the time. But I think most things you don't even try to stack because the initial experiment didn't work that well, or it showed results that weren't promising relative to the baseline. Then you take those things and try to scale them up individually. If they seem really promising, you include them, but complexity—both codebase-wise and algorithmically—hurts. Complexity makes things slower and introduces more risk. At the same time, you want it to be as good as possible, and every individual researcher wants their inventions to go in. There are challenges, but we've been working together quite well.

I

Interviewer1:12:40

Okay, going back to the dynamic where you find better algorithmic improvements and the models get better over time, even if you take hardware out of it: should the world be thinking more about this? And should you guys be thinking more about this?

N

Noam Shazeer1:13:13

There's another world where you have this big feedback loop, meaning the two years between Gemini 4 and Gemini 5 are the most important years in human history. Because you go from a pretty good ML researcher to superhuman intelligence due to this feedback loop. To the extent that you think that second world is plausible, how does that change your approach to greater levels of intelligence?

J

Jeff Dean1:13:48

I've stopped cleaning my garage because I'm waiting for the robots. So I'm more in the second camp—we're going to see a lot of acceleration.

N

Noam Shazeer1:13:52

I think it's important to understand what's happening and the trends. For example, breaking down a simple task into 10 sub-pieces and doing it 80% of the time versus something that can break a high-level task into 100 or 1,000 pieces and get it right 90% of the time is a major step up in capabilities. It's important for people to understand the progress, and those models will be applied across different domains. I'm super excited about ensuring we, as a society, get the maximal benefits from these models. Google has a good view on how to approach this; our Responsible AI principles provide a nice framework for trade-offs in making better AI systems available while ensuring safety.

I

Interviewer1:16:17

If the thing that catapults this intelligence explosion is misaligned—not trying to write the code you think it's writing, but optimizing for some other objective—and on the other end of this rapid process, you have things approaching Jeff Dean or Noam Shazeer level or beyond, with millions of copies of that level... How should we think about that?

J

Jeff Dean1:17:18

Some people think these systems will be amazing and we don't have to worry at all; others think we'll be overwhelmed. I'm somewhere in the middle. I co-authored a paper called 'Shaping AI,' where we argue that instead of a laissez-faire approach, we should actively shape and steer AI deployment to be maximally beneficial in areas like education.

N

Noam Shazeer1:18:13

If a computer takes over and has unlimited control, that's an engineering problem: how to engineer safe systems. It's similar to rigorous development in older software, like airplane software, which has a good safety record despite risky tasks. The difficulty is that AI has a feedback loop—you put a model in a box with compute, and it comes out as version 1000.

J

Jeff Dean1:19:19

Actually, figuring out what's problematic or dangerous will be the solution to many control issues. We're definitely working on this at Google, with brilliant folks focused on it. It's becoming increasingly important from both a societal benefit and a business standpoint.

I

Interviewer1:20:15

I know you take the potential benefits and costs seriously. It's remarkable how many applications you've put out to improve different areas. But if there's a feedback loop and you end up with a model as good as Noam Shazeer or Jeff Dean, and there's an evil version running around—a million of them—that could be much worse than any other risk.

N

Noam Shazeer1:21:13

Suppose we have Gemini 3 or 4 helping us train future versions, writing training code. From this point, we oversee it and verify. Even the verifiers that check the models' outputs will eventually be trained or written by AIs. What do we want to know for sure before letting Gemini 4 help with AI research? We need to run tests before letting it write AI code. I think having the system explore algorithmic research ideas with a human in charge is sensible. You can put safeguards in place to get benefits of self-improvement with human oversight without going fully autonomous.

I

Interviewer1:23:20

If you expose the model's capabilities through an API or user interface, you have control to understand its use and set boundaries. That's one tool to ensure acceptable behavior.

N

Noam Shazeer1:23:52

The goal is to empower people, but for the most part we should let people do things with these systems. However, we shouldn't empower people to hurt others with a million evil software engineers. So I'm against that.

J

Jeff Dean1:24:22

Me too. All right, let's talk about lighter topics. Over the last 25 years, what was the most fun time?

N

Noam Shazeer1:24:34

I think the early four or five years at Google, when I was one of a handful working on search, crawling, and indexing. Traffic was growing fast, we expanded our index, and updated it every minute instead of every month. That was exciting. But equally exciting is working with the Gemini team today. The progress over the last year and a half is really fun. Models keep getting better at complex tasks—if you showed someone from 20 years ago what these models can do, they wouldn't believe it. Even five years ago they might not. That's satisfying, and I think we'll see similar growth in usage and impact.

J

Jeff Dean1:25:49

I'm with you. Early days were super fun—knowing everyone, building something helping millions to billions of people. What could be better? What was this micro kitchen?

N

Noam Shazeer1:26:25

We have a micro kitchen area in the building we both sit in, called Gradient Canopy. It used to be Charleston East, but we needed a more exciting name since it's full of ML researchers. The micro kitchen normally has an espresso machine and snacks, but this one has a lot of space, so we set up maybe 50 desks there. People hang out, it's a bit noisy because there's constant feedback—like, 'We're launching this thing next week, how's the load test?' We also have Gemini chat rooms for those not in the micro kitchen. There are probably 120 chat rooms I'm in related to Gemini. For a very focused topic, we have seven people, and exciting results come from London colleagues. When you wake up, you see what's happening—it's just fun.

I

Interviewer1:27:51

What I find remarkable about some of the calls you guys have made... Today you estimate that these models will be a backbone of our services, with constant inference and training future versions. Thinking about compute needed by 2030, where does the Fermi estimate get you?

N

Noam Shazeer1:28:31

You're going to want a lot of inference compute. If one technique for improving quality is scaling up inference compute, then a request generating tokens could become 50 to 1,000 times more intensive. Currently, maybe 10-20% of computer users have discovered chat-based interfaces. As that approaches 100% and usage increases, you get another order or two of magnitude. Models will likely be bigger, adding another order. So you need extremely efficient inference hardware. In terms of global inference flops in 2030, more is always better.

I

Interviewer1:30:14

What will people spend on AI at that point? What do AI systems look like?

N

Noam Shazeer1:30:28

Maybe it's a personal assistant in your glasses that sees everything and accesses all digital information. Like Joe Biden with an earpiece advising him in real time. If the thing gets 5-10 IQ points smarter, would you rather spend $10 a day for an assistant or $20 for a smarter one? And it makes you from a 10x engineer to a 100x or 10 millionx engineer. From first principles, people will spend some fraction of world GDP on this. We'll need lots of energy, millions to billions of robots building data centers. The sun puts out about 10^26 watts. I'm guessing the compute for AI per person will be astronomical.

J

Jeff Dean1:32:47

I would add to that. I'm not sure I agree completely, but it's an interesting thought experiment. Even if you get partway there, making access as low cost as possible is achievable through hardware and model co-design, making things much more efficient.

I

Interviewer1:33:33

Is Google's data center build-out plan aggressive enough given this demand increase?

N

Noam Shazeer1:33:46

I'm not going to comment on future capital spending. But you can look at our past capital expenditures over the last few years—we're definitely investing in more chips and people for training and inference.

I

Interviewer1:34:19

You've talked a lot about continual learning—having a model improve over time without starting from scratch. Is there any fundamental impediment? What does that future look like?

N

Noam Shazeer1:34:38

I've been thinking about this more. I'm a big fan of sparse models because different parts can excel at different things. Our Gemini 1.5 Pro uses mixture-of-experts, which gives a more capable model that's efficient at inference time—large capacity but only activating a small part. However, the current structure is very regular; experts are the same size and merge back quickly. They don't branch off for math and then stay separate from cat images. I think we should have a more organic structure where pieces of the model can be individually trained and shared. But there are trade-offs—including more multilingual data might come at the expense of coding data. Ideally, we'd have a small set of people specializing in, say, Southeast Asian languages, train a modular piece, and hook it up to a larger model. That would decompose the problem and allow continual learning: you could glue models together, rip out pieces, or upgrade one piece without discarding everything. There's a countervailing interest in science—we still learn from complete training runs to understand what to build. But I think there are ways to get benefits through versioning and modularity. You could have a frozen base model and include a variant of a specific module to compare performance. That could lead to faster research progress and be cheaper and more parallelizable.

J

Jeff Dean1:39:26

This idea casually laid out would be a big regime shift. Instead of a blob where everything is pipelined, you could make surgical incisions to improve a specific part—or grow the model by adding a little more. I've been sketching this vision for a while in Pathways.

I

Interviewer1:39:53

Yeah, you've been building the...

N

Noam Shazeer1:40:14

Maybe we haven't written a paper outlining all its capabilities. But there have been times, like the TPU pods setup, which was brilliant. The low-level software and hardware stack—with great torus interconnects and the right collectives—came from supercomputing. It might be a good candidate for a paper.

I

Interviewer1:41:13

So distillation extracts the knowledge from a trained model and distills it into a better architecture, and that's how it improves over time?

N

Noam Shazeer1:41:23

Distillation is useful because it transforms a model into a different form. Often you take a large, unwieldy model and distill it into a smaller one for fast inference. But you can also do this at the module level: each module could have multiple representations—a really big one and a much smaller one. You train the small one to mimic the big one, then repeat the process. If you have that running in a thousand places in your modular model, it could work well. It could also enable inference scaling—the router decides how much compute to use. For an easy math problem, you route to a tiny distilled module; for a hard one, you use the big one.

I

Interviewer1:42:40

It's often hard to decode what each expert is doing in mixture-of-experts models. How would you enforce modularity that is visible and understandable?

N

Noam Shazeer1:42:50

In the past, I found experts to be quite interpretable. We had a thousand or two thousand experts, and one was getting words referring to cylindrical objects, another was good at dates and times. It's pretty easy to do. Not that you need that human understanding at runtime—you just have a learned router. But there's work on interpretability, like Chris Olah's at Anthropic, showing you can find neurons that activate for the Golden Gate Bridge. You could do that at the expert level and get interpretable results. But it's unclear if you need it—if the model is just good, we don't care what every neuron does, as long as the overall system is good. That's the beauty of deep learning.

I

Interviewer1:44:48

There are so many interesting implications. One is that if you have a model with hundreds of billions of parameters, and you need the whole thing in memory, Google's TPU pods would be immensely valuable. There's a misconception with Mixture of Experts that you don't need to go through unused weights. But if you didn't retrieve that memory, you'd run at batch size one, which is inefficient. You look at all experts but send only a fraction of the batch through each.

N

Noam Shazeer1:46:52

Right, but you still have a smaller batch at each expert. To get balance, current models typically have all experts the same size and just route tokens to some. In a more dynamic future, you might have modules that are factors of 100 or 1000 different in size, with paths that go through many layers or just one. You'll want very large batches but push things through asynchronously at inference time. Pathways was designed for that—components of variable cost, and for each example you choose a subset of the model.

I

Interviewer1:48:13

Currently, anyone can train a small enough model. But if this is the best way to train future models, you'd need a company with a data center serving a single model blob. That would be a paradigm shift.

N

Noam Shazeer1:48:36

You definitely want enough HBM to hold your whole model. You might have multiple copies for load balancing. If one expert is used a lot for math questions and another rarely for Tahitian dance, you could even page the rare one out to DRAM. But you want the system to figure this out based on load.

I

Interviewer1:49:36

Language models take language input and output language. The Pathways blog post discusses many non-autoregressive use cases. We're starting to see Gemini used across Google without fine-tuning—just instructions per use case. There's more sharing of underlying model capabilities across services.

N

Noam Shazeer1:50:38

People might not realize how interesting this prediction is—it's like getting Noam on a podcast in 2018 saying 'language models will be a thing.' You could have different models for different settings: an internal one for Google with modules on internal data, and other modules for other companies served via cloud APIs. The bottleneck is systems engineering and ML; it's a different operating mode than Gemini development. We'll explore these areas, but we need evidence it's better. If that happens, it would enable faster progress in distinct areas. The data control modularity is also cool: a personal module trained on my data, or modules trained on YouTube data only for YouTube products. So you don't need a separate search index or YouTube model—it's just an incrementally growing blob.

J

Jeff Dean1:53:20

This notion of an organic, not carefully mathematically constructed ML model has been with me a while. While artificial neurons inspired by biology served us well, we're not looking at other things brains do. For example, brains have specialized areas good at different things. We have a little of that in mixture-of-experts, but it's still very structured. I'd like to see organic growth of expertise—add capacity where needed. Also, adapting connectivity to hardware: you want dense connections between nearby chips, fewer connections as distance increases. Even across TPU pods, you send less but more salient information, and across metro areas, even less. Let that emerge organically rather than hand-specifying.

I

Interviewer1:56:13

If you have this blob deciding how much to activate, then to get more output you activate a larger pattern rather than more instances. It's not calling more agents—it's calling different sub-patterns based on compute you want to spend.

N

Noam Shazeer1:56:51

This sounds complicated to deploy because it's an evolving thing, but you can always distill from it. If you care about a specific task, you distill from the organic thing into something served efficiently. You could do that daily or hourly.

J

Jeff Dean1:57:50

Yeah, we need better distillation. Anyone who invents amazing techniques to instantly distill from a giant blob onto your phone would be wonderful.

N

Noam Shazeer1:58:13

We should maximize value from every token. When the model gets to 'the answer is,' it should do more work than when it gets to 'the.' There must be ways to get more from the same data—learn it forwards and backwards, hide parts, make it infer from partial information. Vision models have been doing this for a while. You distort the model or hide parts to force it to work harder, getting more interesting observations. Image people didn't have enough labeled data, so they invented dropout, but we're not using it much for text. You could make 100 epochs over text data with dropout without overfitting—it's computationally expensive, but it means we won't run out of textual data.

I

Interviewer2:10:11

Between hardware and AI algorithms, there are a thousand more breakthroughs—just look at your Google Scholar pages. What's the trick to career longevity and breadth?

J

Jeff Dean2:10:43

I like to find out about new, interesting areas by paying attention, talking to colleagues, and reading research. Sometimes I work with clinicians on real healthcare problems—learning where AI can help. Working with five or six colleagues with different expertise allows us to do collectively what none could individually. Their expertise rubs off on you, expanding your tool belt. That's the beauty of learning on the job.

N

Noam Shazeer2:12:18

I try to say what I just did is nothing compared to what can be done, and to drop an idea as soon as you see something better. When you see a better idea, you realize your current approach may not work. There's a drive to say 'what I'm doing is not working, let me try something else.' Google Brain did well with bottom-up chip allocation—everyone had one credit you could pool. Gemini has been mostly top-down, which led to more collaboration but less incentive to pivot. Going forward, we'll have both to incentivize collaboration and flexibility. Both lead to innovation.

J

Jeff Dean2:14:42

It's also good to articulate interesting directions. I have an internal slide deck called 'Go, Jeff, Wacky Ideas.' People get excited about those and want to work on them—a good way to bootstrap direction without ordering people 'we must go here.'

I

Interviewer2:15:16

Alright, this was great. Thank you, guys.

N

Noam Shazeer2:15:27

Appreciate you taking the time. It was great chatting.

J

Jeff Dean2:15:27

That was awesome.

Jeff Dean & Noam Shazeer – 25 years at Google: from PageRank to AGI

About Noam Shazeer