Should we do like a show intro? I'm Ben Gilbert.
And we are your hosts. And we are going to be having a great, very interesting conversation today. If the conversation backstage is any indication, you guys are in for a real treat. For those who don't know, the Acquired podcast is a super long-form history of tech and other companies that David and I do. Last year, we did a three-part, 10-12 hour series on the entire history of Google from the origin of search to this AI moment that we are in now. And when Google called and said, well, do you want to interview Amin and Jeff on stage? It's kind of a no-brainer. So excited to do that with you here today.
Share it with you all. And with that, please welcome to the stage Google's SVP and Chief Technologist for AI and Infrastructure, Amin Vahdat, and Google's Chief Scientist and Technical Lead for Gemini, Jeff Dean.
Great seeing you. Great seeing you, Jeff.
Amin, congratulations on your new babies.
Beauty is in the eye of the beholder, but I have to tell you these are beautiful chips.
This one looks a little bigger than that one. Is that right?
TPU. This is our training powerhouse and this is Ironwood. It is a little physically bigger, but actually the way that it works out is there is actually more compute here. There's more SRAM here.
Yeah. So and then that's part of the reason why we have inference super low latency. We can hold things like the KV cache and other aspects all on the chip. Transfers between HBM and the SRAM is super fast. This one is built for throughput, raw horsepower at scale, goes up to 9,600 TPUs per pod. And then through our data center network and our software, actually if you heard it this morning, a million chips can work together thanks to Pathways and JAX. Jeff played a significant role in developing both of those. They really act as one computer. In other words, you can write a single piece of code and the compiler figures out and the runtime figures out how to distribute it across all up to a million TPUs.
Yeah, even a single Python process is driving that whole thing.
It's like the most googly thing I've ever heard.
Yeah, we got to ask. We thought to start, Jeff, we'd ask you when you joined this small little, you know, startup search engine that nobody thought was going to succeed in 1999. Is that right?
Employee 30.
Yep. We were all wedged above what's now a T-Mobile store in downtown Palo Alto.
It always is bizarre going by. It's like a cell phone shop now. Did you think that someday you would be designing cutting-edge silicon?
Well, I certainly didn't because I had no experience in doing so. Yeah, but I mean, I think Google has always been a company with pretty large ambitions in terms of what we're trying to set out to do. You know, I like our timeless mission of organizing the world's information. I think that's still what we're trying to do. And it has led to lots of interesting technical challenges that are still not solved over the years.
Well, speaking of that, we thought we would start today to go back a little over a decade to 2013, to the beginning of the TPU project and the coming technical challenge that you and Google saw that led to this crazy idea that you would develop your own chip. Could you take us back to that moment? How did the whole program start?
Yeah. So a little bit earlier than that even, in 2011 and 2012, we were starting to use very large numbers of CPU machines to train progressively larger and larger neural networks because we had seen that every time we scaled up the size of the model we were trying to train, trained it on more data, we had a saying, you know, bigger model, more data, better results. And that was true in numerous problems that we've tried in like the speech recognition domain, in lots of different kinds of vision tasks, and in language tasks and translation tasks in 2012, early 2013.
And this went on to kind of be colloquially known as scaling laws.
Yeah, so we never actually formally characterized that. We just had around the micro kitchen like, oh, we made the model twice as big and it's better kind of thing. But that really was something we were seeing in sort of progressively larger experiments over 2012 and early 2013. And so we had in speech in particular, we'd managed to get the word error rate to drop as much from training a larger model in the last few months as the previous 20 years of speech recognition research had improved things. And so at that time the existing speech systems just still didn't work quite well enough for people to really want to use them because every fifth word or something would be an error. So it's kind of frustrating to dictate something and then go back and fix every fifth word.
And we're like five years into the mobile era at this point.
Yeah. So in particular the phones didn't have enough horsepower to run the speech recognition on device. So you would end up sending to the cloud and doing recognition there. And so but if we have the error rate, like clearly people are going to start using it more. So I just did a thought experiment, as normally you'd say back of the envelope, back of a slide deck calculation, where I said, okay, suppose 100 million people start talking to their phone for three minutes a day. How much compute would we need for that for the best quality model we'd like to launch? And it turns out we would need to have a little bit more than twice as many computers as Google currently had to launch this new speech recognition feature. That seemed like a lot.
For one feature. Yeah, it's like the entire infrastructure.
For better speech recognition, we would need the entire Google infrastructure again. And so I kind of scratched my head. I'm like, that doesn't seem really that good. But it turns out that neural network inference has this incredible property that it's very tolerant of reduced precision. And it's a collection of linear algebra primitives, matrix multiplies, vector operations kind of strung together in different ways for different kinds of models. And so if you can build a chip that is essentially an incredibly good low-precision linear algebra machine, which is what a TPU is, then you have a hope of having much better performance. And we published a paper in 2017 about the TPU v1 that originated from that back of the slide deck experiment showing that it was 30 to 80 times more efficient than CPUs and GPUs of the day and 15 to 30 times lower latency. And so that was the initial TPU story.
And so the logic was if we are going to do things like this speech-to-text and other things, we're going to need an ASIC, a custom piece of silicon that we make specifically to handle operations like that.
Yeah. So in the design process we kind of had a few different kinds of problems we were trying to tackle. We had speech recognition which had one particular class of model. We had various kinds of convolutional neural network models for vision tasks of various kinds. And then at the last minute, we started to say, oh, these LSTM things are interesting. Let's add in a little bit of extra support even though we've already started designing the chip to support those. And that was how we were able to support Google Translate and a bunch of other kind of LSTM-like uses.
LSTM was the precursor to transformers.
Yeah. Long short-term memory recurrent models.
Wow. So how, I mean we spoke with you a little bit about this last night but to talk about it again here, most companies at this point faced with this problem, even other large technology companies with lots of resources, would have gone to, I don't know, Intel or Broadcom or Nvidia and said, hey, make me a chip. That's not what Google did.
You know, I think I'll tell a story. It's an apocryphal story. It was before I got to Google. Jeff, you were there. Maybe you can verify this. It's actually been verified where early on we realized, it supports some of the systems that Jeff and others had built in the early days of Google MapReduce, Google File System, etc. We realized that we wanted a 10,000-port, one-gigabit-per-second, which was a lot back in 2003, 10,000-port, one-gigabit-per-second non-blocking Ethernet switch. That's what we wanted. So we actually then put out...
Trying to imagine 10,000 ports on one.
So maybe it'd be however the people wanted to build it. That's what we wanted. And back then it was like 100 megabits per second and you'd be lucky to get 32 ports or something like that. So we actually did an RFQ, a request for quotes, and we put it out there and all the networking companies, everyone laughed at us.
Every single one said, this is a joke. You can't do it. 10,000 ports. No way. This is what you want. And they came back to us with their 100-megabit-per-second thing, etc. So Google being Google said, we're going to just do it ourselves. Networking, how hard could it be?
So we built this first system and we actually built it as a PCI card because we thought network boxes didn't make sense. We know about servers. We're going to build a PCI card and stick it into the servers. The first one literally caught fire. Turned out building hardware was hard. It failed. But no, we didn't quit. We built a second one. And the second one actually went into production. I think at this point I can say it wasn't very good. People hated it actually. They used it but they wanted to avoid those servers. The third one was that non-blocking by then 10-gigabit-per-second, 10,000-port switch. It wasn't one box. It was hundreds of them arranged in a rearrangeably non-blocking topology called the Clos topology. And it took off all across Google. It changed Google.
Right. And I think it was lessons like that that we were willing to invest in something that seemed crazy, that all the conventional wisdom, like everybody who knew anything about networking said this is ridiculous. Why would you do this? We were willing to step up and say we're going to try. We were willing to fail the first time miserably with fire, right? Like fire, smoke, right? The second time it was an okay product but kept going and the third time, the fourth time, and Ironwood, which we announced today, the million-TPU network, it is a logical continuation of that third iteration.
And I had the privilege of working on the fourth version actually as my starter project at Google in 2010. And we were willing to, and so I didn't have to deal with the fire but I got the benefits of the fire. But I think it's somewhat similar because if you go back to 2003 and you said, what's the conventional wisdom for building accelerators? No, you don't build accelerators. You don't build ASICs for individual workloads because they're going to get outdated really quick. By the time you launch it, the world will have passed you by. What I think, so I think we probably could have gone and maybe we even did go to Nvidia, I can't remember, another, I don't want to name one company, and said, hey, would you be able to build something specific for this? And probably were told, okay, it's going to take this amount of time, etc. We put out the first version, now different than our networking, the first version worked.
V1. It worked and it worked well.
15 months I think from idea to deployment.
Yeah. Really, really amazing.
Oh, we also speculatively bought a bunch of them without knowing how we would use them.
Yeah. I accosted Patrick Pichette at the time who was our CFO and I said we should buy a lot of these and he's like, how many? I'm like, a lot.
Well, it's one of these things where it's like, look, if we have all these Google engineers we probably should give them something fun to play with and see what new products they come up with. And there were even before the, call it 2021-2022-ish real explosion of AI that we're in now, there was a lot of narrow applications of machine learning that was really shining in Google products.
Yep. I mean, I'll give you an example. Like this wasn't a Google product but the AlphaGo match against Lee Sedol used two racks of TPU v1 chips for inference.
I actually went and put a commemorative Go board on the side of that rack when I visited the data center.
And you know, that was a big factor in how we were able to do so much exploration of the move space by having lots and lots of parallel specialized... as Lee Sedol didn't realize.
You know, we didn't talk about this before, but I'm recalling something from our episode research. Is it true that TPU v1s fit in hard drive bays?
They were PCIe-based cards, so you just slide them into PCI slots. I always thought that was a very clever way of saying, okay, there's a lot of stuff we don't want to do. We want to focus just on this very narrow problem that we want to solve and can we just fit it into existing hardware.
Yeah. I mean, I think the fact that we were trying to fit models that would fit on a single card made it easier because you don't actually need that much bandwidth to go into the card or come out.
Yeah. Whereas if you're trying to fit much larger models now, they don't fit on a single chip even for inference and you need much more complicated networks like we have for the Ironwood or, you know, subsequent TPU generations.
And it was really this TPU v2 which came out a few years later that took 256 of these slightly beefed-up v1s and connected them in a network topology and the realization there is that we could use them for training these big models as opposed to just serving them.
Yeah. And again with reduced precision, like that was a big...
That was big. Yeah, for the less technical of us in the room, Jeff, I've heard you give explanations of this before and I'd love to hear it. When you say reduced precision, why is that good? Sounds bad to me.
Oh, because so the reason you want to do this is think about multiplying a, you know, a fraction with six decimal places times another number with six decimal places. That's super hard for you, right? Well, it's super hard for computers too. So if you can multiply numbers with one digit after the decimal place instead of six, you just can multiply way more numbers with simpler multiplier units and do more of them in parallel. So you essentially can put many more multipliers into the same amount of chip area and that gives you much more throughput, can reduce latency substantially, reduces the amount of data you need to move from memory and back out to memory when you compute results. So this is just, you know, the cat's meow for deep learning and computer architects. My way that I characterize this is if you think back maybe to high school physics or maybe college physics classes and when the professor or teacher was working the problems out on the board, they'd write all these approximations. Like this used to drive me nuts because I was all about precision, but the professor would say 3 pi, let's call that 10. No, it's 9.42, blah blah blah. No, it's 10. And then these numbers, the ups and downs would cancel out. And then he'd come up or she'd come up with a number, an answer that was remarkably close to the real one by just approximating everything through. So less precision is good when you can do it in your head and get almost the same answer with let's say one-hundredth the work.
And it seems like there would be a trade-off where that affects the downstream model training where, hey, let's make a good model that uses lots of precision. I always find it remarkable that the amazing frontier models that we have today are trained on the smallest possible precision hardware.
Yeah. I mean, I think one way to think about it is you have a bunch of different weights in the model, right? And if you reduce the precision of those weights, if you reduce it a little bit, it doesn't matter that much. If you reduce it a lot, then you'll lose some quality, but you can recover some of that quality by adding, say, 20% more weights even though you have fewer bits in the precision. So then, you know, that's a good trade-off because it makes your model a little bit bigger in terms of number of parameters but smaller in terms of number of bits you need to manipulate.
So this brings up a great point, one we covered extensively in our series on Google. You were really unique in that you have a leading-edge silicon accelerator operation, construction business, if you will.
You make TPUs. Yes, that's the right way to put it. And that is married with a frontier lab, all under the same roof.
When did that collaboration start getting really tight? I mean, obviously the frontier lab didn't quite exist yet in today's conception when TPU started, but today...
Well, I mean, I would say the early work on large-scale training of neural networks actually informed the v1. So in some sense, we've been doing this for quite a while.
From the beginning. And you know, Jeff ran Google Research which did much of the precursor work here and ongoing still as well for many years and his involvement with the TPU program has been deep, not just with v1. I mean, we talked about the slide deck which was remarkably, remarkable foresight, but I mean, Jeff has been deeply involved, v1, v2, v3, the architecture, the details, and bringing all the knowledge from the research and where things are going into every single generation of TPUs including the two that we see here today.
The YouTube cat paper was TPUs, right? Was that still TPU v1 or was that...
No, that was even pre-TPU. So that was in 2012. That was when we were trying to scale up large neural networks. And so we had some ideas around unsupervised learning objectives for images. So we said, well, let's just take 10 million random YouTube frames and see what the model learns from that. And so we trained a neural network on, let's see, 16,000 CPU cores using 2,000 different machines. And it turns out we trained a multi-layer model and you could look at after you trained it what response, what kinds of images did different neurons in that top layer get excited by. And it turns out since we trained it on 10 million random YouTube videos, one of the neurons got very excited when it saw a cat face because there's a lot of cats in YouTube and other ones would get excited by kind of like pieces of cars or things like that. And that was completely unsupervised. We never told the model this is a cat or this is a car or a person. But because we trained a model 50 times bigger than anyone else had ever tried, you know, the model had started to develop those kinds of representations purely unsupervised. And I think that was part of our bigger model, more data kind of revelation that we had internally.
I'm going to sidetrack because I think this is an interesting conversation. What has that research developed into today? Trying to stare at specific neurons and saying, you know, is this a cat neuron? Is this more abstract concepts?
We had a raging debate at Google in terms of whether we should be using these deep models for search. Jeff knows this well because a challenge we had with these models was many times, maybe most of the time, they outperform the sort of human-tuned algorithm, right? Like the algorithm that you could read and understand, oh, I understand what this algorithm is trying to do and here's the search results. The model would outperform it. But here's the problem. You couldn't explain why. Like why did this neuron become the cat neuron and this other neuron become... So if there's a bug, how do you debug it? I think that continues today. I mean, Jeff, you can probably say more about this.
Yeah. I mean, I think unsupervised learning has taken a little bit of a backseat to what I would call self-supervised learning now because what you do for training the most capable language models these days is you just take lots of text and then you hide parts of it for the model and you make it guess what's missing. And typically you make it guess, you know, you get to see a prefix of the document like the first part of a sentence and then you have to guess the next word. That's literally the entire training objective for the language models that are like having all these amazing capabilities today. There's other variants where you can take a piece of text and you can hide like 10% of the words and then the model gets to look both left and right when trying to fill in the blank. That's a bit more like Mad Libs if you might have played that as a kid. But that's not super great for conversational use because you don't get to see the subsequent part of the conversation until you've actually had it. But that has also some good uses for like in representing document summaries and that kind of thing.
All right, I want to zoom in on a concept that David is bringing up. You've got the TPU team over here. You got Google DeepMind here. Can you tell us about the interactions? Like if I were to parachute into some meetings, how does it look? Who's bringing what to the table? What are each team sort of asking for? How much flexibility is there?
A day in your life basically.
Yeah. I mean, there's a lot of flexibility and I think the beautiful thing here is, and the goal is the same, build the best models right at possible, the most efficient models possible. So I think there's just a lot of give and take. So for example, the research team might have seven different directions that look really, really promising and they actually have more than seven. Hardware team has seven different directions that look really, really promising. Now, how do you mesh those two? And then there's a bunch of give and take because what they'll settle on is, well, what if you change one of your seven ideas in this direction for your model and what if I change one of my seven ideas on the hardware? Like we can't do exactly what you want. That usually happens, like the research team will come to the hardware team and say, we want X.
And the hardware team says that's impossible. Now, if you're at separate companies, you just stop. Okay, bye. We'll maybe talk to another hardware person. You said no to our RFQ.
We'd like a 10,000-port switch.
Yeah, exactly. Goodbye. On the flip side, and many other cases, the hardware team would come to the research team and say, we've got this awesome new widget. What would you like to do with it? And the research team would say, nothing. That sounds useless. I'm sure that your hardware widget is cool, but we don't need it. So it's that give and take that says, well, what if I took your hardware widget and I could do something different with my model? So that spirit of give and take is there really every single day and this exploration of the joint space, it's not my space, your space, it's the joint space and how do we build a system that can deliver the models, serving, training, agents, everything that can be as efficient as possible. That happens at all levels. Now there's a lot of people at Google, so a lot of this is sort of engineer to engineer, which is exactly how you want it.
And then some number of those ideas bubble up.
Yeah. I would add like it's super hard as a hardware designer in an extremely fast-moving field like AI and ML because you're basically, if you think about starting a chip today, it might sort of be done designing in two years and out in your data center and then take, you know, it needs to last three to six years or something. So you're trying to predict where is this incredibly fast-moving field going in the two to six to eight-year time horizon. And like six months ago we just completely changed like lots of aspects of ML with some new algorithm. So that's super hard. So it's really good to have the insights of, you know, researchers who are really trying ideas at small scale that seem like they might want to work. Like the best thing for a researcher is an idea that just seems to want to work because you know that you're going to be able to make it work and with more...
More effort, whereas ideas that kind of really don't want to work and you've tried everything, those are probably ones that are less exciting. But I think that give and take and co-design is really important. I'll give you an example from the TPU actually.
One of the things we wanted to do was very long traces for reinforcement learning, where you decode lots and lots of tokens and you generate, you know, some 500-line piece of code, and then you want to be able to run it and see if the code runs and passes the unit test, and they get a reward for reinforcement learning. Well, in order to decode those tokens, you have to do whatever the latency is for each token generation, you know, for 60,000 tokens a generation. And if that takes a long time, then your reinforcement learning will take a lot longer because it's not as low latency as it could be. And so the research team came to the TPU hardware team and said, "We really need a way to make much lower latency because it'll improve both online inference but also reinforcement learning."
And so then people kind of put their heads together and said, "Hey, we actually could do relatively small changes to the chip that's being designed, but change the topology of the network." And that would actually not be a super dramatic change, but it would really help the latency for decoding.
This is the butterfly topology.
The butterfly topology. Mhm.
So I mean, I think that's a really crisp, concrete example of the kind of co-design that happens.
For years, we'd built these TPUs for throughput, and the inference example that Jeff was talking about, we would manage the latency by basically pipelining many, many queries together. So we would do lots of queries, but for reinforcement learning, you can't pipeline many queries. This is the one where you're trying to generate this 500 lines of code, to Jeff's nice example, run it and then test it and get the results. You can't do the pipelining. So the throughput-optimized engine, which continues, because by the way, for other use cases you still need that massive throughput, wasn't the best fit for this latency, okay, let me get a single result out case. And the hardware team would have naturally been inclined to say, let me just make the last thing faster.
But it completely misses the use case of the coming research problem, and without having that tight conversation, you're going to miss that.
How do you... So the future is not known, the future is fuzzy, and every incremental day that you get out in the future gets less and less clear. You've got all these world-class researchers who are developing, you know, what's the next transformer? What's the next... and there's various, there's a huge gradient of confidence that you have in the next iteration.
How do you allocate bets when you only have a fixed amount of space in silicon, and then that thing needs to live for two to eight years?
This is the raging debate every single day.
I mean, that's the whole process of co-design.
Yeah. You got these probably like thousands of ideas and you can't put them all into silicon. They're all, as you said, it's a probability distribution function looking out two or three years. And you got to pick the four per generation or whatever. You can't—maybe it's five, but it's not a thousand. You get four or five things that you get to put in that's big and risky and bold. Okay. So now, how do you project out into the future? You write simulators, actually. So in other words, it's not that we're completely saying, "Okay, I like this guy. I don't like this person, so I'm going to go read them." Actually, runoffs. Yeah. And having high-fidelity simulators is really, really important. You have to make predictions about the workload. Many, many variables. The simulators aren't... in the end they'll winnow out probably 90% of the thousand.
But now you've still got 50.
Right? And then it's a judgment call of like, how important do we think this thing will be three years from now versus this other thing five years from now or two years from now. And I'm sure you're looking for clever ways of saying like, can we kind of accomplish both with some unique... I don't know, multipliers or something on here.
