Even before we started the Gemini effort, there were a lot of people thinking about building incredibly general-purpose models that could do things. Oriol was leading some efforts in DeepMind and I was sort of helping steer some efforts around the Pathways project and things like PaLM and PaLM 2 and so on. I actually said this is silly. We are fragmenting our efforts and fragmenting our compute, and if we're going to build an incredibly powerful model, we need to all come together and work on building a single model. That's actually where the name Gemini comes from — the twins.
We mapped and then we reduced.
Yeah. Yeah. Yeah. Exactly. Yeah, something like that.
I thought it was because I had twins.
Hey everyone, how's it going? My name is Logan Kilpatrick. I'm on the Google DeepMind team. Today we're talking with Jeff, Demis, Noam, and Oriol about all things Gemini, the origin of the Gemini project, and so much more. So, thank you all for sitting down and chatting. We're sitting here in Gradient Canopy. We just launched the Gemini 3.5 era of models starting with Flash. Contextualize the moment. I think this is like the third and a half generation of Gemini models. We've shipped a lot of stuff, a lot of models in between. Maybe Oriol, you want to sort of fill us in on the moment with Gemini 3.5?
Yeah, we could do one each maybe. But contextualizing, I guess we started 2023. I want to say we've had a few releases, some half models or even point one. And been building on some foundation of multimodal, tool use, agentic from the get-go and just kept building the capabilities up. So today it's exciting to be releasing the Flash version of 3.5. It's a very powerful series and the focus of this one probably is on coding and then of course preserving and enhancing the rest of capabilities.
I think everyone feels like this is really the times where coding capabilities and agentic experiences are defining what it means to experience AI, and 3.5 is a huge, huge step there.
Right. And I think everyone is actually experiencing that and it is being recognized as a very strong model. These big release moments have in one way gotten less exciting because the thing that's most at the top of everybody's mind is not even the big release to the public. It's like, what am I going to be using tomorrow to do my engineering and my research? And what are my friends around the office going to be using for their engineering and research? And will they be complaining at me? Will they be happy? It's always fun and exciting on a day-to-day basis.
Reflecting back on the initial moment of coming together and forming the Gemini project and shipping those initial models — was it obvious to you all that the product story of how we bring Gemini models to the world was going to be so important? Not from, obviously at Google we have lots of products and we bring AI to our customers through the products, but actually to improve the model — was that like a 'we hope that this happens and it's something we're going to intentionally work towards,' or has it just become obvious over time that we need to do that because the use cases are much more complicated than the initial version of Gemini?
I'm actually curious what you all think.
I mean, I can weigh in. I think that was actually pretty obvious — that if you have a model that is used by a lot of people, that you're going to get a lot of lessons and experience of what's working well, what's not working well. And we've seen this in search for many, many years — the use of search by our users really informs what are the things that's not doing well, what are the things we should be doing better. Aggregating lots of interesting usage statistics to understand that in a deeper way and then working on improving those things is important, and AI models should be no different.
Right, so it seemed like that was pretty obvious from the beginning, but we had to have a thing out there that people were using.
Yeah, that's the true test — people using it and is it useful to people. Because if you go in the box and you try to hill-climb benchmarks, then you end up hill-climbing benchmarks and maybe you leak your benchmark. It doesn't end up well.
You don't want to build intelligence in a black box. You want it to be useful. You want people to be using it — therefore understanding what is required, what is scratching the frontier is both scratching the frontier of the research in terms of technical capability but also scratching the surface of what is the next thing that you can enable users. You can't do it if you don't actually do it with the products, and those two hand in hand define what the frontier means.
So at the time Gemini starts, there's already a lot of machine learning models making it into products, and I think what seemed obvious is that if we create a single model more powerful than the average of the other models powering everything, that had to be a leap forward. Whether there was a single product that could be created around a single model — that maybe wasn't as clear at the time — but I think it was very clear that putting all the compute and intelligence into a single powerful model was going to leapfrog many things Google was using already machine learning for. And that was very exciting — to be given that amount of compute and responsibility initially. But I think it's proven to be indeed kind of the core engine of Google intelligence.
Even before we started the Gemini effort, there were a lot of people thinking about building incredibly general-purpose models that could do things. Oriol was leading some efforts in DeepMind and I was sort of helping steer some efforts around the Pathways project and things like PaLM and PaLM 2 and so on. I actually said this is silly. We are fragmenting our efforts and fragmenting our compute, and if we're going to build an incredibly powerful model, we need to all come together and work on building a single model. That's actually where the name Gemini comes from — the twins.
We mapped and then we reduced.
Yeah. Yeah. Yeah. Exactly. Yeah, something like that.
I thought it was because I had twins.
Jeff, that's a great segue to sort of going back to the formation of the Gemini project. I'm curious how controversial was that? Obviously as you say it now and we've done three and a half iterations, the organizational complexity of bringing teams together is now behind us. Was it blatantly obvious at the time that we won't win and actually deliver on building the right product and models for our customers if we don't do this? Or was it more of a pie-in-the-sky idea? I'm just curious what was your level of confidence?
I was sure the right thing was to come together. I actually articulated in a half-page memo like, 'This is silly to fragment.' We shared this half-page — we should put it somewhere, release it somewhere. But it felt like fragmenting our best ideas across different research teams that weren't really working together and also fragmenting our compute — both of those issues seem like things we should fix. And it was a little bit organizationally complicated and time-zone-wise — there were lots of people in London, lots of people here, eight hours apart is never a recipe for easy collaboration. But I think we've done a really good job of navigating that and bringing people together. And now we have a really good amazing team all over the world and we're cranking out good models.
There were a bunch of teams building LLMs at the time that you just needed to mash together, basically.
At some point, research in AI was a lot more academic — you go back ten years, a lot more academic research. And at that point, how you organize it is not really the key element; it is more about the exploration and the speed of exploration is important. But as things get more and more focused, what you want is to really, to Jeff's point, this focused operation where rather than us trying to build things in parallel — well, because these things require a lot more focus, effort, and each one of them is like a major operation in terms of many researchers coming together solving many problems. At that point I think it was really, really a good idea to say, okay, this is the moment that we need to change. I think both organizations acted with great urgency on that and enabled it. I think that was an experience — of course it's never easy bringing two organizations together — but I think everyone realized that this is the right moment and there's a huge value to be gained from this. And I think all of us can see the whole organization is very proud of what we have built together. Gemini is really the fruit of that.
It's the scale. It's the fact that when you build one big, beautiful, giant LLM, it can do so many things. And so you really do need to put together that many people, that much compute, and infrastructure teams, data teams, and so on.
Yeah. Better to have one of those teams than five little ones.
Yeah. I mean, to have one model than five. One thing I would say is, from the start we wanted Gemini to be — I mean, even pre-Gemini, one of the origins of the Pathways project was to explore a single model that could do many things, a multimodal model that could deal with all different modalities, a very large model that was sparse so you would activate different pieces of it for different kinds of things. And all three of those things are sort of represented in the Gemini models that we have today. And I think now with Omni we've gotten the multimodal aspects — now we can even generate video. We used to be able to just generate images and audio. Pretty awesome, because you have the full capability of this amazing reasoning model that can deal with lots of input modalities and can edit the video it's just produced.
I think Omni is a whole new capability, right? I mean, we had Veo and Imagen, of course — you can do text-to-video, text-to-images. But what you want is really a model that understands all the modalities of the physical world so that it can understand the physics and everything, but together with text, because there's a lot of information about the world in text as well at a very high level.
Cory, really quick, I have a question about this. In the IO keynote, we were sort of framing Omni in the world model section, and I'm curious — does it actually have a bunch of the Genie world model stuff, or is this sort of just positioning for the next stage where it takes in anything and puts out anything and that's sort of our representation of world model? That wasn't abundantly clear to me.
I hadn't thought about it that way. I'll give my opinion — Oriol has worked on these things a lot. World modeling means that you really understand the dynamics, the physics, the visuals, and then you have to be able to simulate that as well, because that simulation aspect is critical for both us understanding if the model has it right and also when you want to rely on the model. You want a model that is going to be able to roll forward that simulation and the decisions that are coming out of the model are based on those future simulations. That's why I think Gemini only is a different category that is really transforming what we had with Gemini, which is mostly understanding and text output, and Veo, which is text input and doing the video modeling, into turning to a really true world model by training jointly. There's a hope that of course everything will transfer and making a better text understanding model will help the world modeling aspect. But I think we're seeing this every time we try — it's not easy, but as we get the recipes right, we see that back in the day, rolling out a complex video scene forward, consistency, all these things — you kind of had to manually think about them and almost pre-specify how to get the visuals right over time, and when you turn the thing, the object disappeared. And just by training at scale and mixing all the data more and more, we're seeing these capabilities emerge. And that's what's exciting and sort of the main premise I guess we were putting forward. And now finally we're going to be outputting also amazing consistent 3D worlds, sounds, all the things. It feels almost impossible if you asked me a few years ago this approach would work — otherwise probably we would have done it ten years ago. But it did happen.
When you say multimodal, you instinctively are drawn to human modalities like text and images and audio and video. But I think really you want the model to understand a much richer set of modalities — like understand interesting scientific data that comes in genomic sequences, or in chemical structure, or robotic grasping data, or LiDAR data. Exposing the model to a little bit of this kind of data makes it much better at understanding it when it does encounter more of it.
I feel like part of the story of Google DeepMind being able to pull off this model and actually the formation story is people and the fact that you all actually know each other. We were talking off-camera before this about when did you all meet each other and start working together and hear of each other. I'm curious for all your versions of that story.
Maybe I can go first since I think I've known people the longest. So, for many years I did a lot of engineering hiring and recruiting in the very early days of Google. So I screened all the engineering resumes that came into Google for three years or something.
It was amazing. They would just bring this like a giant stack of resumes. He'd be like, 'No, yes, yes, no, no, no, yes.' Like extremely fast.
So I didn't actually interview Noam, I don't think, but he had interviewed and gotten an offer. And I think you were debating should you take the offer. So I called you up on the phone in 2000 and I said, 'Hey, you know, let me just chat. I want to introduce myself and I really like the kinds of things you're excited about and working on. I think you'd really enjoy it here.' And I finished the phone call.
Honest question. Were you just selling at that point or was there something — like he had an offer?
So I'm selling. I want him to accept the offer.
He did. He did. And then Noam became my office mate for like three and a half years or something. Oh yeah, I mean, I remember joining and everyone got a mentor to ask questions as a new hire because there are like a million things you don't know. And I would ask my mentor and every time the mentor would know the answer. And it was like, 'Wow, everyone knows everything here.' And it turns out that Jeff was my mentor and it was just that Jeff knows everything and had written like half the codebase.
Yeah. So then fast forward I guess maybe to 2012, I think. So Oriol had interviewed with us and I don't think I interviewed you, but I think you had an offer and I was trying to convince you — you were considering this and another company. So I called you up and I said, 'Hey, you should really come here. We're really doing really interesting deep learning, choice deep learning models, and we're having an awesome time.' We were all in the Google Brain team wedged into like probably a 30-person office just outside on the main patio of the main four buildings of the Googleplex. Somehow I managed to convince you to come, which was awesome.
Yeah, there was — I mean, I remember lots of back and forth. I had like the last maybe one year of my PhD. So I was just writing the thesis — no LLM at the time. So you actually had to write every single word. Lots of pondering. But I joined and maybe not exactly like a mentor like Noam mentioned, but we started two projects, one of which was distillation. So I remember — the codebase was complex, like C++, you're in academia so you don't know exactly how to implement things, but the idea was clear. And literally I remember sitting by Jeff's desk and he was just coding the classes — like, okay, distillation and KL divergence and so on and so forth. So we didn't have coding agents at the time, but I can say maybe for a little bit Jeff was kind of acting as the coding agent for the project.
I mean, a hard benchmark to beat still today. That project was good because Jeff Hinton did some of the very early exploration on MNIST, which is like a tiny, tiny data set that he could run on his laptop, and he had some good ideas about how to get a bigger model to transfer into a smaller model. I'm like, we got to show this thing at scale. So we trained a 50-model ensemble for 300 million images, which was a lot at that time, and 50 distinct ones. So we grouped the categories — this one was going to be good at cars and this was going to be good at wild animals. Then we transferred the knowledge with distillation into a single model and it was much more accurate than the single model you could have trained on the raw data. And by the way, at the time I remember compute was already constraining. But all you needed to do is ask Jeff, 'Hey, we ran out of CPU,' and he would just go to some website, change a number, and we doubled it. And we did that a few times.
Yeah. I had superuser power thing.
That was not — I missed that.
Yeah. Sadly, exponential growth sometimes stops happening.
I remember the first time we really sat together and talked was actually during the acquisition discussions of DeepMind that you flew to London. There was this moment where there were all sorts of discussions and such, there's a bunch of people in the room, but then Jeff comes to me and says, 'Let's look at the code.' I'm like, 'Okay.' So I sat down at the keyboard, I'm like, 'Okay, don't show me anything too sensitive, but I want to see that directory.' So he pokes at that directory and we go inside. I'm like, 'Okay, let's see this file.' He's like, 'Okay.' And then I go and explain, 'Okay, here's what we are doing here. Here's what we are doing there. This is this idea. This is that idea.' I mean, at the time for me it was a big deal, right? Like I'm sitting together with Jeff, I'm explaining to him these are the ideas and this is the code and we are walking through it.
Our first code review together. Looks good to me.
Was it actually, Jeff? You were just like pointing at random directories and then Oriol just happened to know what was happening.
We'd seen 15 talks, which was great. At the time I would review pretty much all the code at DeepMind. So I would know pretty much everything going on.
Yeah. And I think the company was like 55 or 60 people or something. So we all flew over to London and then we didn't sleep super well that night and then we go in and we see 13 consecutive 30-minute talks.
Yes. Jeff Hinton had a bad back, so he's laying on the floor in the back of the conference room. We just flattened him and then —
Yeah. Towards the end of the day, I'm like, 'Okay, this seems pretty promising. Let me just — but let's see the code,' because we'd seen some nice slide decks. But —
That's crazy. We need a movie about this. I feel like this would be a good movie. Oh, actually another thread of this — reflecting back to three and a half years, maybe even longer than that, something now sort of as we sit here that is both positive surprising and also negative surprising — like something maybe you wish we had made more progress and it's surprising we haven't, and also something maybe we've made way more progress than you obviously imagined. So much of the stuff is so hard to have imagined five years ago, but anything that sticks out for all of you?
Maybe I'll start with positive and very timely for today. I really didn't expect we could keep doing what we've been doing generation after generation, which is to pack the intelligence of Pro back into Flash. So it's kind of like that happened in 1.0 and you could say, well, it was the first run and it was fairly suboptimal in some way, so that makes sense we improved the recipe. But in a way even that seems to be sometimes even accelerating, depending on which version we look — that Flash next-gen outperforms Pro previous-gen. I mean, just even understanding distillation, how it works, I'm still mesmerized — how can we pack so much intelligence per byte or per parameter?
Has distillation fundamentally changed? I know of the concept, I don't know the details. Is there architectural improvements to the way we do distillation which is part of how we can keep packing more in, or is the technique relatively the same of what you all came up with originally?
Yeah, I would say it's even simpler. I mean, we had some trick with temperatures in the softmax and we had to take an ensemble of models. This —
I'm going to spill the recipe.
You have a really, really good teacher and then you have a student. But you didn't need an ensemble of 50 teachers. You just have one really good teacher and then one student. And you pretty much use the recipe described in the original paper with some modest tweaks, but the basic spirit of the idea is pretty much the same.
Wow. Let me give you the most technical explanation. It's like squeezing the lemon. You squeeze the lemon, the juice comes out, it's the good bits, you put it in a glass, which is your small model.
You should read the intro of the paper. It has some poetic intro as well about larvae and insects.
The original paper that was just like soft labels and —
Anything that's sort of — you're surprised we haven't been able to pull off, given how much progress across the board Gemini has made over the last three and a half generations?
I mean, the good side — thinking back to —
Good — it's also about the beginning of Google, right? We have this — what was it — this one-box philosophy, right? Jeff, you must remember the one box for everything, like the search box — also you could use it for —
Type in something, it would show you sports scores. Type in something else, it would show you stock quotes.
Right. And on the back end, these were all separate, very separate backends, and our custom-built — whatever, some of them were AI-ish and some of them weren't.
Spelling — did you — I think that was largely Noam's starter project, I think.
Oh yeah, the user would assume, 'Oh, there must be some brilliant general-purpose AI behind the whole thing and it knows — it can do all these different things.' And now we actually built it, like —
We built the general-purpose AI that —
