Oriol Vinyals is the co-lead of Gemini alongside Noam Shazeer and Jeff Dean. He's had an incredible career in AI, pioneering many of the breakthroughs in deep learning in the last decade. And it was a ton of fun to get to sit down with him after Google IO. If you've been following Google IO, they basically shipped a bunch of products across a ton of interesting surface areas throughout AI. And so Oriol and I hit all of them. We talked about what's required for further advances in multimodal models and what's going to make these world models actually usable. We talked about the increase in memory and the importance of memory and how the advances there will look like reasoning these next few years as well as what Oriol thinks the path forward is. And we hit on the state of scaffolding today, what folks are building and what Oriol thinks persists. It's a ton of fun to get to basically take all the top questions that founders, investors are thinking through and just pose them to Oriol. So I think folks will really enjoy this conversation. Without further ado, here he is. Oriol, thanks so much for coming on the podcast.
Yeah, it's great to be here. Thanks, Jacob.
Yeah, very exciting to have you a day after IO. I know things have been busy, but I've been really excited for this because you're one of the people kind of most directly shaping the frontier of models today and your work at Google. And you obviously in the releases that happened yesterday at IO, they hit on like pretty much all the themes that people are thinking about in the space, where these products and models are going. And so I feel like there's just our goal today is to talk through kind of the research behind those announcements. You know, where this is all headed, you know, the kind of future path of RL and post-training and, you know, get your read on the space as a whole. I figured where I'd start was with world models because I think that was just a really impressive part of yesterday and also I think a pretty, you know, where Google's pretty distinct from a lot of the rest of the field. So you obviously shipped this incredibly impressive world model in Omni yesterday and you know I think Demis has talked a lot about seeing world models as a path to AGI and it's interesting right because it seems like other labs maybe are more focused on code and getting to recursive self-improvement and so I'm wondering if it's a fair characterization and you know why you think you and the team in Google have been somewhat uniquely focused on this world model space.
First of all, I guess the coding or like self-improvement angle is at a bit of a different layer, right? So, you can certainly bet and believe that these models can reprogram and improve themselves and it's something I've been actually quite actively working on at the moment. But then the object that they improve, the model whether it's multimodal and closer or a world model as we call it and that even how to define that is a bit abstract, since day one and way before actually Gemini program started we were working on not just language but understanding the visual world and kind of jointly modeling words in the context of vision, video, etc. So I think that part, it's been at the core of Gemini and before our research and I think maybe one way to characterize it is, you know, language clearly there's a lot of information collectively that we wrote about the world. So that's clearly paid off big time. We've kind of distilled in a way all the knowledge written and that is being written at the moment into these weights.
It was definitely convenient that we put it all on the internet too.
Yes, exactly. Right. So, and also like this now with users, right? There's obviously like also a flywheel effect, but at the same time, there is lots of knowledge in videos and images and what I would say it kind of has happened but softly. I think there probably might be a big moment is how would you extract all the knowledge that you would acquire if you were to look at all the videos and images which we certainly use right in our training mixtures but could that knowledge somehow add value and efficiency to the language component and I think we've seen constructive sort of let's say transferred learning right from one to the other we see that and we see generalization but probably what I would characterize as the GPT moment of video and images. I'm not sure we quite have seen that.
Do you have any thoughts on like what that GPT moment might be for video and image as you kind of have this intuitive feeling that it hasn't yet been reached.
Yeah. So at the moment we train all the modalities we mix them and we keep enhancing the recipe. So Omni is a good way to see that progress in which we not only input videos and images, we've seen amazing capabilities with long context understanding etc. But we also now are able to output, you know, video but also interact with it in a very natural way through language, editing it, combining the modalities in a way that feels almost magical right so that progress is absolutely there but maybe one of the deep learning dreams and it might be an original kind of dream from way before large language models would be hey can I train on all the image data without text perhaps as a hard challenge and still somehow extract all the meaning and nuance from that modality or set of modalities and vast amounts of data. Right? So could we train on all the videos ever produced and images and get to the same level of understanding that clearly the language models using language get to although probably slightly superficially and some missing links with cause effect and so on that for instance Demis talks about often right so that is the moment that have I seen that probably not and most likely we have the most advanced or one of the most advanced multimodal recipe that mixes everything but that pure transfer is I think one of the core quest of machine learning for the last decade plus.
I mean to the extent you can talk about it I'm curious could you give our listeners some context on like what are still the key problems that need to be solved around this or as you think about the types of problems that you're trying to work on to further advance this. It's hard to describe the solution space but the idea of, you know, you could imagine observing or learning from all the video data and then somehow deriving the rules of gravity is one that is used often right like how could you precisely describe how the world works based only on images right and so the issue there is linking language or these concepts as we sometimes call to what you see in the image without the explicit language linkage is fairly tricky, right? So, so what you end up doing is trying to explicitly create data sets where there's some sort of correlation or connection between the images and video and some language like maybe it's labels or descriptions and so on. But of course, the amount of data now at your disposal is much less because we haven't clearly described and transcribed every single piece of media out there. So I think that's kind of extracting those concepts in the purest form not in just some language that we associate to the words and what we see would be very very powerful and there's lots of early research on discrete representations, representation learning and I mean that's one of the things that probably I would say is in fairly research stage. So it's not something we can possibly scale up but I think that's one of the possibly I'm not sure it's needed. I mean whether we agree with that or not is another question but it's if it was to be unlocked it would be massive.
You mentioned kind of this the term world model and how it's thrown around a bunch and you know obviously you kind of Omni was positioned as a world model and I'm curious you know how you thought about that categorization versus you know you obviously had really good video models for a while right what makes Omni like a world model and you know how is it different from kind of the generation of video models that you guys have been working on?
I guess a pure aspect of world model would be representation learning, right? So you could imagine we take these modalities like the videos which are like a sequence of images or even just images and then compressing that into sort of a set of concepts and what those, you know, the movements, the objects etc are within those. That's kind of called representation learning and it models the world in a very compact way that compresses away what's probably not relevant right so probably that one is the more classical but also probably not exactly what we mean or we see or we feel when we interact with Omni right what you see there is a bit more about you being able to really change how the video behaves or the kinds of videos you're getting out of an initial maybe image that you ask to animate. You explicitly ask all the movements or even like actions that would be like move forward and you can see that being kind of precisely simulated. And so that is more of like the world model itself is acting as a renderer of the world that you can really just change by a language and then having that now object besides being a cool product to play with of course like we love to generate all sorts of different movements or situations and so on very rigidly. It could also meaningfully add maybe a dimension of simulation that could make us, you know, use for example things like prediction before acting in the world and of course obvious applications for these kind of 3D or video world models would be clearly self-driving cars or robotics.
It seems so relevant to robotics and it feels like everyone's kind of still trying to figure out the right data mix of simulation data versus teleop data and egocentric video data but it seems like as these simulations continue to get better it's more and more of a compelling thing to put into the data mix and I'm curious does this work then directly intersect with the broader robotics work you all are doing and how do you think about what's actually required to append robotic actions onto these types of models.
There's a bit of a also beautiful connection because of course if we acquire even if it's obviously a bit more expensive or time consuming but if we get more data that is captured from robots that we certainly are investing in, that data could make it into the model enhancing the world model capabilities themselves and then the other direction which is kind of what you're asking about perhaps is okay now we can simulate and we could create lots of different scenarios in which these robots or whatever 1D, 3D groups etc could be training on without the cost and the time latency of the physical world right so for the latter to work better I mean it's still a very open problem there's also all sorts of issues with transfer but the more powerful these models get clearly there's kind of an inflection point where things start to be worth doing and we might see an acceleration in robotics indeed in that definitely we're seeing in the hardware space lots of investment so things are accelerating and picking up there but for the world models to be useful at least from my limited knowledge but of course I've been able to interact with these systems and see them that the precision of even grasping a model which we get for granted as humans, the visuals, the exact how would this feel to your hand which is a modality we currently obviously don't even have data for and then the exact forces how things would move it needs to be very very accurate right so that's where there's a gap and perhaps then some creativity and research is still required and lots of investment in robotics over the years but it's promising and at some level maybe not at the precise motor control but at the kind of planning and gross we are going to start seeing how these models accelerate our progress into the quest of robotics.
A huge part of these models is kind of like learning implicitly learning physics through consuming lots of video data. And so I think you mentioned gravity is like the canonical example of what people look for. Do you have any kind of gut sense being so close to these models of like when you think that will just be a solved problem within world models?
Yeah, it's a good question actually. You kind of made me think about evaluation, right? Like how would you evaluate if you train a very good video.
Yeah. How do you evaluate physics in a model?
It is a good question right you could imagine the problem is as soon as you add language all of a sudden that knowledge is there in the weight. So if you ask basic questions about gravity of course you would answer them by just having read explanations of them online and so on. So you would need to somehow connect the concept of gravity which could be present or not in a world model to then decoded that into an explanation that would satisfy, you know, maybe initially would be some basic explanation later on could even derive the equations and so on that's how you could build an eval. I don't think to my knowledge we've been thinking about this from this point of view there's definitely lots of early work on unsupervised machine translation where you would try to translate to a language that you would never see during training and you could align the representation. So there's probably some ideas on you get a language model that can speak or you can decode from you get this world models that would create this kind of conceptual level understanding and aligning both, you know, there's some papers I mean these are like old papers the one I remember from I think it was Stefan Gaus et al was from 2014 but then you could try to start decoding that and converting that to an eval seems then a trivial step but again these need to then be meaningful from an application point of view. So ultimately you could also say look I mean we have a world model can we decode or I don't know like induce movement in a complex system from its representation for example right that would be another indirect eval so many ideas but yeah are so important always.
You know, shifting gears to some of the other stuff that you all shipped yesterday, you know, I definitely want to talk agents. You shipped some really interesting consumer agents in Spark right as part of IO and you know, I think it's so interesting because it seems like from the outside at least like a really improved version of some of maybe the stuff you guys had explored in Project Mariner in 2024 and some of this other computer use work and so it does feel like there's been a real step change in what the capabilities are and so I'd love to hear you just riff on the research breakthroughs that enabled that and then kind of how people should think about what these agents can and can't do today.
We knew that was going to be a very important modality actions right acting and changing a state of let's say a digital computer and then I think as you evolve and make the model better you start realizing sort of out, you know, you get the model really good and then you focus on the system building a system around the model then optimizing the system and the model jointly as much as you can and so on so forth. So in terms of what creates the delta or the increase in capability, it's mostly focused and about it's about sequencing sort of releases. And also in some sense the model capability needs to reach certain level for you to then be able to dream about what's the next stage of capability, what the model might do next.
Yeah. And I guess one thing that's so interesting about the consumer footprint is like there's just such a broad array of things people want to do with it. And so, you know, I wonder like to date and how you see this evolving over time that work of model plus system, how bespoke is it to subcategories of the problems people want to do versus, you know, incredibly general and like, hey, you're just optimizing a system and model combination that works across pretty much everything you might want to do in Spark.
There's always a sequencing to specializing to something that feels controllable and that, you know, very useful already. Where, you know, if you look at Spark I mean it has access to information that it would be needed for it to be able to assist you in sort of scheduling and organizing your day and even thinking about how you should tackle different problems because it has this very rich context. So it's useful to build kind of the system slightly more narrowly around something you care deeply about. But if you look at the history of machine learning and deep learning, we always go from the components we're building are general. So and there's a big hypothesis which goes a bit to the world model point actually that training on everything jointly must be better than just focusing narrowly on just one domain. So even from the modeling perspective that is very clear. But even from the systems perspective a system that is fairly generic and then based on how you instruct it or you interact with it you can then of course put it in the space of like hey I mean this user wants to do this but I have all these capabilities let me just figure out which ones to use kind of at train time not necessarily building it for that but building something generic and then the specialization happens through a layer of intelligence right of the model and the generality of the system. I think that's fairly clearly already here and then maybe sometimes in practice limiting or making it maybe more efficient still makes sense to specialize but the specialist to general we've seen it just keeps happening both from even architectures right I mean the transformer was a machine translation neural net right and now it does everything from Omni to controlling your computer so yeah I think that's a step that I expect.
You've been vocal about the bitter lesson over the years. And I'm curious like as you look out at the field, are there places where you think it's not currently being followed or you know basically places where you look out and you see kind of structure or clever scaffolding that you think scale is just kind of eventually going to wash out?
Yeah, I think so. I mean one area that I find exciting there's some research on this already kind of published is that in the limit the system that we build now sort of by coding sometimes a complex scaffold around the model, you know, multi-agent, sub-agents, delegation, very long running that system itself is a piece of code that eventually the model itself could write on the fly right so you could imagine not having just a system that is very general but actually maybe no system and just the model being able to write those depending on what it's being asked to do.
Like almost the most token efficient like highest quality output set of sub-agents and whatever it is around a set of problems.
Yeah, exactly. I mean we've seen this also in the kind of one of the paradigm shifts we've seen in the last one year and a half or so is of course the reasoning models that can reason for a long time in token space but of course eventually what becomes more important is should you reason for how long should you reason and adding that level of intelligence based on the complexity of what a user might be asking will make it more efficient. So I think what you do around these systems there is going to be a level of not sure exactly if it's going to be right from scratch or some automation that will make it smart to create the right scaffold for the right task on the agent side.
You know, I think there's a lot of everyone's messing and experimenting with building these kind of long-running agents and I think obviously they run into all sorts of issues trying to get them stable across hundreds of steps how do you think about what's required to get to further agentic reliability.
Yeah, I mean I think that the answer to these questions in the most obvious way is kind of improving both the scaffold around the model. If you think of how you train a neural network it trains on some distribution of tasks or modalities or how to connect different words to video or whatnot, right? All these are all about how you train or pre-train or post-train these weights. So if you think well there's a new type of work or modality that requires these very long running systems that need to somehow learn from this very long context which we have also always kind of innovated and pushed in 1.5 was kind of our long context breakthrough then it becomes obvious that the model also will catch up right to meeting the users and the futuristic use cases and that's a bit of the researcher challenge, right? Predicting what can be possible and then focusing not only on building a system that is robust to that, but also how would the weights get less unhappy or happy about when you push all the context and all these crazy things that you do and not just hoping on generalization from the prompt that induces that behavior, so to speak.
A pattern everyone's trying to figure out is like memory, right? And how to kind of solve this across these agents. Do you have any gut instincts on where you think that ultimately gets solved?
Yeah, I mean memory is fascinating. I think since very early days, right? You can sort of think of I think initially we characterize this and this is probably through biases from actually Demis having had a PhD in memory systems in the brain, right? But there's a few ways to think about memory but the simpler one that I like is just working memory right things that are very present because of what we're doing or we're talking about and then what's called episodic memory that is kind of a retrieval system that you can access and it's probably less precise but of course longer context or potentially has all the context that you or I care to remember right holistically with all our experience that is accumulated. Now there's not only two levels of memory but it's useful to think about these kind of levels of memory computers have the same with cache L1 L2 and so on. So when it comes to models, I think working memory because of transformers and so on is we have a very powerful mechanism to kind of use that memory, you know, have hundreds thousands millions of tokens at our disposal to modify that memory and then do amazing things with it. Proving complex theorems, you know, gold medal level maths and so on. And I think where I'm seeing a lot of momentum is through then how can we consolidate then things that happen either previously in different interactions or throughout an interaction that might be longer than you could possibly remember in this working memory. How do we store that knowledge and through different experiments I think other, you know, like I think now the standard name is called like what we call skills but it's more general than that we do have access because it's an agent to a memory system which is the computer itself. So you can start thinking about writing your thoughts into files structure it into directories or folders and doing that as you either interact with the same user multiple episodes or a very very long episode. So the mechanism that is fairly good at the moment but again I don't think the weights of the model have caught up to this is this adding this kind of knowledge base into a file system or any format that is storage that you can modify and read from with some basic retrieval mechanism. So that's very powerful already yet I think there's still a lot to be untapped there. I think many of us call this kind of a form of continual learning. But I think the mechanism I want it to work kind of or I mean it's going to clearly work better and better is this kind of file system style like non-parametric. It's a bit more convenient than integrating those back into the weights because even from a practical point of view we try to serve one model at scale. So it would be really painful to have to serve one model with different memories to users. So even from a practical point of view, I think we'll see better evaluations and ways in which these models accumulate this knowledge as they interact and I think that's probably paradigm shifting as well in a way similar to how we saw reasoning a year and a half or so ago.
Does that look like everyone having models that then have their own, you know, the file systems themselves being distinct or do you think over time people have models whose weights look different based on what they've done?
Well, as I said, the ways different would be a challenge.
Yeah, it would be I mean, if it's the best way, then we'll find a way, right, to have hardware that of course we have lots of investment as well on hardware design that would allow you to have more personal weight, so to speak. But at the very least of course you will have your own knowledge base that is maybe personal to you. You're seeing already many examples of this realized over the last maybe even years in LLM space. And then perhaps there's another layer of knowledge which is more common to all the users for a given model that you could imagine having access to and enriching or enhancing the model capabilities without touching the weights. So that's very interesting and getting to that would be awesome.
I feel like continual learning has been the topic du jour and everyone's talking about it. And you've seen kind of a few interesting examples now and high-profile examples of folks spinning out of OpenAI or other places and saying, you know, hey, sure, I mean, you can keep scaling what we're doing now. And I think that no one's denying that those scaling laws are there, but they're saying it feels like you need kind of almost a new research bet to achieve real continual learning. And you know maybe it makes sense to pursue that outside of the kind of path of continually improving these core LLMs. I'm curious what you make of that whole dynamic and your reflections on that.
I was in Google Brain very early days and then moved to DeepMind in 2016. And at the moment I think there is a challenge and an opportunity on you want to obviously investigate some research questions that might not be viable for hey in the next three months this makes it into the next training run. But at the same time this cannot be very disconnected from the head right where the LLMs are moving. I mean we're improving Gemini I mean it's fascinating to see Flash outperforming Pro that of only few months ago and that keeps happening. So keeping kind of at the head of capability which might enable or disable certain research whilst having the protection for research and of course that's not multi-year anymore things are moving fast but combining these two is kind of the magic of building these organizations and all of us of course have different angles and can figure out how to bridge this and identify the opportunity that's a bit of what it takes right to I mean not have full visibility this is too large of an organization but have some intuitions and then be able to pull in these ideas eagerly sometimes right because it feels like the right thing to do. So that's really what defines actually organizations at that level, right? From a research perspective. So I can see from investment in robotics to the peak of the LLMs to research that either has made it or will make it through, right? So but it's challenging. It's resources are constrained. So it is an interesting trade-off and not one always you get right. I mean I think it's a fascinating kind of different angle of research not just what is the idea that will make it to the next paper or now into the model but actually how to even organize this whole organization.
