Hello and welcome to the Rethink Reality podcast. Fei-Fei, your work is incredible. You've helped machines see. But what I've been really fascinated by when I look at your work and read your book, do you think an intelligent system can truly see without having to act on what it's seeing?
That's such a great question. First of all, Don, thank you for inviting me on your show. It's great to be here. You know, when one of my kids was six years old, we talked about why trees don't have eyes, and it's very much in the spirit of your question. Of course, evolution did it the other way. Evolution in the early days, 540 million years ago, the simple animals started to acquire sensory capability, very rudimentary perceptual sensing as well as some tactile sensing. And that got them to move a lot more. So from an evolutionary point of view, perception and sensing is actually a driving force towards embodied movements. We don't have an evolutionary counterfactual to ask the question that without moving, do you see, do you need to see? I think from a technology point of view, passive seeing is actually a legit technology, right? Like the abstraction at the software layer of just observing and understanding is possible. That's actually the early part of my own work. But from a full intelligent, unlocking full intelligence, especially for physical and embodied intelligence, I think the loop between perception and action and the interaction of the body is so intertwined.
Right? Like that movement is like what you're saying about evolutionary, that was the motivation to get animals to go do something and act on the world. They had to go see it first. So do you think that perception of seeing is incomplete without action?
That's a great question. These are very nuanced definitions. If seeing is seeing something, I think perception can be done without action. If seeing is defined to the fullest scope of what we do, what we can do as humans, then it's so entangled with our action. Now obviously even humans have individuals who are less mobile and we cannot possibly say that they don't have full scope perception. Right? These are clinical cases. Some are just very, you know, not statistically the majority but it does happen. Then we have to get into how is the immobility affecting perception? Is it, did they acquire halfway through life and also are there other tools to help the person to be mobile? So there is a lot of nuance, but overall the coupling between perception and action for me is extremely tight.
Hearing you say that, just random curveball here, the animal, the bats. What do you think seeing is for them? Because, you know, I feel like they have a very good sense of awareness. They catch prey, they hunt, they have mates, they know where shelter is, but they're not seeing in the traditional sense. So, what are your thoughts on bats? What are they seeing?
They're definitely animals like bats. There are also underwater animals that use sonar. And first of all, they're sensing. It's very important, their sensing. They're just sensing without the spectrum of light that we typically, many animals do. So, they're sensing the world. They have their own representation of the world through sonars, through the echo. So they are getting a great loop between sensing and action. Now is this a counter example to the importance of perception? I don't think so because perception fundamentally is sensing. And in fact why don't we have more species like bats? Well sound-based sensing is a less powerful way of seeing. Right? Bats are not as sophisticated as dogs in terms of their range of motion and habitats and behaviors and it's limiting to their behaviors for sure.
So it sounds like sensing is a prerequisite for something to understand the world and the type of sensors you have would you say impact what kind of worlds you can be in?
Absolutely. In AI we now have this concept data defines your model. You know, we even have this thing that garbage in, garbage out. I think for animals and humans the same. How we glean information, how we sense information defines how we understand and act upon this world.
Wow. I mean, that's a good point. You know, I love your logo and I love the work that you do at World Labs. We're going to talk a lot about that, but one question I have is on the philosophical side is what counts as a world?
Oh, wow. This is such a great question. There's so many dimensions to dial here from religious, spiritual to physical, material. What counts as a world? I'm actually really, really, really curious. Are you in front of a computer? What would AI say? What counts as a world?
Uh yeah, I am in front of a computer or Gemini. Let's see what comes as a world. I'm actually really curious. I have many thoughts but I'm, you know, a world can be defined as a geometric space right where things can be put in, things can happen, things can move. You know, it's almost can almost define it as equivalent to space, that's one way to think about it, very physics way of thinking about it. But you can also think about it in a much more abstract way where even if thoughts are allowed to happen there is... you know, there's an inner world, you know, as long as there's the flow of thoughts there's world.
So one fun challenge I would have to you on that one is cells. Do you think cells, individual cells going about their business in the body, do you think they have like a world that they understand?
Oh absolutely. There's a beautiful poem now I forgot, I would like to remember the poet's name. It's slipping me. It was, it's a very Buddhist poem that says even a grain of sand or even a petal of a flower is a world. And of course, a cell can be a world. First of all, it satisfies the more physical definition. It's a space. It has things happen in it. The molecules and the structures and all that. So absolutely a cell is a world. I think a grain of sand is a world but I also do think there is an abstract world.
So I did get a definition from Gemini.
Says the term world is remarkably flexible shifting its meaning based on whether you're talking about an astronomer, philosopher, game designer, poet. Said at its core a world is a self-contained system defined by a set of boundaries, rules and inhabitants.
First of all, it touched poem, philosophy, physics. You know, we're doing well. Second, yeah, self-contained. I mean, how self-contained? Like I said, a world can be thoughts moving in it, and that might not be as bounded as a physical world.
Yeah, they did this study recently with worms. They're trying to see if memory can live outside of a nervous system. They did a really interesting way of testing this theory with worms. Have you seen that study or familiar with it?
No, but my first reaction is called muscle memory.
Okay. Right. Yeah. I think that totally, there's this worm and they taught it how to get through a maze and it remembered how to do that and then they cut up this worm into much smaller pieces and had it kind of like regrow itself and that new worm was able to get through the same maze and as if it had, so what they were confused by is it's like where was the memory of that stored if they had to regrow everything.
Yeah, that's fascinating, right?
I don't know that study, but here's something that people don't appreciate. The morphology of us, of every flower, every tree, every animal is a memory of evolution. So, our morphology is part of our intelligence. And there's actually a term called morphological intelligence. So, you know, we're so holistic in that way. It's not just our brain or just our eyes or just our body. We are one integrated system within our own body as well as within evolution.
Wow. I've actually never heard that term before. I'm definitely going to dive deep into the, is it what you said? Morphological intelligence. Wow. Yeah. Because you can't really separate humans from the fact that we have to be in constant interaction with our environment. It's like we...
And we evolve that, right? Wow. So, here's a fun thought question for you. If an AI can perfectly describe an environment, but they've never interacted with it, does that constitute understanding, or is that just pattern recognition?
I think that's understanding. I actually think pattern recognition is part of understanding. You know, a child is allowed to understand tiger without ever interacting with a tiger.
Yes. Exactly. So I absolutely think that's understanding. Is it all there is to understanding? I don't necessarily think so. So now we can get more nuanced. For example, if you've never interacted with the environment, then learning affordance is a much more complicated task. You can always be trained by big data. But you know the famous example is making just ad hoc tools, right? It's if you're stuck in a forest, how do you without hammer, without nails, without anything, how do you build a little shed? At this point, that level, the human level of creativity really shines. It's very hard for a machine to have that. So, there is a lot more, but the short answer was yes.
Do you think AIs can ever have common sense?
Common sense is not that hard. Common sense actually might be even easier than one thinks because many of our common sense are codified in our language, in the data there is that the AI can learn from. For example, in a very extreme hot take, and I might be disagreed by linguists and that's okay because I'm not a linguist, is that language itself is common sense of communication is that we learned through generations and generations, thousands of years, right? They distilled these commonly used things distill into language. Now, you know things like draw a sun and then even today's AI tends to put the sun in the top half of the picture rather than bottom half of the picture. These are common sense. And where did this come from? You can say it comes from deep Newtonian understanding of gravity and celestial bodies and all that. But you can also say this is a statistical norm.
So these common sense AI absolutely can learn it. In fact, a five-year-old and AI probably are not that different.
Okay. Wow. I wasn't expecting that. That was really cool. Wow. I was thinking that common sense was a hard problem. But if you're describing it as like the average of a norm, then it seems like that's not that complicated.
Common sense has, there's a lot of data about common sense in our language, in our videos, in our images, and these data are abundant. But extracting common sense into deeper, more profound laws. For example, you talk about the movement of the stars, that's a different level, right?
Yeah. I mean, kind of transitioning a little bit into physics and more specifically world models. I noticed like when I was reading your book, you talked about like how ImageNet had a lot of human labeling involved in helping it understand the world. My question for you is with world models, physics feels objective to me, but is there a subjective layer like telling an AI that this is heavy or slippery or fragile? Like what do those words mean? Is that us adding human bias to physics?
Great question. I think our world and when I say the word world is the complex human world is always going to be complex. Some part is going to be objective and factual, some part is going to be subjective. World model the ones that World Labs is building is very multimodal. It has language and semantics. Our model has language and semantics as well as geometry and eventually physics. So that the interaction between language and geometry and physics definitely intermixes objectivity and subjectivity. So for example even the word blue right that is subjective and also the word heavy right if you go to today our model and say you know put a heavy couch in this middle of this living room it is very subjective and that subjectivity is prone. So this is all part of the entanglement between data and modeling in AI today.
Yeah. I mean I love Gaussian splatting in general just like when you can pull apart the particles and you can do a lot with them. One question I had for you is around resolution like what do you think is the optimal resolution? Should it be like Gaussian splats modeled eventually at the atom level?
Oh my god that's such a, that's a great question. I do think about our world as fractal of resolutions. So for you and I living our daily life, I don't think we need atomic level resolution. But when we get treated by a medicine, yeah, we better hope that that medicine works at the molecular level, right? So that means the scientists who were testing and developing and discovering the medicine are definitely working at the atomic and molecular level. So I think the resolution almost depends on the use case and the scenario the world is very nested. Speaking of definition of world.
So it's like because if a world is a closed system there's systems within systems within systems within systems.
Absolutely. It's extremely nested.
Wow. I wonder, okay. Do you think there's a resolution of reality itself? Like if you, I don't even know what that is. Like what's the pixel? Is it the Planck length? Is there a limit to how small you can go?
I think pixel today is mundanely defined by hardware.
Right. The resolution of your screen, the resolution of the digital camera. That's how it's defined.
Okay. That's fine. I was just curious if, yeah, because the one thing I really like actually about your particular world model is you can go behind things.
And around things. I know I play with a lot of tools and right now that is kind of difficult. That is very difficult for AI to properly do, but it seems like consistently when I build something in World Labs, I can get around and behind and underneath. So, how does it know that?
Intentional. That's intentional. I think we are very clear from day one. A world model needs to be 3D. You know the concept of behind and underneath and above is a spatial concept, right? And why do we believe that? Because so many use cases from gaming to VFX to robotics to simulation needs that to design, you know, put the chair behind the couch, you know. So, yes, that was intentional in our model representation.
It shows because like I can move my 3D cameras around and I'm like, 'Wait, there's stuff back here.' What? How does it work?
And also, it allows you the perfect control. You can control your camera in a pixel perfect way. Right. That's very different from other AI tools where the camera just pans roughly in certain ways. Exactly.
You're just rolling a dice usually. But here, I can be precise. It could be like I need the camera to only go 20 ft. That's it.
Not this proverbial, you know, random motion.
And I guess that's because it's actually 3D. It actually has that data. Otherwise, you're kind of just guessing what the depth might be.
There's no guesswork. It's like this is the depth. Here it is.
Exactly. It's refreshing. If you had to, oh, this is a fun one I had for you. If you had a fictional dial that could set the physics resolution of a perfect simulation, what would you set it to?
Oh my god, that's again, it depends on what I want to do.
Well, knowing you probably researching or making something, so like.
Yeah. Well, knowing me, I'm going to do robots and with World Labs, our target industries are gaming, entertainment, robotics, and simulation of the real world. So the resolution would be at, it would not be at subatomic level for sure because I don't think I need that. But if I'm a high energy physicist or if I'm a drug discovery scientist, I would dial that resolution down on Planck or whatever.
Okay. So it's really dependent on the domain that you want to focus in and what scales they need to work with.
Yes. I was surprised to see that you added a collision mesh to the splats. That was like a what when I first saw that because every, you know, usually I'd have to try to like approximate those things, but yours comes with it out of the box. That's very helpful.
Yeah. Because game developers need it, robotic simulation needed, you know, also iterative design of a space needs it, right? So I don't send the chair through the wall, right? One thing I was curious about, I was just following some different research papers and I'm seeing there's some research where people are trying to animate Gaussian splats because worlds are not just static, you know. So, is there a world where we might see in the future where World Labs has, I don't know if that's spoiling things, but what if there's movement in those splats like a tree sways and.
Yeah, we call it dynamics. Absolutely. 3D 4D modeling, right? Dynamics is on our road.
Wow. Oh my goodness. Because then you can really train a robot. It would be a perfect scenario if the world did not move at all and then you ask the robot to go learn. But that's all I see from all the training models is people have these static worlds and then they teach the robot how to walk in it and then it's like well what if the ground moves? You know what if a car goes by or you know a stick falls in front of that pathway?
On the topic of robots kind of switching gears here. Would you say that proteins, insects, or simple organisms could be a form of biological robot?
I've seen nanobots that can go down in blood vessels to deliver drugs. Now, I've seen prototypes of, yes, they call it nanobots. I do think the word robot is very stretched right now. You know, I used to half joke. I actually mean it that a washing machine is a robot, right? It's a machine that moves and does tasks. Obviously, that's not your typical vision of a robot, but so is not a nanobot that goes to deliver molecular level treatment in your blood vessels. So, I think if our definition of robots is anything artificially made that can do tasks, then yeah.
Okay. Because you like build a protein is if I go and engineer a protein that does delivery of a medicine. Is that a nanobot that I just developed?
Yeah. This is where you know coming from the great Stanford campus I have so many colleagues in synthetic biology in robotics that I almost need to respect their more professional definition or more expert definition of robot. I don't know a particular protein that can deliver, some artificial protein that can deliver a particular function. Is it called a bot or not? I don't even know. But that's how fascinating our science is. We're starting to see the intersection of engineering with biology, you know, physics with chemistry and that's what's fascinating now.